Grading of Recommendations, 

Assessment, Development, and 

Evaluation (GRADE): 

Janssen COVID-19 Vaccine 

ACIP COVID-19 Vaccines Work Group

Dr. Julia Gargano

ACIP Meeting

28 February 2021
2

 Should vaccination with Janssen COVID-19 vaccine (single-dose, IM) be 

recommended for persons 18 years of age and older under an emergency 

use authorization?

Policy Question
3

PICO Question

Population Persons aged ≥18 years

Intervention Janssen COVID-19 vaccine Ad26.COV2.S (5×1010

viral particles, single-dose IM)

Comparison No COVID-19 vaccine

Outcomes Symptomatic lab-confirmed COVID-19

Hospitalization due to COVID-19

All-cause death

SARS-CoV-2 seroconversion to a non-spike protein

Asymptomatic SARS-CoV-2 infection

Serious Adverse Events

Reactogenicity
Outcomes

4

OutcomeImportancea Description

Benefits

Symptomatic lab-confirmed 

COVID-19

Critical Primary outcome; current studies use PCR + specific symptoms

Hospitalization due to 

COVID-19

Critical COVID-19 requiring medical intervention

All-cause death Important Death from all causes

SARS-CoV-2 seroconversion Important Measured using antibodies to non-spike protein to differentiate seroconversion 

due to natural infection from immunogenicity to vaccine

Asymptomatic SARS-CoV-2 

infection Important No serial PCR; no systematic PCR after day 1 – not assessed

Harms

Serious adverse eventsCritical Evaluating balance of events between arms; also reporting on number deemed 

vaccine-related

Reactogenicity Important Evaluating grade ≥ 3 severity of systemic events and local reactions

aThree options: Critical; Important but not critical; Not important for decision making
Outcomes

5

OutcomeImportancea Description

Benefits

Symptomatic lab-confirmed 

COVID-19

Critical Primary outcome; current studies use PCR + specific symptoms

Hospitalization due to 

COVID-19

Critical COVID-19 requiring medical intervention

All-cause death Important Death from all causes

SARS-CoV-2 seroconversion Important Measured using antibodies to non-spike protein to differentiate seroconversion 

due to natural infection from immunogenicity to vaccine

Asymptomatic SARS-CoV-2 

infection Important No serial PCR; no systematic PCR after day 1 – not assessed

Harms

Serious adverse eventsCritical Evaluating balance of events between arms; also reporting on number deemed 

vaccine-related

Reactogenicity Important Evaluating grade ≥ 3 severity of systemic events and local reactions

aThree options: Critical; Important but not critical; Not important for decision making
Outcomes

6

OutcomeImportancea Description

Benefits

Symptomatic lab-confirmed 

COVID-19

Critical Primary outcome; current studies use PCR + specific symptoms

Hospitalization due to 

COVID-19

Critical COVID-19 requiring medical intervention

All-cause death Important Death from all causes

SARS-CoV-2 seroconversion Important Measured using antibodies to non-spike protein to differentiate seroconversion 

due to natural infection from immunogenicity to vaccine

Asymptomatic SARS-CoV-2 

infection Important No serial PCR; no systematic PCR after day 1 – not assessed

Harms

Serious adverse eventsCritical Evaluating balance of events between arms; also reporting on number deemed 

vaccine-related

Reactogenicity Important Evaluating grade ≥ 3 severity of systemic events and local reactions

aThree options: Critical; Important but not critical; Not important for decision making

Less common outcomes
Outcomes

7

OutcomeImportancea Description

Benefits

Symptomatic lab-confirmed 

COVID-19

Critical Primary outcome; current studies use PCR + specific symptoms

Hospitalization due to 

COVID-19

Critical COVID-19 requiring medical intervention

All-cause death Important Death from all causes

SARS-CoV-2 seroconversion Important Measured using antibodies to non-spike protein to differentiate seroconversion 

due to natural infection from immunogenicity to vaccine

Asymptomatic SARS-CoV-2 

infection Important No serial PCR; no systematic PCR after day 1 – not assessed

Harms

Serious adverse eventsCritical Evaluating balance of events between arms; also reporting on number deemed 

vaccine-related

Reactogenicity Important Evaluating grade ≥ 3 severity of systemic events and local reactions

aThree options: Critical; Important but not critical; Not important for decision making

Partial data available 

from day 29 and day 71
Outcomes

8

OutcomeImportancea Description

Benefits

Symptomatic lab-confirmed 

COVID-19

Critical Primary outcome; current studies use PCR + specific symptoms

Hospitalization due to 

COVID-19

Critical COVID-19 requiring medical intervention

All-cause death Important Death from all causes

SARS-CoV-2 seroconversion Important Measured using antibodies to non-spike protein to differentiate seroconversion 

due to natural infection from immunogenicity to vaccine

Asymptomatic SARS-CoV-2 

infection Important No serial PCR; no systematic PCR after day 1 – not assessed

Harms

Serious adverse eventsCritical Evaluating balance of events between arms; also reporting on number deemed 

vaccine-related

Reactogenicity Important Evaluating grade ≥ 3 severity of systemic events and local reactions

aThree options: Critical; Important but not critical; Not important for decision making

No data available; not 

included in evidence 

profile
Evidence Retrieval 

 Databases: Medline, Embase, and Cochrane Library, written in English, restricted to 2020

 Search terms: coronavirus, COVID-19, SARS-CoV-2, respiratory (symptom, disease, illness, 

condition), vaccine, immunization, trial, double blind, single blind, placebo, comparative 

study, phase 3, immunogenicity, efficacy, effective, adverse, evidence, and variations on 

these terms

 Inclusion: provided data on vaccination with Ad26.COV2.S and 1) involved human subjects; 

2) reported primary data; 3) included adults (ages 18 and older) at risk for SARS-CoV-2 

infection; 4) included data relevant to the efficacy and safety outcomes being measured; and 

5) included data for the dosage and timing being recommended (5×1010 viral particles, 

single-dose IM) 

 Additional resources: unpublished and other relevant data by hand-searching reference 

lists, and consulting with vaccine manufacturers and subject matter experts

 Title and abstracts were screened independently by two separate reviewers.

9
Evidence Retrieval

10

Records identified through 

database searching

(n = 3997)a

Records excluded

(n = 3970)

Records screened

(n = 3999)

Full-text articles assessed for 

eligibility

(n =29)

Full-text articles excluded, with reasons

(n =26)

26 Different vaccine

Studies included in evidence 

synthesis

(n =3)

Additional records identified 

through other sources

(n = 2)

a. Search results as of 2/8/2021
GRADE Evidence Type

 Type 1 (high certainty): We are very confident that the true effect lies close 

to that of the estimate of the effect.

 Type 2 (moderate certainty): We are moderately confident in the effect 

estimate: The true effect is likely to be close to the estimate of the effect, 

but there is a possibility that it is substantially different.

 Type 3 (low certainty): Our confidence in the effect estimate is limited: The 

true effect may be substantially different from the estimate of the effect.

 Type 4 (very low certainty): We have very little confidence in the effect 

estimate: The true effect is likely to be substantially different from the 

estimate of effect.

11

NOTE: Evidence type is not measuring the quality of individual studies, but how much certainty we 

have in the estimates of effect across each outcome.
GRADE Criteria 

 Initial evidence type (certainty level) determined by study design

– Initial evidence type 1 (high certainty): A body of evidence from randomized controlled trials

– Initial evidence type 3 (low certainty): A body of evidence from observational studies

 Risk of bias: Can include failure to conceal allocation, failure to blind, loss to follow-up. Risk 

of bias may vary across outcomes.

 Inconsistency: Criteria for evaluating include similarity of point estimates, extent of overlap 

of confidence intervals, and statistical criteria including tests of heterogeneity and I2.

 Indirectness: Considers the generalizability of the evidence to the original PICO components 

(e.g., patients, intervention, comparison, or outcomes differ from those of interest1).

 Imprecision: Considers the fragility of the relative and absolute effect measures based on 

the interpretation of the 95% CIs and the optimal information size.

 Other considerations: Includes publication bias or indications of dose-response gradient, 

large or very large magnitude of effect, and opposing residual confounding.

Guyatt GH, Oxman AD, Kunz R et al. GRADE guidelines: 8. Rating the quality of evidence--indirectness. J Clin Epidemiol. 2011. DOI:10.1016/j.jclinepi.2011.04.014. 12
Benefits
Outcome 1: Symptomatic Lab-confirmed COVID-19

Studies with Unvaccinated Comparator (n=1)

14

 Janssen phase III randomized controlled trial (RCT) (unpublished, data obtained 

from sponsor)

 Persons aged ≥18 years in United States, Argentina, Brazil, Chile, Colombia, Peru, 

and South Africa

 Data evaluated: final scheduled analysis, data cut-off Jan 22, 2021

 Full analysis set: 21,895 vaccine; 21,888 placebo (used for serious adverse events)

 Per-protocol set: 19,630 vaccine, 19,691 placebo (used for most efficacy estimates)

– No immunologic or virologic evidence of prior SARS-CoV-2 infection, no major 

protocol deviations

 Safety subset: 3,356 vaccine, 3,380 placebo

– Subset of full analysis set for the analysis of solicited, unsolicited, and 

immediate adverse events
PCR testing and molecular confirmation 

 SARS-CoV-2 positive RT-PCR or molecular test result from any available 

respiratory tract sample

 Specimens from suspect COVID cases tested with FDA-approved PCR assay 

locally

 All positive specimens also sent to central laboratory (U. Washington) for 

confirmatory PCR testing

– Protocol required molecular confirmation for primary endpoints

– Not all specimens had been tested at central laboratory at time of 

interim analysis; 90% of re-tested specimens were confirmed

– Analyses reported using any PCR-positive and centrally confirmed

15
Staggered clinical trial enrollment

 Enrollment stratified by age group (18-59 years, ≥60 years); age groups concurrently 

enrolled

 In each age group, protocol specified initial enrollment of 2000 persons without 

comorbidities

– Data Safety and Monitoring Board reviewed safety data before persons with 

comorbidities were enrolled

 Median follow-up time varied by subgroup: 

– ≥60 year-olds with comorbidities : 50 days

– ≥60 year-olds without comorbidities: 54 days

– 18-59 year-olds with comorbidities: 57 days

– 18-59 year-olds without comorbidities: 64 days

16
Symptomatic COVID-19 case definition

 “Moderate to severe/critical COVID-19”

 PCR-positivea (± centrally confirmedb) AND

 ≥1 of: respiratory rate ≥ 20 breaths/min, abnormal SpO2, pneumonia, DVT, 

shortness of breath/difficulty breathing OR

 ≥2 of: Fever (380C), Heart rate ≥90, shaking chills, sore throat, cough, 

malaise, headache, myalgia, GI symptoms (nausea, vomiting, diarrhea, 

abdominal pain), olfactory/taste disorder, red/bruised toes

 Timing of outcome ascertainment: co-primary

– ≥14 days post vaccination 

– ≥28 days post vaccination

a. PCR could have been performed at local labs, central lab at U Washington, Covance, or labs external to study. All PCR used FDA approved assays.

b. According to Phase III protocol, “Molecular confirmation of SARS-CoV-2 infection by a central laboratory will be used for the analysis of the case 

definition” 17
4 options for symptomatic COVID-19 case definition

 Timing of outcome ascertainment

n = 464 n = 259

n = 682 n = 437

≥14 days post vaccine ≥28 days post vaccine

PCR+ at central laboratory

PCR + from any source

 Confirmation at central lab incomplete at time of analysis; of samples tested 

at central lab, 90% were confirmed (“centrally confirmed”)

 Case numbers limited post 28 days; limited follow-up time

 ≥14 days, PCR+ from any source selected for GRADE 18
Outcome 1: Symptomatic Lab-confirmed COVID-19

Studies with Unvaccinated Comparator (n=1)

19

Population Events/Vaccinea

(n/N)

Events/Placeboa

(n/N)

Vaccine efficacy 

(95% CI)

Primary Outcomeb,c

Aged ≥18 years 173/19,514 509/19,544 66.3% (59.9%, 71.8%)

Aged 18–64 years 157/15,544 441/15,552 64.7% (57.6, 70.8)

Aged ≥65 years 16/3,970 68/3,992 76.5% (59.1, 87.3)

Aged ≥75 years 1/751 9/690 89.7% (26.0, 99.8)

Any comorbidity 70/7,777 194/7798 64.2% (52.7, 73.1)

a21,895 and 21,888 persons were randomized to vaccine and placebo

bCases diagnosed ≥14 days post vaccination among persons without evidence of prior SARS-CoV-2 infection

C Primary efficacy population (per protocol); includes a total of 3113 person-years of observation in vaccine group and 3089 personyears in placebo group

CI: confidence interval
Outcome 1: Symptomatic Lab-confirmed COVID-19

Studies with Unvaccinated Comparator (n=1)

20

Population Events/Vaccinea

(n/N)

Events/Placeboa

(n/N)

Vaccine efficacy 

(95% CI)

Aged ≥18 yearsb,c 173/19,514 509/19,544 66.3% (59.9, 71.8)

Aged 18–64 years 157/15,544 441/15,552 64.7% (57.6, 70.8)

Aged ≥65 years 16/3,970 68/3,992 76.5% (59.1, 87.3)

Aged ≥75 years 1/751 9/690 89.7% (26.0, 99.8)

Any comorbidity 70/7,777 194/7798 64.2% (52.7, 73.1)

a21,895 and 21,888 persons were randomized to vaccine and placebo

bCases diagnosed ≥14 days post vaccination among persons without evidence of prior SARS-CoV-2 infection

C Primary efficacy population (per protocol); includes a total of 3113 person-years of observation in vaccine group and 3089 personyears in placebo group

CI: confidence interval
Outcome 1: Symptomatic Lab-confirmed COVID-19

Studies with Unvaccinated Comparator (n=1)

21

Population Events/Vaccinea

(n/N)

Events/Placeboa

(n/N)

Vaccine efficacy 

(95% CI)

Aged ≥18 yearsb,c 173/19,514 509/19,544 66% (60%, 72%)

By geography

United Statesc,d (96.4% D614G) 51/9,119 196/9,086 74% (65%, 82%)

South Africac,d (94.5% B.1.351) 43/2,473 90/2,496 52% (30%, 67%)

Brazilc,d (69.4% P.2 lineage) 39/3,370 114/3,355 66% (51%, 77%)

Other Latin American countriesc 40/4,552 109/4,607 63% (47%, 74%)e

a. 21,895 and 21,888 persons were randomized to vaccine and placebo

b. Cases diagnosed ≥14 days post vaccination among persons without evidence of prior SARS-CoV-2 infection

c. Primary efficacy population (per protocol); includes a total of 3113 person-years of observation in vaccine group and 3089 personyears in placebo group

d. Sequencing was performed on a subset of centrally confirmed cases to determine lineage. In South Africa, 94.5% of cases were 

from B.1.351 lineage. In Brazil, 69.4% of cases represented P.2 lineage. In US, 96.4% were D614G variant, and 3% were CAL.20C.

e. Calculated using country-specific n/N supplied by sponsor. CI: confidence interval
Outcome 1: Symptomatic Lab-confirmed COVID-19

Studies with Unvaccinated Comparator (n=1)

22

Population Events/Vaccine 

(n/N)

Events/Placebo 

(n/N)

Vaccine efficacy 

(95% CI)

Co-primary outcomes

≥14 days after vaccination, 

all PCR + (any laboratory)

173/19,514 509/19,544 66.3% (59.9, 71.8)

≥14 days after vaccination, 

centrally confirmed only

116/19,514 348/19544 66.9% (59.0, 73.4)

≥28 days after vaccination, 

all PCR + (any laboratory)

113/19,306 324/19,178 65.5% (57.2, 72.4)

≥28 days after vaccination, 

centrally confirmed only

66/19,306 193/19,178 66.1% (55.0, 74.8)

CI: confidence interval
Outcome 1: Symptomatic Lab-confirmed COVID-19

Studies with Unvaccinated Comparator (n=1)

23

Population Events/Vaccine 

(n/N)

Events/Placebo 

(n/N)

Vaccine efficacy 

(95% CI)

Primary outcome

No evidence of prior infection, 

≥14 days after vaccination

173/19,514 509/19,544 66.3% (59.9%, 71.8%)

Secondary outcomes

± evidence of prior infection, 

≥14 days after vaccination

176/21,636 513/21,574 66.1% (59.7%, 71.6%)

Including milda cases 181/19,514 516/19,544 65.2% (58.7%, 70.8%)

CI: confidence interval

a. Mild COVID-19 defined as PCR-positive plus one of the symptoms in the moderate COVID-19 symptom list (excluding elevated 

heart rate) or chest congestion, runny nose, wheezing, skin rash, eye irritation/discharge. 
Evidence Table: Symptomatic Lab-confirmed COVID-19

24

Certainty assessment No of patients Effect

No of Certainty Importance

studies

Study 

design

Risk of 

bias Inconsistency Indirectness Imprecision Other considerations

Janssen COVID19 vaccine No vaccine Relative

(95% CI)

Vaccine efficacy against symptomatic COVID-19

1 RCT Not 

serious

a

Not serious Serious 

b,c,d

Not 

serious

None 173/19514 

(0.9%)

509/19544 

(2.6%)

RR 0.34

(0.29 to 0.40)

Type 2 CRITICAL

CI: confidence interval; RR: risk ratio

a. Risk of bias related to blinding of participants and personnel was present. Although participants and study staff were blinded to intervention assignments, 

they may have inferred receipt of vaccine or placebo based on reactogenicity. This was deemed unlikely to overestimate efficacy or underestimate risk of 

serious adverse events, therefore the risk of bias was rated as not serious.

b. Serious concern for indirectness was noted due to the short duration of observation in the available body of evidence. Vaccine efficacy or adverse events 

observed at a median 2-month follow-up may differ from those observed with ongoing follow-up. 

c. The effects noted are from an analysis of the per protocol population with outcomes assessed at least 14 days post vaccination, who had no evidence of prior 

SARS-CoV-2 infection, and counting cases who met the case definition with symptoms for moderate to severe COVID-19 and were PCR positive but not 

necessarily molecularly confirmed at the central laboratory. In an interim analysis using the full analysis set (persons with or without evidence of prior SARSCoV-2 infection), there were 267 cases among 21,895 persons in the vaccine arm and 621 cases among 21,888 persons in the placebo arm (RR = 0.43 (0.37 to 

0.50)). 

d. The RCT excluded persons with prior COVID-19 diagnosis, pregnant women or women planning to become pregnant within 3 months of vaccination, and 

persons who were immunocompromised due to conditions and/or treatments (participants with stable/well-controlled HIV infection were not excluded). The 

population included in the RCT may not represent all persons aged >=18 years. 
Outcome 2: Hospitalization for COVID-19

Studies with Unvaccinated Comparator (n=1)

25

 Janssen Phase III RCT (unpublished, data obtained from sponsor)

 Data on COVID-19 cases needing medical intervention was provideda

 Data on severe COVID-19: COVID-19 case with ≥1 of following:

– Clinical signs at rest indicative of severe systemic illness b

– Respiratory failure b

– Evidence of shock b

– Significant acute renal, hepatic, or neurologic dysfunction

– Admission to an intensive care unit

– Death 

a. Defined as hospitalization, ICU admission, mechanical ventilation and ECMO

b. Severe systemic illness: respiratory rate ≥30, heart rate ≥125, SpO2 ≤93% on room air at sea level or PaO2/FiO2<300 mm Hg; respiratory failure: needing 

high-flow oxygen, noninvasive ventilation, mechanical ventilation, ECMO; evidence of shock: SBP <90 mm Hg, DBP <60 mm Hg, requiring vasopressors.
Outcome 2: Hospitalization for COVID-19 

Studies with Unvaccinated Comparator (n=1)

26

Outcome Study/population Events/Vaccine 

(n/N)

Events/Placebo 

(n/N)

Vaccine efficacy 

(95% CI)

COVID-19 needing 

medical intervention a

No evidence of prior infection, 

≥14 d post vaccination

2/19,514 29/19,544 93% (71%, 98%)

COVID-19 needing 

medical intervention a

No evidence of prior infection, 

≥28 d post vaccination

0/19,306 16/19,178 100%c

Severe COVID-19, 

protocol definition b

No evidence of prior infection, 

≥14 d post vaccination

19/19,514 80/19,544 76% (58%, 88%)

Severe COVID-19, 

protocol definition b

No evidence of prior infection, 

≥28 d post vaccination

8/19,306 48/19,178 84% (54%, 97%)

a. Medical intervention defined as hospitalization, ICU admission, mechanical ventilation, and extracorporeal membrane 

oxygenation (ECMO)

b. Severe COVID-19, defined consistent FDA guidance: clinical signs at rest indicative of severe systemic illness; respiratory 

failure; evidence of shock; significant acute renal, hepatic, or neurologic dysfunction; admission to an intensive care 

unit; or death

c. With a standard continuity correction of 0.5 applied, the estimated VE (95% CI) is 97% (50%, 100%)CI: Confidence interval
Evidence Table: Hospitalization for COVID-19 

27

Certainty assessment No of patients Effect

Certainty No of Importance

studies Study 

design Risk of bias Inconsistency Indirectness Imprecision

Other 

considerations

Janssen

COVID-19 

vaccine

No vaccine Relative

(95% CI)

Vaccine efficacy against hospitalization due to COVID-19

1 RCT Not 

serious

a

Not serious Serious

b,c,d

Not serious None 2/19,514 

(0.0%)

29/19,544

(0.1%)

e

RR 

0.07 (0.02 

to 0.29)

Type 2 CRITICAL

a. Risk of bias related to blinding of participants and personnel was present. Although participants and study staff were blinded to 

intervention assignments, they may have inferred receipt of vaccine or placebo based on reactogenicity. This was deemed unlikely to 

overestimate efficacy or underestimate risk of serious adverse events, therefore the risk of bias was rated as not serious.

b. The RCT excluded persons with prior COVID-19 diagnosis, pregnant women or women planning to become pregnant within 3 months of 

vaccination, and persons who were immunocompromised due to conditions and/or treatments (participants with stable/well-controlled 

HIV infection were not excluded). The population included in the RCT may not represent all persons aged >=18 years. 

c. The effects noted are from a per protocol analysis with outcomes assessed at least 14 days post vaccination, among persons who had no 

evidence of prior SARS-CoV-2 infection.

d. Serious concern for indirectness was noted due to the short duration of observation in the available body of evidence. Vaccine efficacy 

or adverse events observed at a median 2-month follow-up may differ from those observed with ongoing follow-up.

e. Includes 15 hospitalized cases in the placebo arm identified from the Serious Adverse Events form rather than from the Medical 

Resource Utilization form. CI: Confidence interval; RR: Risk ratio
Outcome 3: All-cause Death

Studies with Unvaccinated Comparator (n=1)

28

 Janssen Phase III RCT (unpublished, data obtained from sponsor)
Outcome 3: All-cause Death

Studies with Unvaccinated Comparator (n=1)

29

Study/population Events/Vaccine 

(n/N)b

Events/Placebo 

(n/N)

Relative Risk

(95% confidence interval)

All-cause death, 

persons aged ≥18 yearsa

5/21,895 20/21,888 0.25 (0.09, 0.67)

COVID-19 related deaths, 

persons aged ≥18 years

0/21,895 7/21,888c 0.07 (0.00, 1.17)d

a. Deaths in study participants as of February 5, 2021; denominator is full analysis set.

b. Estimate and confidence interval were calculated based on number of participants.

c. One death due to COVID-19 occurred in a participant who was PCR-positive for SARS-CoV-2 at baseline.

d. Relative risk calculated using the standard continuity correction of 0.5.
Evidence Table: All-cause Death

30

a. Serious concern for indirectness was noted due to the short duration of observation in the available body of evidence. Vaccine

efficacy or adverse events observed at a median 2-month follow-up may differ from those observed with ongoing follow-up.

b. The RCT excluded persons with prior COVID-19 diagnosis, pregnant women or women planning to become pregnant within 3 months 

of vaccination, and persons who were immunocompromised due to conditions and/or treatments (participants with stable/wellcontrolled HIV infection were not excluded). The population included in the RCT may not represent all persons aged >=18 years.

Certainty assessment No of patients Effect

No of Certainty Importance

studies Study 

design Risk of bias Inconsistency Indirectness Imprecision Other 

considerations

Janssen

COVID-19 

vaccine

No vaccine Relative

(95% CI)

Vaccine efficacy against death, all cause

1 RCT Not 

serious

Not 

serious

Serious

a,b

Not 

serious

None 5/21,895

(0.0%)

20/21,888

(0.1%)

RR 0.25 (0.09 

to 0.67)

Type 2 IMPORTANT

CI: Confidence interval; RR: Risk ratio
Outcome 4: SARS-CoV-2 seroconversion to a non-spike protein

Studies with Unvaccinated Comparator (n=1)

31

 Janssen Phase III RCT (unpublished, data obtained from sponsor)

 Blood draws on trial days 1, 29, 71, then months 6, 12, 18, 24

 Asymptomatic seroconversion

– Detect N-binding antibody (non-spike protein)

• Distinguishes natural infection from vaccine-induced immunity

– Excluded persons with COVID-19 symptoms or PCR-positive test prior to 

specimen collection

 Evaluated seroconversion at 2 time points: 

– Between days 1 and 29 

– Between days 29 and 71 (less data but more relevant)
Outcome 4: SARS-CoV-2 seroconversion to a non-spike protein

Studies with Unvaccinated Comparator (n=1)

32

Study/population Events/Vaccine 

(n/N)

Events/Placebo 

(n/N)

Vaccine Efficacy

(95% confidence interval)

Asymptomatic seroconversion (Day 1 to 29)a,b 84/14,084 108/14,019 22.6% (-3.9%, 42.5%)

Asymptomatic seroconversion (Day 29 to 71)a,b 10/1346 37/1304 74.2% (47.1%, 88.6%)

a. Among participants in the serology risk set, which included persons with a non-S protein result available on Day 71 or day 29.

b. Asymptomatic SARS-CoV-2 infection is defined as (1) positive serology (non-S protein), and (2) no COVID-19 symptoms or PCR-positive test

prior to specimen collection. Seroconversion to a non-spike protein can distinguish between natural infection and vaccine-induced

immunity.
Evidence Table: SARS-CoV-2 seroconversion to a non-spike protein

33

a. The RCT excluded persons with prior COVID-19 diagnosis, pregnant women or women planning to become pregnant within 3 months of 

vaccination, and persons who were immunocompromised due to conditions and/or treatments (participants with stable/well-controlled 

HIV infection were not excluded). The population included in the RCT may not represent all persons aged >=18 years.

b. Very serious concern for indirectness was noted. Efficacy against seroconversion based on day 71 serology may not be a direct measure of 

efficacy over a relevant period of time for an emergency use authorization. Additionally, serology data were only available for a subset of 

7% of the per protocol population, likely not representing all ages, comorbidities, geographies, and exposures to circulating variants, 

raising additional concern for indirectness.

Certainty assessment No of patients Effect

No of Certainty Importance

studies Study 

design Risk of bias Inconsistency Indirectness Imprecision Other 

considerations

Janssen

COVID-19 

vaccine

No vaccine Relative

(95% CI)

Vaccine efficacy against death, all cause

1 RCT Not 

serious

Not 

serious

Very 

serious

a,b

Not 

serious

None 10/1346

(1.3%)

37/1304

(3.8%)

RR 0.26 (0.13 

to 0.52)

Type 3 IMPORTANT

CI: Confidence interval; RR: Risk ratio
Harms
Outcome 6: Serious Adverse Events

35

Studies with Unvaccinated Comparator (n=3) 

 Janssen phase III RCT (unpublished, data obtained from sponsor)

 Janssen phase II RCT (unpublished, data obtained from the sponsor)

 Janssen phase I/II RCT (Sadoff, 2021, additional data obtained from sponsor)
Janssen Phase II Randomized Controlled Trial

 Janssen phase II RCT (unpublished, data obtained from the sponsor)

 Population: healthy adults aged ≥18 to ≤55 years and ≥65 years, Germany, Spain, and the 

Netherlands

 Data evaluated:

– 276 received 1 dose of 5×1010 viral particles of Ad26.COV2.S 

– 78 received 1 dose of placebo

 Primary outcomes: Safety

– Local and systemic reactions: collected using memory aid 7 days following each dose

– Adverse events: unsolicited AEs during 28 day follow up period

– SAEs for duration of study period

36
Janssen Phase I/II Randomized Controlled Trial

 Janssen phase I/II RCT (Sadoff, 2021, additional data obtained from sponsor)

 Population: healthy adults aged ≥18 years, United States and Belgium

 Data evaluated:

– 323 received 1 dose of 5×1010 viral particles of Ad26.COV2.S

• 162 aged 18 to 55 years

• 161 aged ≥65 years

– 163 received 1 dose of placebo

 Primary outcomes: Safety

– Local and systemic reactions: collected using memory aid 7 days following each dose

– Adverse events: unsolicited AEs during 28 day follow up period

– SAEs for duration of study period

37
Outcome 6: Serious Adverse Events (SAE)a

Studies with Unvaccinated Comparator (n=3)

38

Study/populationa Events/Vaccine 

(n/N)

% SAE

Vaccine

Events/Placebo

(n/N)

% SAE

Placebo

Associated 

with 

vaccination

Janssen, phase III, 

unpublishedb

83/21,895 0.4% 96/21,888 0.4% 3c

Janssen, phase II, 

unpublishedd

0/276 0.0% 0/78 0.0%

Sadoff 2021; Janssen, 

phase I/II, unpublished

1/323 0.3% 2/163 1.2%

a. Excludes COVID-19 related SAEs

b. Proportion of participants who reported at least one SAE from dose 1 to primary analysis cutoff date (January 22, 2021).

c. 9 participants (7 in the vaccine and 2 in the placebo group) were deemed by blinded investigators to have serious adverse events

to be related or possibly related to vaccination. Among the 7 vaccine participants, these included: pericarditis, facial paralysis, 

injection site pain, Guillain-Barre Syndrome, systemic reactogenicity, and hypersensitivity. Through further investigation by the 

FDA, only 3 were classified as related to vaccination: injection site pain, hypersensitivity, and systemic reactogenicity.

d. Proportion of participants who reported at least one SAE from dose 1 to primary analysis cutoff date (January 11, 2021).
Outcome 6: Serious Adverse Events (SAE): Summary of nonfatal vaccine-related SAEs from phase III trial (n=3)

Age/sex Comorbidities SAE Onset postvaccinationDuration 

(days)

Severity

42/M No Hypersensitivity 3 31a Grade 3

30/M No Injection site pain 1 75a Grade 3

35/M Yes Systemic reactogenicityb 2 3 Grade 3

a. Ongoing at the time of report

b. Hospitalized due to exacerbated generalized weakness, originally suspected for demyelinating disorder which was subsequently discarded

39
Evidence Table: Serious Adverse Events

40

a. Data were pooled from one Phase III trial, one Phase I/II trial, and one Phase II trial.

b. Risk of bias related to blinding of participants and personnel was present. Although participants and study staff were blinded to 

intervention assignments, they may have inferred receipt of vaccine or placebo based on reactogenicity. This was deemed unlikely to 

overestimate efficacy or underestimate risk of serious adverse events, therefore the risk of bias was rated as not serious.

c. The RCT excluded persons with prior COVID-19 diagnosis, pregnant women or women planning to become pregnant within 3 months of 

vaccination, and persons who were immunocompromised due to conditions and/or treatments (participants with stable/well-controlled HIV 

infection were not excluded). The population included in the RCT may not represent all persons aged >=18 years.

d. Serious concern for indirectness was noted due to the short duration of observation in the available body of evidence. Vaccine efficacy or 

adverse events observed at a median 2-month follow-up may differ from those observed with ongoing follow-up.

Certainty assessment No of patients Effect

Certainty Importance

No of 

studies

Study 

design

Risk of 

bias Inconsistency Indirectness Imprecision Other considerations

Janssen 

COVID-19 

vaccine

No vaccine Relative

(95% CI)

Serious adverse events

3

a

RCT not 

serious

b

not serious Serious

c,d

not serious none 84/22494 

(0.4%)

98/22129 

(0.4%)

RR 0.85

(0.63 to 1.13)

Type 2 CRITICAL

CI: Confidence interval; RR: Risk ratio
Outcome 7: Reactogenicity, Severe (Grade ≥3)a

41

Studies with Unvaccinated Comparator (n=3)

 Janssen phase III RCT (unpublished, data obtained from sponsor)

 Janssen phase II RCT (unpublished, data obtained from sponsor)

 Janssen phase I/II RCT (Sadoff, 2021, additional data obtained from sponsor)

aGrade 3: prevents daily routine activity. Grade 4: requires emergency room visit or hospitalization. 
Outcome 7: Reactogenicity, Severe (Grade ≥3)

Definitions

42

 All trials collected solicited events through electronic diaries for 7 days following vaccination

 Local reactions (pain at injection site, redness, swelling)

– Grade 3: pain at injection site that prevents daily activity or use of narcotic pain reliever; redness 

> 10 cm; and swelling > 10 cm 

– Grade 4: hospitalization for severe pain at the injection site, necrosis (redness and swelling 

categories) or exfoliative dermatitis (redness category only).

 Systemic events (fever, nausea, headache, fatigue, muscle pain)

– Grade 3: fever >38.9°C to 40.0°C , nausea, fatigue, headache, or muscle pain that prevents daily 

activity or use of narcotic pain reliever.

– Grade 4: fever >40.0°C, nausea, fatigue, headache, or muscle pain that require hospitalization or 

prevents basic self care.
Outcome 7: Reactogenicity, Severe (Grade ≥3)a,b

Studies with and without unvaccinated comparator (n=3)

43

Study/population Events/Vaccine 

(n/N)

% Vaccine Events/Placebo

(n/N)

% Placebo

Janssen, phase III, 

unpublished

75/3356 2.2% 25/3380 0.7%

Janssen, phase II, 

unpublished

8/276 2.9% 0/78 0%

Sadoff 2021; Janssen, 

phase I/IIc 

16/323 5.0% 0/163 0%

a. Grade 3: prevents daily routine activity or requires use of a narcotic pain reliever. Grade 4: requires 

hospitalization or prevents basic self care. There were no grade 4 adverse reactions reported.

b. Includes local and systemic events, grade ≥3.

c. Additional data provided by sponsor

Note: GRADE was conducted considering pooled phase I/II, II and III available data.
Evidence Table: Reactogenicity, Severe (Grade ≥3)

44

a. Data were pooled from one Phase III trial, one Phase I/II trial, and one Phase II trial.

b. The RCT excluded persons with prior COVID-19 diagnosis, pregnant women or women planning to become pregnant 

within 3 months of vaccination, and persons who were immunocompromised due to conditions and/or treatments 

(participants with stable/well-controlled HIV infection were not excluded). The population included in the RCT may 

not represent all persons aged >=18 years.

c. Differences in demographic composition were noted between the full analysis set (FAS) and the safety subset used 

for evaluation of reactogenicity. Notably, compared to the FAS, the safety subset included a higher proportion of 

White race (83.4% vs. 58.7%), a higher proportion from Brazil (38.5% vs. 16.6%), and a lower proportion who were 

seropositive for SARS-CoV-2 at baseline (4.5% vs. 9.6%).

Certainty assessment No of patients Effect

No of Certainty Importance

studies

Study 

design Risk of bias Inconsistency Indirectness Imprecision Other considerations

Janssen 

COVID-19 

vaccine

No vaccine Relative

(95% CI)

Reactogenicity, severe (grade ≥3)

3

a

RCT not serious not serious not serious 

b,c

not serious none 99/3955 

(2.5%)

25/3621 

(0.7%)

RR 3.42

(2.20 to 5.31)

Type 1 IMPORTANT

CI: Confidence interval; RR: Risk ratio
Summary of GRADE

45

OutcomeImportance Design

(# of studies)

FindingsEvidence

type

Benefits

Symptomatic labconfirmed COVID-19

Critical RCT (1) Janssen COVID-19 vaccine is effective in preventing symptomatic COVID192

Hospitalization due to 

COVID-19

Critical RCT (1) Janssen COVID-19 vaccine prevents COVID-19-resulting in hospitalization 2

All-cause Death Important RCT (1) Janssen COVID-19 vaccine is associated with a lower risk of both allcause death and death due to COVID-19 2

SARS-CoV-2 

seroconversion Important RCT (1)

Data from day 71 serology indicates that Janssen COVID-19 vaccine 

prevents seroconversion during the available follow-up period; data 

support an effect on prevention of asymptomatic infection

3

Asymptomatic SARSCoV-2 infection Important No studies No systematically collected PCR data are available to develop an 

estimate for this outcome ND

Harms

Serious adverse eventsCritical RCT (3) SAEs were balanced between vaccine and placebo arms. 3 participants 

had SAEs judged by FDA to be related to study vaccine 2

Reactogenicity Important RCT (3) Severe reactions were more common in vaccinated; any grade ≥3 

reaction was reported by 2.5% of vaccinated vs. 0.7% of placebo 1

Evidence type: 1=high; 2=moderate; 3=low; 4=very low; ND, no data
Conclusion – GRADE for Janssen COVID-19 vaccine

 Phase III RCT conducted on three continents during a time of high COVID-19 

incidence while viral variants were emerging. 

 Vaccine efficacy estimates: 66% for symptomatic laboratory-confirmed COVID19, 93% for hospitalization due to COVID-19, 75% for all-cause death, and 74%

for asymptomatic seroconversion. 

 No deaths due to COVID-19 were identified among vaccine recipients, and 7 

deaths due to COVID-19 were identified among placebo recipients.

 No serious safety concerns identified; balanced reports of serious adverse events 

between arms (0.4% each).

 Grade ≥3 local or systemic reactions more common among vaccine than 

placebo recipients, and were reported by <3% of vaccinated subjects.

 Certainty for all critical benefits and harms was type 2 (moderate).

46
Acknowledgements

 CDC Data team

– Amy Blain

– Jessica Leung

– Megan Wallace

 Kathleen Dooling

 Sara Oliver

 Steven Hadler

 Heather Scobie

 Jack Gersten

 Eddie Shanley

 Karen Broder

 Rebecca Morgan

 Doug Campos-Outcalt

 ACIP COVID-19 Work Group

47
For more information, contact CDC

1-800-CDC-INFO (232-4636)

TTY: 1-888-232-6348 www.cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the 

official position of the Centers for Disease Control and Prevention.

Thank you
Comparability of GRADE across trials: indirectness

 For the mRNA vaccines, we were concerned about indirectness for several 

outcomes due to the median 2-month follow-up

 Certainty in estimates for hospitalization, deaths, and serious adverse 

events were graded down 1 level for indirectness

 For efficacy against the primary endpoint of symptomatic COVID-19, we did 

not grade down for indirectness in consideration of the very strong and 

precise estimate

– Was not plausible that the efficacy would dip lower than 50% during 

time period of EUA

– This exception is not being made for Janssen with 66% overall efficacy

49
Measures used in COVID-19 vaccine trials: 

reactogenicity

Solicited symptom Janssen Moderna Pfizer

Local

Redness X X X

Swelling X X X

Pain, injection site X X X

Axillary swelling/tenderness X

Systemic

Fever X X X

Fatigue X X X

Headache X X X

Chills X X

Nausea X X X

Diarrhea X

Myalgia X X X

Arthralgia X X

50
Measures used in COVID-19 vaccine trials:

symptoms required for symptomatic COVID-19

Janssen ("moderate to severe/critical") Moderna Pfizer

>=1 of: >=1 of: >=1 of:

Respiratory >=20

Abnormal pulse ox but >93%

Clinical/Radiological pneumonia Clinical/Radiological pneumonia

DVT

Shortness of breath/difficulty breathing Shortness of breath Shortness of breath

Cough

OR OR

>=2 of: >=2 of:

Fever >=38C Fever >=38C Fever >=38C

Heart rate >=90

Shaking chills Chills Chills

Sore throat Sore throat Sore throat

Cough Cough

Malaise

Headache Headache

Myalgia Myalgia Muscle pain

GI sx (N/V/D/abdominal pain) Diarrhea or vomiting

Olfactory/taste disorder Olfactory/taste disorder Loss of taste or smell

Red/bruised toes

51
Measures used in COVID-19 vaccine trials:

expanded symptom list (more sensitive case definitions)

Janssen Moderna Pfizer 

Mild COVID-19 Expanded CDC symptom list Expanded CDC symptom list

At least 1 symptom from mod/severe list* or: At least 1 from primary symptom list or : At least 1 of above or expanded CDC symptom list:

Chest congestion Fatigue Fatigue

Runny nose Nasal congestion Nasal congestion/runny nose

Wheezing Nausea/vomiting

Skin rash

Eye irritation/discharge Headache

"FDA harmonized list" Nausea

fever or chills

cough

shortness of breath/difficulty breathing

fatigue

muscle or body aches

headache

loss of taste or smell

sore throat

congestion

runny nose

nausea or vomiting

diarrhea

Diarrhea

*May exclude elevated heart rate

Note: Janssen’s “FDA harmonized list” matches Moderna and Pfizer “Expanded CDC symptom list” 52
Measures used in COVID-19 vaccine trials: 

hospitalization due to COVID-19

Janssen

Hospitalization, ICU admission, 

mechanical ventilation, or death

Moderna

Hospitalized subset of cases meeting FDA 

definition of severe COVID-19

Pfizer

Hospitalization, admission to the ICU, 

intubation or mechanical ventilation, or 

death in a participant who met the 

secondary case definition using expanded 

CDC symptom list*

*some did not meet FDA definition of severe COVID-19 53
Measures used in COVID-19 vaccine trials: severe 

COVID-19 (FDA harmonized)

Janssen

"Severe/Critical"

1 of the following:

Resp rate >=30 bpm

Heart rate >=125 bpm

SpO2 93%

PaO2/FiOs <300 mmHg

Respiratory failure (needing high-flow O2, noninvasive ventilation, mechanical ventilation, or

ECMCO)

Shock (SBP <90, DBP <60, or requiring 

vasopressors)

Significant acute renal, hepatic, or neurologic 

dysfunction

Admission to ICU

Death

Moderna

"Severe COVID-19"

any of the following:

Resp rate >=30 bpm

Heart rate >=125 bpm

SpO2 93%

PaO2/FiOs <300 mmHg

Respiratory failure or ARDS (needing high-flow

O2, non-invasive ventilation, mechanical 

ventilation, or ECMO)

Shock (SBP <90, DBP <60, or requiring 

vasopressors)

Significant acute renal, hepatic, or neurologic 

dysfunction

Admission to ICU

Death

Pfizer

"Severe COVID-19"

at least 1 of the following:

Resp rate >=30 bpm

Heart rate >=125 bpm

SpO2 93%

PaO2/FiOs <300 mmHg

Respiratory failure (needing high-flow O2, noninvasive ventilation, mechanical ventilation, or 

ECMCO)

Shock (SBP <90, DBP <60, or requiring 

vasopressors)

Significant acute renal, hepatic, or neurologic 

dysfunction

Admission to ICU

Death

54
COVID-19 

incidence in the 

Placebo Group, 

Seronegative 

Participants, 

Phase III trial, 

FAS

The observed decrease in COVID-19 incidence depicted above, after 7 January 2021, may be partially due to 

operational reasons: operational time from sampling to PCR confirmation in the central laboratory was estimated to 

be on average 14 days, with a longer confirmation time in some countries in the Latin America region and South 

Africa. Therefore some cases after the database cut-off may be pending.

Source: VRBPAC briefing 

document, Figure 10

55
Follow-up time, by age 

and comorbidities

Source: FDA briefing document, page 18