# International Survey to Establish Prioritized Outcomes for Trials in People With Coronavirus Disease 2019

Nicole Evangelidis, Allison Tong, Martin Howell, Armando Teixeira-Pinto, Julian H. Elliott, Luciano Cesar Azevedo, Andrew Bersten, Lilia Cervantes, Derek P. Chew, Sally Crowe, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Nicole Evangelidis, Allison Tong, Martin Howell, Armando Teixeira-Pinto, Julian H. Elliott, et al. International Survey to Establish Prioritized Outcomes for Trials in People With Coronavirus Disease 2019. Critical Care Medicine, 2020, 48 (11), pp.1612-1621. 10.1097/ccm.00000000000004584. hal-03324313

HAL Id: hal-03324313

https://hal.univ-lille.fr/hal-03324313

Submitted on 23 Aug 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<sup>27</sup>Section of Infectious Diseases Department of Medicine, National School of Tropical Medicine Baylor College of Medicine, Houston, TX.

<sup>28</sup>Evidence-based Medicine center, Tianjin University of Traditional Chinese Medicine, Tianjin, China.

<sup>29</sup>Faculty of Medicine, University of Queensland, Brisbane, Old, Australia.

<sup>30</sup>Department of Infectious Diseases, Perth Children's Hospital, Perth, WA, Australia.

**Objectives:** There are over 4,000 trials conducted in people with coronavirus disease 2019. However, the variability of outcomes and the omission of patient-centered outcomes may diminish the impact of these trials on decision-making. The aim of this study was to generate a consensus-based, prioritized list of outcomes for coronavirus disease 2019 trials.

**Design:** In an online survey conducted in English, Chinese, Italian, Portuguese, and Spanish languages, adults with coronavirus disease 2019, their family members, health professionals, and the general public rated the importance of outcomes using a 9-point Likert scale (7–9, critical importance) and completed a Best-Worst Scale to estimate relative importance. Participant comments were analyzed thematically.

**Setting:** International.

**Subjects:** Adults 18 years old and over with confirmed or suspected coronavirus disease 2019, their family members, members of the general public, and health professionals (including clinicians, policy makers, regulators, funders, and researchers).

**Interventions:** None.

**Measurements:** None.

**Main Results:** In total, 9,289 participants from 111 countries (776 people with coronavirus disease 2019 or family members, 4,882 health professionals, and 3,631 members of the public) completed the survey. The four outcomes of highest priority for all three groups were: mortality, respiratory failure, pneumonia, and organ failure. Lung function, lung scarring, sepsis, shortness of breath, and oxygen level in the blood were common to the top 10 outcomes across all three groups (mean > 7.5, median ≥ 8, and > 70% of respondents rated the outcome as critically important). Patients/family members rated fatigue, anxiety, chest pain, muscle pain, gastrointestinal problems, and cardiovascular disease higher than health professionals. Four themes underpinned prioritization: fear of life-threatening, debilitating, and permanent consequences; addressing knowledge gaps; enabling preparedness and planning; and tolerable or infrequent outcomes.

**Conclusions:** Life-threatening respiratory and other organ outcomes were consistently highly prioritized by all stakeholder groups. Patients/family members gave higher priority to many patient-reported outcomes compared with health professionals. (Crit Care Med 2020; 48:1612–1621)

**Key Words:** clinical trial; coronavirus; critical care; infection; patients; sepsis

As of July 8, 2020, over 11.6 million cases of coronavirus disease 2019 (COVID-19) have been reported, with more than 539,000 deaths in 216 countries (1). The risk of death in patients hospitalized for COVID-19 is estimated to

range from 12% to 33% (2–7). In a study of 5,700 hospitalized patients in the United States, 14% required ICU admission, and 12% received invasive mechanical ventilation (8). Patients with COVID-19 experience fatigue, cough, and dyspnea, which impair daily functioning (4, 9–11), and mental health sequelae such as depression and anxiety are of concern (12–14).

Substantial resources have been invested in clinical trials in COVID-19 (15), with more than 4,000 trials registered on the World Health Organization (WHO) International Clinical Trials Registry Platform. However, the outcomes included are inconsistent, and patient-reported outcomes (16) are infrequently measured. Serious clinical outcomes (e.g., mortality, respiratory failure) are not always reported, and patient-reported outcomes (e.g., cough) are rarely included (17–19). These problems can diminish the relevance of the evidence for decision-making by patients, health professionals, and regulators.

Three core outcome sets for COVID-19 have been developed, with respiratory failure and mortality in hospitalized patients common to all (20–22). However, only one of these initiatives involved consultation with patients and the public (n = 27), all of whom were from China. The global COVID-19-Core Outcomes Set (COS) initiative convened in March 2020, to bring together people with COVID-19 and family members, members of the general public, and health professionals, to establish consensus-based core outcomes for trials in people with confirmed or suspected COVID-19 across the full spectrum of disease, in a two-step process. First this study, and then an online series of workshops. The aim of this study was to identify a prioritized list of outcomes for trials in COVID-19 and to explain the reasons behind the prioritization decisions.

## MATERIALS AND METHODS

We used the Core Outcome Set Standards for Reporting Statement (23).

### Study Design

We followed the Core Outcome Measures in Effectiveness Trials (COMET) methodological framework, which involves a Delphi survey and a consensus workshop (24). The consensus workshops will be published separately. This international survey was initially designed to be a two-round Delphi survey to generate consensus (24). Consensus was achieved in round 1, making round 2 unnecessary. Participants prioritized outcomes for trials in people with confirmed or suspected COVID-19. The survey was conducted in English, Chinese, Italian, Spanish, and Portuguese.

### Participant Selection and Recruitment

People 18 years old and over with confirmed or suspected COVID-19 and their family members, members of the public, and health professionals (including clinicians, researchers, and policy makers) were eligible. We used multiple recruitment strategies to be as broadly inclusive and diverse as possible. Respondents were recruited through the Steering Committee and investigator networks and professional and consumer
<table><caption>TABLE 1. Participant Characteristics (n = 9,289)</caption><thead><tr><th>Characteristic</th><th>Patients/Family Members, n = 776, n (%)</th><th>Health Professionals, n = 4,882, n (%)</th><th>Public, n = 3,631, n (%)</th></tr></thead><tbody><tr><td>Participant type<sup>a</sup></td><td></td><td></td><td></td></tr><tr><td>Person with suspected COVID-19</td><td>420 (54)</td><td>—</td><td>—</td></tr><tr><td>Person with confirmed COVID-19</td><td>170 (22)</td><td>—</td><td>—</td></tr><tr><td>Family/caregiver</td><td>312 (40)</td><td>—</td><td>—</td></tr><tr><td>Clinician</td><td>215 (28)</td><td>3,295 (67)</td><td>—</td></tr><tr><td>Researcher</td><td>123 (16)</td><td>1,480 (30)</td><td>—</td></tr><tr><td>Policy maker</td><td>23 (3)</td><td>69 (1)</td><td>—</td></tr><tr><td>Other health professional</td><td>41 (5)</td><td>813 (17)</td><td>—</td></tr><tr><td>Gender<sup>b</sup></td><td></td><td></td><td></td></tr><tr><td>Male</td><td>253 (33)</td><td>1,578 (32)</td><td>1,221 (34)</td></tr><tr><td>Female</td><td>510 (66)</td><td>3,265 (67)</td><td>2,381 (66)</td></tr><tr><td>Age group (yr)</td><td></td><td></td><td></td></tr><tr><td>18–29</td><td>131 (17)</td><td>1,190 (24)</td><td>763 (21)</td></tr><tr><td>30–39</td><td>203 (26)</td><td>1,441 (30)</td><td>629 (17)</td></tr><tr><td>40–49</td><td>213 (27)</td><td>1,036 (21)</td><td>755 (21)</td></tr><tr><td>50–59</td><td>133 (17)</td><td>772 (16)</td><td>590 (16)</td></tr><tr><td>60–69</td><td>74 (10)</td><td>370 (8)</td><td>599 (16)</td></tr><tr><td>70 and over</td><td>22 (3)</td><td>73 (1)</td><td>295 (8)</td></tr><tr><td>Currently in self-isolation/quarantine</td><td></td><td></td><td></td></tr><tr><td>Yes</td><td>514 (66)</td><td>904 (19)</td><td>1,479 (41)</td></tr><tr><td>No</td><td>262 (34)</td><td>3,978 (81)</td><td>2,152 (59)</td></tr><tr><td>Language of survey completion</td><td></td><td></td><td></td></tr><tr><td>English</td><td>584 (75)</td><td>3,004 (62)</td><td>2,339 (64)</td></tr><tr><td>Chinese</td><td>49 (6)</td><td>1,075 (22)</td><td>356 (10)</td></tr><tr><td>Italian</td><td>59 (8)</td><td>253 (5)</td><td>362 (10)</td></tr><tr><td>Spanish</td><td>47 (6)</td><td>298 (6)</td><td>307 (8)</td></tr><tr><td>Portuguese</td><td>37 (5)</td><td>252 (5)</td><td>267 (7)</td></tr><tr><td>Preexisting/health conditions<sup>a</sup></td><td></td><td></td><td></td></tr><tr><td>High blood pressure/cardiovascular/heart conditions</td><td>150 (19)</td><td>—</td><td>718 (20)</td></tr><tr><td>Respiratory condition (e.g., asthma)</td><td>123 (16)</td><td>—</td><td>382 (11)</td></tr><tr><td>Diabetes/obesity</td><td>120 (15)</td><td>—</td><td>477 (13)</td></tr><tr><td>Other/s</td><td>169 (22)</td><td>—</td><td>964 (27)</td></tr><tr><td>None</td><td>408 (53)</td><td>—</td><td>1,972 (54)</td></tr></tbody></table>

(Continued)
<table><caption>TABLE 1. (Continued). Participant Characteristics (n = 9,289)</caption><thead><tr><th>Characteristic</th><th>Patients/Family Members, n = 776, n (%)</th><th>Health Professionals, n = 4,882, n (%)</th><th>Public, n = 3,631, n (%)</th></tr></thead><tbody><tr><td>Symptoms<sup>a</sup> (suspected or confirmed patients only, n = 590)</td><td></td><td></td><td></td></tr><tr><td>Fatigue</td><td>393 (67)</td><td>—</td><td>—</td></tr><tr><td>Cough</td><td>361 (61)</td><td>—</td><td>—</td></tr><tr><td>Fever</td><td>312 (53)</td><td>—</td><td>—</td></tr><tr><td>Shortness of breath</td><td>243 (41)</td><td>—</td><td>—</td></tr><tr><td>Loss of taste and/or smell</td><td>228 (39)</td><td>—</td><td>—</td></tr><tr><td>Nausea, vomiting, diarrhea</td><td>202 (34)</td><td>—</td><td>—</td></tr><tr><td>Chest pain</td><td>201 (34)</td><td>—</td><td>—</td></tr><tr><td>Other</td><td>223 (38)</td><td>—</td><td>—</td></tr><tr><td>None/do not know</td><td>67 (11)</td><td>—</td><td>—</td></tr><tr><td>Complications<sup>a</sup> (suspected or confirmed patients only, n = 590)</td><td></td><td></td><td></td></tr><tr><td>Pneumonia</td><td>71 (12)</td><td>—</td><td>—</td></tr><tr><td>Organ failure</td><td>21 (4)</td><td>—</td><td>—</td></tr><tr><td>Sepsis</td><td>20 (3)</td><td>—</td><td>—</td></tr><tr><td>None</td><td>451 (76)</td><td>—</td><td>—</td></tr><tr><td>Other</td><td>58 (10)</td><td>—</td><td>—</td></tr><tr><td>Treatment<sup>a</sup> (suspected or confirmed patients only, n = 590)</td><td></td><td></td><td></td></tr><tr><td>Antiviral therapy</td><td>50 (8)</td><td>—</td><td>—</td></tr><tr><td>Oxygen</td><td>50 (8)</td><td>—</td><td>—</td></tr><tr><td>Hydroxychloroquine/chloroquine</td><td>41 (7)</td><td>—</td><td>—</td></tr><tr><td>Mechanical ventilation</td><td>13 (2)</td><td>—</td><td>—</td></tr><tr><td>Other/s</td><td>115 (19)</td><td>—</td><td>—</td></tr><tr><td>None/do not know</td><td>394 (67)</td><td>—</td><td>—</td></tr><tr><td>Country</td><td></td><td></td><td></td></tr><tr><td>United Kingdom</td><td>268 (35)</td><td>753 (15)</td><td>772 (21)</td></tr><tr><td>China</td><td>51 (7)</td><td>1,113 (23)</td><td>370 (10)</td></tr><tr><td>United States</td><td>146 (19)</td><td>461 (9)</td><td>787 (22)</td></tr><tr><td>Australia</td><td>50 (6)</td><td>765 (16)</td><td>390 (11)</td></tr><tr><td>Italy</td><td>58 (7)</td><td>280 (6)</td><td>355 (10)</td></tr><tr><td>Other<sup>c</sup></td><td>206 (27)</td><td>1,532 (31)</td><td>975 (27)</td></tr></tbody></table>

COVID-19 = coronavirus disease 2019.

Dashes indicate data not applicable.

<footnote><sup>a</sup>Percentages may not add up to 100 due to multiple selections.</footnote>

<footnote><sup>b</sup>Prefer not to say: patients /family members 13 (2%), health professionals 39 (1%), and public 29 (1%).</footnote>

<footnote><sup>c</sup>Other includes 106 countries in descending order of total number of participants: Portugal, Canada, Brazil, Mexico, Spain, Chile, France, Netherlands, Ireland, Germany, New Zealand, India, Switzerland, South Africa, Argentina, Belgium, Iceland, Sweden, Nigeria, Colombia, Costa Rica, Singapore, United Arab Emirates, Japan, Guatemala, Angola, Korea, Republic of, Saudi Arabia, Lithuania, Uganda, Norway, Denmark, Pakistan, Greece, Poland, Uruguay, Austria, Dominican Republic, El Salvador, Malaysia, Turkey, Cameroon, Kenya, Peru, Panama, Egypt, Bangladesh, Ecuador, Islamic Republic of Iran, Iraq, Israel, Gambia, Indonesia, Malawi, Philippines, Serbia, Zimbabwe, Albania, Bulgaria, Ethiopia, Jordan, Kuwait, Malta, Paraguay, Russian Federation, Sri Lanka, Côte d'Ivoire, Estonia, Honduras, Qatar, Senegal, Trinidad and Tobago, Tunisia, Ukraine, United Republic of Tanzania, Bolivarian Republic of Venezuela, Viet Nam, Marshall Islands, Yemen, Afghanistan, Andorra, Antigua and Barbuda, Barbados, Belize, Botswana, Cape Verde, Czech Republic, Finland, Georgia, Hungary, Lebanon, Liechtenstein, Luxembourg, Mozambique, Nepal, Oman, Puerto Rico, Romania, Rwanda, Sao Tome and Principe, Slovakia, Slovenia, Sudan, Thailand, Timor-Leste, and Zambia.</footnote>
<figure><img src="image_3.png" /><figcaption>**Figure 1.** Mean scores of patients/family members, health professionals, and the general public (ordered by patient/family scores). Total sample size differs for each outcome. Death, cardiovascular disease, shortness of breath, chest pain, and cough (n = 9,289); fever, pneumonia, hospitalization, depression, and impact on family (n = 9,072); gastrointestinal problems, fatigue, life participation, muscle pain, and respiratory failure (n = 8,907); sepsis/septic shock, taste and smell, anxiety, lung function, and recovery (n = 8,789); and oxygen level in the blood, viral load or clearance, lung scarring (fibrosis), hospital-acquired infection (HAI), and organ failure n = 8,653.</figcaption></figure>

organizations using email, social media, and a market research company. The University of Sydney provided ethics approval.

## Data Collection

**Selection of Outcome Domains.** We included outcome domains from systematic reviews and published core outcome sets for COVID-19, after discussion among the Steering Committee, which included patients and members of the public (20–22, 25) (eTable 1, Supplemental Digital Content 1, http://links.lww.com/CCM/F730) The order of outcomes was randomized. The survey was administered online using Qualtrics software (SAP, Provo, UT). The survey was open for 16 days from March 31, 2020, to April 16, 2020.

**Prioritization of Outcomes.** Participants rated the importance of each of the 25 outcome domains using the Grading of Recommendations Assessment, Development, and Evaluation process 9-point Likert scale (26). Scores 1–3 indicated “limited importance,” 4–6 indicated “important but not critical,” and 7–9 indicated “critical importance.” Participants could enter comments in free-text boxes and suggest new outcomes.

Participants also completed a Best-Worst Scale survey comprising of five-choice sets, including six of the possible 25 outcomes. The Best-Worst Scale survey is a well-established method for eliciting relative preferences. The outcomes included in each set were determined using a balanced, incomplete block design (27, 28). Participants selected the most important and least important outcome from each block of outcomes presented.

## Data Analysis

**Quantitative Analysis.** We calculated the mean score, median, and proportion of participants who rated the outcome from 7 to 9 (critically important) for each outcome. We calculated the scores separately for patients/family, members of the public, and health professionals. We compared the “public” and “health professional” group with the “patients/family” group using mean differences. We did not rely on the normality assumption and used nonparametric tests to compare the importance of the outcome given by the three groups. The overall differences were compared using Kruskal-Wallis tests. If significant differences were found, we performed the pairwise comparisons
<figure><img src="image_4.png" /><figcaption>**Figure 2.** Difference in mean scores among patients/family members, public, and health professionals. Patients/family members are the reference group. The distance between each value represents the difference in mean scores between groups (patients/family members—health professionals; patients/family members—public). For example, patients/family members rated chest pain higher than health professionals (mean score difference of 0.45). Health professionals rated fever higher than patients/family members (mean score difference of -0.27). $diff = \text{difference}, \text{HAI} = \text{hospital-acquired infection}$.</figcaption></figure>

we used a scale of 1 (least important) to 9 (most important). Statistical analyses were conducted using SPSS (Version 25.0; IBM SPSS Statistics for Windows, Armonk, NY) and NLOGIT 6 (Econometric Software, Plainview, NY). A *p* value of less than 0.05 was considered statistically significant.

**Definition of Consensus.** The thresholds for consensus could not be defined a priori because the distribution of scores was not known prior to data collection. After reviewing the results, we set the threshold to identify the top 10 outcome domains indicated as critically important by all three stakeholder groups on the Likert scale. This was based on the patient/family, public, and health professional groups, each having mean greater than 7.5, median greater than or equal to 8, and greater than 70% of each group rating the outcome as “critically important.” The 10 outcomes of highest priority were used to select the proposed 3 to 5 core outcomes to discuss at the consensus workshop.

**Qualitative Analysis.** Survey comments were imported into Hyper RESEARCH (ResearchWare Inc, version 3.7; Randolph, MA) software for data analysis. Investigators (N.E., A.T.) used thematic analysis to code the text and inductively identify themes to explain the reasons for prioritizing outcomes among stakeholder groups.

using Mann-Whitney *U* tests with Bonferroni correction to adjust for multiple comparisons. The relative importance score was derived using a multinomial logistic regression model. Utility functions containing all outcomes and interaction terms for participant characteristics were constructed for the Best-Worst Survey. Following this approach, the mean regression coefficients of this function provided the relative importance scores for each outcome (27). The regression coefficients have the same underlying scale; therefore, for ease of interpretation,

# RESULTS

## Participant Characteristics

In total, 9,289 participants from 111 countries completed the survey, of whom 776 (8%) were patients with confirmed or suspected COVID-19 or their family members, 4,882 (53%) were health professionals, and 3,631 (39%) were members of the public. The participant characteristics are provided in **Table 1**. The 776 patients/family members were from 40 countries. Seventy-one of the 590 patients (12%) with suspected or
<figure><img src="image_5.png" /><figcaption>**Figure 3.** Mean relative importance scores of patients/family members, public, and health professionals based on the Best-Worst Scale, ordered by the mean importance scores of patients/family members (*error bars* are 95% CI). HAI = hospital-acquired infection.</figcaption></figure>

confirmed COVID-19 reported having pneumonia, 21 (4%) reported organ failure, and 20 (3%) sepsis. Fifty (8%) were treated with oxygen and 13 (2%) required mechanical ventilation. Of the 4,882 health professionals (from 95 countries), 2,249 (46%) were physicians, 1,218 were researchers (25%), 866 (18%) were nurses, 78 (2%) were psychologists, 55 (1%) were policymakers, 48 (1%) were social workers, 37 (1%) were industry representatives, and 1,153 (24%) were “other health professionals.” Among the physicians, 364 (16%) specialized in internal medicine, 320 (14%) were intensive/critical care specialists, 155 (7%) were general practitioners, 103 (5%) were emergency physicians, 101 (4%) were infectious disease specialists, 98 (4%) were pulmonary/respiratory specialists, and 97 (4%) were public health physicians, and 1,011 (45%) were “other specialist physicians.” The 3,631 members of the general public were from 76 countries.

## Absolute Importance Scores

The top four outcomes with the highest mean score for patients/family members, health professionals, and the public were the same (Fig. 1; eTables 2 and 3, Supplemental Digital Content 1, http://links.lww.com/CCM/F730). This included death (patients/family: mean 8.2, health professionals: mean 8.4, and public: mean 8.3); respiratory failure (patients/family: 8.2, health professionals: 8.4, and public: 8.4); pneumonia (patients/family: 7.9, health professionals: 8.1, and public: 8.2); and organ failure (patients/family: 7.9, health professionals:

8.1, and public: 8.1). Lung function, lung scarring, sepsis, shortness of breath, and oxygen level in the blood were in the top ten outcomes for each stakeholder group.

Differences in mean scores among patients/family, health professionals, and the public are shown in **Figure 2**. Patients/family rated the following six outcomes higher than health professionals on the Likert scale: chest pain (absolute mean difference, 0.5; *p* < 0.001), fatigue (0.3; *p* < 0.001), muscle pain (0.3; *p* = 0.003), gastrointestinal problems (0.3; *p* = 0.005), anxiety (0.2; *p* = 0.034), and cardiovascular disease (0.2; *p* < 0.001). Health professionals rated the following four outcomes higher than patients/family on the Likert scale: fever (0.3; *p* < 0.001), shortness of breath (0.3; *p* < 0.001), respiratory failure (0.2; *p* = 0.002), and cough (0.2; *p* = 0.042).

## Relative Preferences—Best-Worst Scale

Patients/family members considered death to be the most important, followed by respiratory failure, organ failure, lung function, and sepsis (Fig. 3). The differences between death and respiratory failure were small with preference scores of 7.5 (95% CI, 7.2–7.9) and 6.6 (95% CI, 6.6–7.4), respectively. Death and respiratory failure were also ranked as the top two most important outcomes by the public and health professionals. However, they were of equal importance with preference scores for death and respiratory failure being 8.3 (95% CI, 8.1–8.5) and 8.2 (95% CI, 8.1–8.4) for the public and 9.0 (95% CI, 8.8–9.2) and 9.0 (95% CI, 8.8–9.1) for health professionals. Shortness of breath the most important patient-reported outcome for patients/family, the public, and health professionals.

The differences in mean preferences scores between patients/family and health professionals and the public were generally small, being less than 2.0 (scale 1 to 9) for all outcomes (Fig. 4). Compared to patients/family, the most notable differences were for death (mean differences in relative preference scores of −1.3 [95% CI, −1.7 to −0.9] and −2.0 [95% CI, −2.4 to −1.6] for the public and health professionals, respectively) and respiratory failure (mean differences of −1.1 [95% CI, −1.6 to −0.7] and −1.2 [95% CI, −1.6 to −0.8] for the public and health professionals, respectively). Impact on family, anxiety, depression, fatigue, recovery, and life participation were all ranked
<figure><img src="image_6.png" /><figcaption>**Figure 4.** Difference in mean scores among patients/family members, public, and health professionals based on the Best-Worst Scale. Patients/family members are the reference group. diff = difference, HAI = hospital-acquired infection.</figcaption></figure>

as more important by patients/family members compared to the public; chest pain, muscle pain, gastrointestinal outcomes, and fatigue were ranked as more important by patients/family compared to health professionals.

The mean, median, and Best-Worst Scale scores according to survey language (English, Chinese, Italian, Spanish, and Portuguese) are provided in eTable 4 and eFigure 1 (Supplemental Digital Content 1, http://links.lww.com/CCM/F730). The 10 most highly prioritized outcomes by country were generally similar. Respiratory failure was in the top two for all languages. Based on the mean score, death was first in the English survey, third in the Portuguese survey, fourth

for Chinese, and sixth for the Spanish and Italian surveys. Subgroup analysis by disease severity was not possible due to the limited number of patients with severe disease.

## Additional Outcomes

Thirteen additional outcomes were each suggested by 30 or more respondents, including post-traumatic stress disorder (PTSD), anemia/iron, delirium, diabetes, cognition, immunity and antibodies, dizziness, dysphagia, acute kidney injury, financial impact, headache, and physical function.

## Themes From Comments

We identified four themes underpinning the prioritization of outcomes, which are described below. Selected quotations to support each theme are provided in eFigure 2 (Supplemental Digital Content 1, http://links.lww.com/CCM/F730).

**Fear of Life-Threatening, Debilitating, and Permanent Consequences.** Participants gave higher priority to outcomes that were seen as a threat to survival—“(respiratory failure) is thought to be the main cause of death.” The visible risk of death explained the high priority for mortality—“you see death coming for you” and that you “imagine the worse possible scenario, particularly when you are short of breath.” Severe outcomes

that required isolation or admission to intensive care or invasive interventions were distressing and “could be linked to longer-term issues such as PTSD.” Symptoms that “lingered” and impaired quality of life (e.g., shortness of breath) were given higher priority. For patients, the high prioritization of impact on family, depression, and anxiety reflected angst about isolation and profound guilt of infecting others. Anxiety was also rated high because it exacerbated symptoms—“this uncertainty causes anxiety and makes breathing harder.”

**Addressing Knowledge Gaps.** Because of the uncertain trajectory and prognosis of COVID-19, participants gave higher priority to outcomes related to disease progression (e.g., mortality, respiratory
failure, pneumonia, lung function, and recovery). Some gave higher priority to outcomes they believed were specific to COVID-19 (e.g., respiratory failure, shortness of breath) over those regarded to be more general (e.g., anxiety) or may have been preexisting.

**Enabling Preparedness and Planning.** Outcomes were given higher priority if they had implications for healthcare resource use. Respiratory failure was a high priority because it could require admission into intensive care and interventions including oxygen, mechanical ventilation, and extracorporeal membrane oxygenation—"it informs all measures that should be taken (lockdown, production of ventilators, etc.)" Some also considered outcomes for decision-making—"this would inform my decision to continue very invasive treatment or not."

**Tolerable or Infrequent Outcomes.** Participants gave lower priority for outcomes such as cough, gastrointestinal symptoms, fever, and muscle pain because these were perceived to be manageable, uncommon, or transient. Some patients based their decisions on whether they had experienced the symptom personally.

## DISCUSSION

For patients, family members, health professionals, and the public, mortality, respiratory failure, pneumonia, organ failure, lung function, lung scarring, sepsis, shortness of breath, and blood oxygen were the outcomes of highest priority. Mortality, respiratory failure, pneumonia, and organ failure of highest priority for all three groups. The patient-reported outcome of highest priority for all groups was shortness of breath. Hospitalization was in the top 10 for patients/family and health professionals. Viral load/clearance was in the top 10 for the public. Recovery was the top-rated long-term outcome. The top prioritized outcomes reflected the prevailing fear of death and debilitating and permanent consequences, uncertainty of disease progression, and concerns about the burden of disease on health systems.

Mortality and respiratory failure have also been identified in the three core outcomes set previously established for COVID-19 (20-22), albeit with previous initiatives focused on hospitalized patients only. The WHO core outcome measures set for COVID-19 include viral burden and a clinical progression scale that includes five states: uninfected, ambulatory, hospitalized with mild disease, hospitalized with severe disease, and death. The "ambulatory" state refers to the patient being symptomatic. However, no specific symptoms were identified. Shortness of breath, recovery (defined as feeling better, no longer having symptoms), and chest pain were the three patient-reported outcomes of highest importance as identified in our survey. Of note, chest pain, fatigue, muscle pain, and gastrointestinal problems were rated higher by patients/family compared with health professionals. Anxiety was rated higher by patients/family compared with both health professionals and the general public. People with COVID-19 attributed anxiety to the fear and guilt of infecting others and the need to be quarantined and isolated from family and support persons. Anxiety exacerbated their symptoms, particularly shortness of breath. The inclusion of patients and the public brings nuanced insights and attention to include outcomes of importance (e.g., shortness of breath) that may be underrecognized and not explicitly included in core outcome sets.

The survey included a large number of participants ($n=9,289$) from 111 countries, with 47% comprising of patients, family members, and the public. The survey was available in five languages. The data were triangulated by assessing both the absolute and relative importance of scores, using four measures (mean, median, proportion, and Best-Worst Scale score). However, there are some potential limitations. The online survey would have precluded involvement of those who do not have access to the internet. Most participants were from high-income countries and completed the English language survey, and only a small number of patients with severe disease were included. Not all participants completed the entire survey; however, we have indicated the number of responses for each outcome and of those who completed the Likert ratings and Best-Worst Scale.

## CONCLUSIONS

Mortality, respiratory failure, organ failure, sepsis, and shortness of breath are of critical importance to people with confirmed or suspected COVID-19 and their family members, the public, and health professionals. These outcomes will comprise the proposed core outcomes to be reviewed and discussed at international, on-line multi-stakeholder consensus workshops to establish the core outcome domains for COVID-19. The use of outcomes of importance to all stakeholders, including patients, the public, and health professionals, can help to improve the relevance of trials to better inform decision-making in practice and policy.

## ACKNOWLEDGMENTS

We thank all the patients, family members, and members of the general public and health professionals who gave their time and effort to participate in the survey. We thank Yen Ho, who designed and programmed the survey. COVID-19-Core Outcomes Set (COS) Survey Investigators are as follows: Ximena Atilano-Carsi (National Institute of Health Sciences and Nutrition, Mexico City, Mexico); Samaya Anumudu (Baylor College of Medicine, Houston, TX); Amelie Bernier-Jean (The University of Sydney, Sydney, NSW, Australia); Amanda Dominello (The University of Sydney, Sydney, NSW, Australia); Carolina Echegoyen (Centro de Salud Familiar Santa Amalia, Santiago, Chile); Sally Green (Monash University, Melbourne, VIC, Australia); Elyssa Hannan (The University of Sydney, Sydney, NSW, Australia); Camilla S. Hanson (The University of Sydney, Sydney, NSW, Australia); Tess Harris (PKD International, London, United Kingdom); Sophie Hill (LaTrobe University, Melbourne, VIC, Australia); Rabia Khalid (The University of Sydney, Sydney, NSW, Australia); Katharina Kohler (The University of Cambridge, Cambridge, United Kingdom); Yong-Lim Kim (Kyungpook National University, Daegu, South Korea); Pamela Lopez-Vargas (The University of Sydney, Sydney, NSW, Australia); Karine Manera (The University of Sydney, Sydney, NSW, Australia); Gabriela Pomoglio (Aterym SRL Cordoba Military Hospital, Cordoba, Argentina); Sarah Rowan (Denver Health, Denver, CO); Benedicte Sautenet (University of Tours, Tours, France); Nicole Scholes-Robertson (The University of Sydney, Sydney, NSW, Australia); Jenny Shen (University of
California Los Angeles, Los Angeles, CA); Anneliese Synnot (La Trobe University, Melbourne, VIC, Australia); Rhiannon Tate (Monash University, Melbourne, VIC, Australia); Marcelo Urra (The University of Chile, Santiago, Chile); Angela Yee Moon Wang (The University of Hong Kong, Hong Kong, China); and David C. Wheeler (University College London, London, United Kingdom). Drs. Evangelidis and Tong are co-first authorship.

Drs. Evangelidis and Tong share co-first authorship.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's website (http://journals.lww.com/ccmjournal).

The project is funded by the Flinders University and the National Coronavirus Disease 2019 Clinical Evidence Taskforce, convened by the Australian Living Evidence Consortium, hosted by Cochrane Australia, School of Public Health and Preventive Medicine, Monash University supported by the Australian Government, Victorian Department of Health and Human Services, Ian Potter Foundation, Walter Cottman Endowment Fund (managed by Equity Trustees), and the Lord Mayor's Charitable Foundation).

Dr. Tong is supported by The University of Sydney Robinson Fellowship. Dr. Morris is supported by a Clinical Research Career Development Fellowship from the Wellcome Trust (WT 2055214/Z/16/Z). Dr. Douglas is principal investigator of clinical and translational research studies of acute respiratory distress syndrome, sepsis, and coronavirus disease 2019 from National Institutes of Health, Roche Pharmaceuticals, and Genentech. Research grants are to his institution, Denver Health Medical Center. Dr. Povoa had received lecture fees from Orion, Pfizer, and Technofage. Dr. Azevedo received funding from Pfizer and Halex-Istar. Dr. Mer received funding from Pfizer, MSD, Sanofi Aventis, Aspen, and Astellas. Dr. Morris received support for article research from Wellcome Trust/COAF. Dr. Smyth's institution received funding from Vertex; he received funding from Vertex, Teva, and Novartis; and he has a patent issued "Alkyl quinolones as biomarkers of Pseudomonas aeruginosa infection and uses thereof." Dr. Turner's institution received funding from Stroke Foundation, Australian Medical Research Future Fund, Victorian State Government, and she disclosed that she is employed by Monash University; provides technical advice to WHO and Australian Red Cross Lifeblood; has undertaken consultancies for USAID, the Burnet Institute, and WHO; works on a shared MRFF research grant with the Stroke Foundation, and conducts research funded by Australian Commonwealth and State research funding agencies. Dr. Cho's institution received funding from Baxter and project grants through the NHMRC and HRC of New Zealand; he received funding from Baxter; he receives stipend from the Peritoneal Dialysis International (Associate Editor); and he is the current recipient of the National Health and Medical Research Council early career fellowship awarded by the Australian government. The remaining authors have disclosed that they do not have any potential conflicts of interest.

For information regarding this article, E-mail: Allison.tong@sydney.edu.au

## REFERENCES

1. World Health Organization: Coronavirus Disease (Covid-19) Pandemic. 2020. Available at: https://www.who.int/emergencies/disases/novel-coronavirus-2019. Accessed April 27, 2020

2. Li X, Xu S, Yu M, et al: Risk factors for severity and mortality in adult COVID-19 inpatients in Wuhan. *J Allergy Clin Immunol* 2020; 146:110-118

3. Du RH, Liang LR, Yang CQ, et al: Predictors of mortality for patients with COVID-19 pneumonia caused by SARS-CoV-2: A prospective cohort study. *Eur Respir J* 2020; 55:2000524

4. Zhou F, Yu T, Du R, et al: Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: A retrospective cohort study. *Lancet* 2020; 395:1054-1062

5. Bhatraju PK, Ghassemieh BJ, Nichols M, et al: Covid-19 in critically ill patients in the Seattle region - case series. *N Engl J Med* 2020; 382:2012-2022

6. Grasselli G, Zangrillo A, Zanella A, et al: Baseline characteristics and outcomes of 1591 patients infected with SARS-CoV-2 admitted to ICUs of the Lombardy Region, Italy. *J Am Med Assoc* 2020; 323:1574-1581

7. Auld SC, Caridi-Scheible M, Blum JM, et al: ICU and ventilator mortality among critically ill adults with coronavirus disease 2019. *Crit Care Med* 2020; 48:e799-e804

8. Richardson S, Hirsch JS, Narasimhan M, et al: Presenting characteristics, comorbidities, and outcomes among 5700 patients hospitalized with COVID-19 in the New York City Area. *JAMA* 2020; 323:2052-2059

9. Chen N, Zhou M, Dong X, et al: Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: A descriptive study. *Lancet* 2020; 395:507-513

10. Tian S, Hu N, Lou J, et al: Characteristics of COVID-19 infection in Beijing. *J Infect* 2020; 80:401-406

11. Guan WJ, Ni ZY, Hu Y, et al; China Medical Treatment Expert Group for Covid-19: Clinical characteristics of coronavirus disease 2019 in China. *N Engl J Med* 2020; 382:1708-1720

12. Holmes EA, O'Connor RC, Perry VH, et al: Multidisciplinary research priorities for the COVID-19 pandemic: A call for action for mental health science. *Lancet Psychiatry* 2020; 7:547-560

13. Zhang K, Zhou X, Liu H, et al: Treatment concerns for psychiatric symptoms in COVID-19-infected patients with or without psychiatric disorders. *Br J Psychiatry* 2020; 217:351

14. Liu Y, Cao L, Li X, et al: Awareness of mental health problems in patients with coronavirus disease 19 (COVID-19): A lesson from an adult man attempting suicide. *Asian J Psychiatr* 2020; 51:102106

15. World Health Organization: International Clinical Trials Registry Platform (ICTRP). 2020. Available at: https://www.who.int/ictrp/en/. Accessed April 20, 2020

16. Calvert M, Kyte D, Mercieca-Bebber R, et al; the SPIRIT-PRO Group: Guidelines for inclusion of patient-reported outcomes in clinical trial protocols: The SPIRIT-PRO extension. *JAMA* 2018; 319:483-494

17. Sarma P, Kaur H, Kumar H, et al: Virological and clinical cure in COVID-19 patients treated with hydroxychloroquine: A systematic review and meta-analysis. *J Med Virol* 2020; 92:776-785

18. Cortegiani A, Ippolito M, Ingoglia G, et al: Update I. A systematic review on the efficacy and safety of chloroquine/hydroxychloroquine for COVID-19. *J Crit Care* 2020; 59:176-190

19. Yang Z, Liu J, Zhou Y, et al: The effect of corticosteroid treatment on patients with coronavirus infection: A systematic review and meta-analysis. *J Infect* 2020; 81:e13-e20

20. Jin X, Pang B, Zhang J, et al: Core outcome set for clinical trials on coronavirus disease 2019 (COS-COVID). *Engineering (Beijing)* 2020 Mar 18. [online ahead of print]

21. Qiu R, Zhao X, Liang T, et al: Core outcome set for clinical trials of COVID-19 based on traditional Chinese and Western medicine. *Front Pharmacol* 2020; 11:781

22. WHO Working Group on the Clinical Characterisation and Management of COVID-19 infection: A minimal common outcome measure set for COVID-19 clinical research. *Lancet Infect Dis* 2020; 20:e192-e197

23. Kirkham JJ, Gorst S, Altman DG, et al: Core Outcome Set-STARdards for Reporting: The COS-STAR Statement. *PLoS Med* 2016; 13:e1002148

24. Williamson PR, Altman DG, Bagley H, et al: The COMET Handbook: Version 1.0. *Trials* 2017; 18:280

25. COMET: The COMET Initiative: Core Outcome Set Developers' Response to COVID-19 15th April 2020. 2020. Available at: http://www.comet-initiative.org/Studies/Details/1538. Accessed April 22, 2020

26. Schunemann H, Brozek J, Guyatt G, et al: GRADE Handbook for Grading Quality of Evidence and Strength of Recommendation. 2020. Available at: https://gdt.gradepro.org/app/handbook/handbook.html. Accessed April 20, 2020

27. Flynn TN, Louviere JJ, Peters TJ, et al: Best-worst scaling: What it can do for health care research and how to do it. *J Health Econ* 2007; 26:171-189

28. Bradley RA, Terry ME: Rank analysis of incomplete block designs: I. The method of paired comparisons. *Biometrika* 1952; 39:324-345