# A multivariate model for successful publication of intensive care medicine randomized controlled trials in the highest impact factor journals: the SCOTI score

Joris Pensier, Audrey de Jong, Gerald Chanques, Emmanuel Futier, Elie Azoulay, Nicolas Molinari, Samir Jaber

► To cite this version:

Joris Pensier, Audrey de Jong, Gerald Chanques, Emmanuel Futier, Elie Azoulay, et al. A multivariate model for successful publication of intensive care medicine randomized controlled trials in the highest impact factor journals: the SCOTI score. Annals of Intensive Care, 2021, 11 (1), 10.1186/s13613-021-00954-x. http://hal-03453658v2

HAL Id: hal-03453658

https://hal.science/hal-03453658v2

Submitted on 25 Jan 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

RESEARCH

Open Access

<figure><img src="image_3.png" /></figure>

Check for
updates

# A multivariate model for successful publication of intensive care medicine randomized controlled trials in the highest impact factor journals: the SCOTI score

Joris Pensier¹, Audrey De Jong¹, Gerald Chanques¹, Emmanuel Futier², Elie Azoulay³, Nicolas Molinari⁴,⁵ and Samir Jaber¹,<fnref n="6" />*

**Abstract**

**Background:** Critical care randomized controlled trials (RCTs) are often published in high-impact journals, whether general journals [the New England Journal of Medicine (NEJM), The Lancet, the Journal of the American Medical Association (JAMA)] or critical care journals [Intensive Care Medicine (ICM), the American Journal of Respiratory and Critical Care Medicine (AJRCCM), Critical Care Medicine (CCM)]. As rejection occurs in up to 97% of cases, it might be appropriate to assess pre-submission probability of being published. The objective of this study was to develop and internally validate a simplified score predicting whether an ongoing trial stands a chance of being published in high-impact general journals.

**Methods:** A cohort of critical care RCTs published between 1999 and 2018 in the three highest impact medical journals (NEJM, The Lancet, JAMA) or the three highest impact critical care journals (ICM, AJRCCM, CCM) was split into two samples (derivation cohort, validation cohort) to develop and internally validate the simplified score. Primary outcome was journal of publication assessed as high-impact general journal (NEJM, The Lancet, JAMA) or critical care journal (ICM, AJRCCM, CCM).

**Results:** A total of 968 critical care RCTs were included in the predictive cohort and split into a derivation cohort (n = 510) and a validation cohort (n = 458). In the derivation cohort, the sample size (*P* value < 0.001), the number of centers involved (*P* value = 0.01), mortality as primary outcome (*P* value = 0.002) or a composite item including mortality as primary outcome (*P* value = 0.004), and topic [ventilation (*P* value < 0.001) or miscellaneous (*P* value < 0.001)] were independent factors predictive of publication in high-impact general journals, compared to high-impact critical care journals. The SCOTI score (Sample size, Centers, Outcome, Topic, and International score) was developed with an area under the ROC curve of 0.84 (95% Confidence Interval, 0.80–0.88) in validation by split sample.

**Conclusions:** The SCOTI score, developed and validated by split sample, accurately predicts the chances of a critical care RCT being published in high-impact general journals, compared to high-impact critical care journals.

**Keywords:** Intensive care unit, Critical care, Critically ill patients, Intensive care medicine, Precision medicine

**Background**

Since the late twentieth century, evidence-based medicine has emerged as a new paradigm to produce knowledge and improve clinical practice [1]. Randomized controlled trials (RCTs) are considered as the pinnacle

<figure><img src="image_4.png" /></figure>

<footnote>*Correspondence: s-jaber@chu-montpellier.fr</footnote>

<footnote n="6">⁶ Département d'Anesthésie Réanimation B (DAR B), 80 Avenue Augustin Fliche, 34295 Montpellier, France</footnote>

<footnote>Full list of author information is available at the end of the article</footnote>
of evidence for evaluating new and existing interven-
tions in medicine [2], as long as they are appropriately
designed, conducted, and reported [3, 4]. They have
been further promoted by the development of online
databases, reviews, meta-analyses, and education of cli-
nicians and students [5].

Critical care is no exception. Many RCTs have been
performed in this field and have often led to publi-
cations in the highest impact critical care journals,
such as Intensive Care Medicine (ICM), the Ameri-
can Journal of Respiratory and Critical Care Medicine
(AJRCCM), and Critical Care Medicine (CCM) [6–8].
Moreover, landmark trials have changed the manage-
ment of critically ill patients [9–12]. Many of those
studies have been published in the highest impact gen-
eral journals: The New England Journal of Medicine
(NEJM), The Lancet, and the Journal of the American
Medical Association (JAMA).

It is the final purpose of medical research to broad-
cast its results as widely as possible to health care prac-
titioners to most impact the management of patients
worldwide [13]. In this context, RCTs performed in the
field of critical care aim to be published in the highest
impact general journals [2]. As up to 97% of trials are
rejected outright or post-review, it might be appropri-
ate to assess pre-submission probability of being pub-
lished. To our knowledge, no study has yet explored the
factors associated with the publication of a critical care
RCT in a high-impact general journal.

We designed this cohort of studies to identify inde-
pendent predictive factors of publication of a criti-
cal care RCT in a high-impact general journal (NEJM,
Lancet, JAMA) compared to the high-impact critical
care journals (ICM, AJRCCM, CCM). The secondary
objectives of this study were to develop and validate a
simplified score predicting publication of an RCT in
a high-impact general journal. We hypothesized that
most predictive factors of publication of a critical care
RCT are determined before the inclusion of its first
patient.

**Materials and methods**

Study design

We conducted a retrospective cohort of RCTs accord-
ing to the Transparent Reporting of a Multivariable
Prediction Model for Individual Prognosis or Diagno-
sis (TRIPOD) reporting guideline statement [14]. The
cohort of RCTs was gathered as a systematic review of
RCTs performed in critically ill patients between 1999
and 2018, following the Preferred Reporting Items
for Systematic reviews and Meta-Analyses (PRISMA)
statement [15].

Data sources and study selection

The search strategy is detailed in the electronic supple-
mentary material (Additional file 1: eAppendix S1). We
screened for relevant RCTs performed: (1) on critically ill
patients, (2) that enrolled adults, and (3) were published
between 1999 and 2018, (4) in the three highest impact
general journals [The New England Journal of Medicine
(NEJM), The Lancet and the Journal of the American
Medical Association (JAMA)] or the three highest impact
critical care journals [Intensive Care Medicine (ICM),
American Journal of Respiratory and Critical Care Medi-
cine (AJRCCM) and Critical Care Medicine (CCM)].
RCTs performed on animals, on pediatric patients, on
non-critically ill patients, bench studies, and simulation
studies were not eligible. The three journals for each cat-
egory (general journals and critical care journals) were
selected according to the values of their mean Impact
Factor obtained over 3 years from 2016 to 2018 [16]. We
chose to only take into account the three “princeps” gen-
eral journals, which were continuously available from
1999 to 2018. The “sister” journals of a general journal
(such as Lancet Respiratory Medicine or JAMA Net-
work) were not included in our research, as they have not
been continuously available over the 20-year period, and
their respective impact factors are not in the same range
as the “princeps” journals.

Data collection

First, two authors (JP and ADJ) independently screened
the studies retrieved by title and then by abstract for
exclusion. They assessed the full text of possibly relevant
studies for inclusion and exclusion criteria. Disagree-
ment was resolved by discussion and arbitrated, if neces-
sary, by a third author (SJ). Data were then added to an
excel database, specifically designed for this review [15].
Journal of publication, year of publication, sample size,
numbers of centers involved, country of the first author,
number of countries participating, primary endpoint of
the RCT, the result of the RCT according to its primary
endpoint, the type of intervention tested, and the topic of
the RCT were extracted.

Journal of publication was classified either as a high-
impact general journal (NEJM, Lancet, JAMA) or as a
high-impact critical care journal (ICM, AJRCCM, CCM).

The result of the RCT was classified either as unsig-
nificant, significant for benefit, or significant for harm.
We adapted a previously published classification [17] to
include equivalence and non-inferiority designs. It was
considered insignificant if the *P* value was higher than
0.05 for superiority trial, or failed to prove the equiva-
lence or the non-inferiority for equivalence and non-
inferiority trials. The result of the RCT was considered
significant for benefit if the *P* value was equal or lesser than 0.05 with a better outcome in the intervention group for superiority trials, or if the equivalence or non-inferiority was reached in equivalence or non-inferiority trials, or if the superiority of the intervention was reached in equivalence or non-inferiority trials. The result of the RCT was considered significant for harm if the *P* value was equal or lesser than 0.05 with a worse outcome in the intervention group for superiority trials, or if the inferiority was reached in equivalence or non-inferiority trials.

The type of intervention tested assessed whether the RCT evaluated a drug or another type of intervention [18].

## Statistical analysis

The study size was determined by the total number of RCTs published in the six journals taken into account over the studied period. The database was then split into two cohorts according to the year of publication, to control time-effect and change of policy in the journal's editing [19]. The derivation cohort included the RCTs published on even years, and the validation cohort included the RCTs published on odd years. We described the cohorts using means, ranges, and SDs as appropriate for continuous variables and frequencies with proportion for categorical variables. We compared proportions using a $\chi^2$ test. We compared ordinal categorical variables using a Kruskal-Wallis rank sum test. There was no missing data, so no missing data imputation technique was used [20].

Logistic regression was used to identify predictive factors for publication in a high-impact general journal in the derivation cohort [19, 20]. Continuous variables were split into multiple categories according to their quartiles. A multivariate model was established to predict publication in a high-impact general journal [19, 21]. All variables were selected (regardless of their *P* value in the univariate analysis) and a stepwise procedure was used to select the final model, according to their Akaike Information Criteria (AIC) [22]. No time effect was found by entering the year of publication variable in a multivariate model. To establish a simplified score, we gave a score to each of the variables included in the final prediction model in relation to each one's *b* parameter (regression coefficient) in that model [19]. The discriminative ability of the score was evaluated in both cohorts with receiver operating characteristic (ROC) curves to estimate the area under the curve (AUC), to internally validate the simplified score [21]. We used the bootstrap to internally validate the simplified score by sampling with replacement for 500 iterations [23]. The calibration of the score was graphically assessed by plotting the observed probability (Kaplan-Meier estimates) against

the mean predicted probability within tenths of the predicted probabilities [21]. A probability of publication in a high-impact general journal compared to a high-impact critical care journal according to the simplified score was considered low under 20% and high over 80%. A *P* value of less than or equal to 0.05 was considered statistically significant.

We used SAS, version 9.4 (SAS Institute), for data analysis. The simplified score was developed using rigorous methodological standards, was internally validated using both split-sample and bootstrap validation, and was reported according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline statement [14]. The sample size of the validation cohort met recommendations for validation studies of prediction tools, namely, a minimum of 100 events and a minimum of 100 nonevents [24].

## Results

### RCTs population

We identified 18,515 articles using the search strategy. We excluded 1780 citations because of duplications and 15,489 citations on the initial abstract screen, because inclusion criteria were not met. After examination of the full text of the 1246 selected papers, we included 968 RCTs in this study. Figure 1 shows the study selection flowchart. 510 RCTs were included in the derivation cohort, and 458 RCTs were included in the validation cohort. 129 RCTs (25%) in the derivation cohort and 106 RCTs (23%) in the validation cohort were published in a high-impact general journal. Overall, the median sample size was 120 [interquartile range (IQR), 46 to 352], the median number of centers was 1 (IQR, 1 to 11) and the median number of countries was 1 (IQR, 1 to 1). For the analysis, the sample size was split into four categories (<46, 46–120, 121–352, >352) the number of centers was split into three categories (1, 2–10, >10), and the number of countries was dichotomized (national design, international design). Characteristics of the derivation and validation cohorts are described in the electronic supplementary material (Additional file 1: Table S1).

### Final model development

Univariate and multivariate analyses in the derivation cohort are presented in Table 1. In the derivation cohort, the sample size, the number of centers involved, and mortality or composite including mortality primary outcome and topic (ventilation or miscellaneous) were independent predictive factors significantly associated with publication in a high-impact general journal.

The final multivariate model was constructed with the 510 RCTs of the derivation cohort and all available data.
<figure><img src="image_5.png" /></figure>

The main predictors for publication in a high-impact general journal were sample size, the number of centers involved, primary outcome, the RCT topic, and an international design. Results of the multivariate logistic regression are presented in Table 2. The AUC of the model was at 0.91 (95% Confidence Interval (95% CI), 0.88–0.94) (Fig. 2a). The calibration plot for the derivation cohort showed that the model calibration line is very close to the ideal calibration line (Additional file 1: Figure S1). There was no collinearity in the model. No time effect was found by entering the year of publication variable in a multivariate model (Additional file 1: Table S2). No significant interactions were found, neither between the outcome and the interpretation result, nor between the number of centers and the sample size (Additional file 1: Table S3).

## Final model validation

Univariate and multivariate analyses in the validation cohort are presented in the electronic supplementary material (Additional file 1: Table S4). Internal validation

of the model with the validation cohort indicated high discrimination with a model AUC at 0.85 (95% CI 0.81–0.88) (Fig. 2b). The calibration plot for the validation cohort showed that the model calibration line is very close to the ideal calibration line (Additional file 1: Fig. S2).

After internal validation by bootstrap, the final model AUC was at 0.90 (95% CI 0.87–0.92).

## SCOTI score development and validation

The simplified score (SCOTI score for Sample size, Centers, Outcome, Topic, and International score) constructed using the final model is described in Fig. 3a, ranging from 0 to 100. The model AUC was at 0.85 (95% Confidence Interval (95% CI), 0.82–0.89) (Fig. 2c). The calibration plot for the derivation cohort showed that the model calibration line is very close to the ideal calibration line (Additional file 1: Fig. S3). Distribution of the SCOTI score in the derivation cohort is presented in Additional file 1: Fig. S4.
<table><caption>**Table 1** Predictive factors of publication in high-impact general journals in the derivation cohort, compared to high-impact critical care journals</caption><thead><tr><th rowspan="2">Characteristic <sup>a</sup></th><th rowspan="2">Total (%)<br>(n = 510)</th><th colspan="2">Univariate analysis</th><th colspan="2">Multivariate analysis</th></tr><tr><th>OR</th><th>95%CI</th><th>OR</th><th>95%CI</th></tr></thead><tbody><tr><td>Sample size</td><td>&lt;46</td><td>107 (21%)</td><td>1</td><td>1</td><td>1</td></tr><tr><td></td><td>46-120</td><td>141 (27%)</td><td>3.58</td><td>0.76-16.92</td><td>3.00</td></tr><tr><td></td><td>121-352</td><td>137 (27%)</td><td>15.35</td><td>3.58-65.77</td><td>8.56</td></tr><tr><td></td><td>&gt;352</td><td>125 (25%)</td><td>120.17</td><td>28.19-512.23</td><td>61.09</td></tr><tr><td>Number of<br>centers</td><td>1</td><td>258 (50%)</td><td>1</td><td></td><td>1</td></tr><tr><td></td><td>2-10</td><td>117 (23%)</td><td>4.74</td><td>2.55-8.79</td><td>1.84</td></tr><tr><td></td><td>&gt;10</td><td>135 (27%)</td><td>17.21</td><td>9.65-30.70</td><td>3.28</td></tr><tr><td>International trial</td><td>Yes / No</td><td>76 (15%) / 434 (85%)</td><td>4.93</td><td>2.96-8.20</td><td>1.76</td></tr><tr><td>Primary<br>endpoint</td><td>Mortality<br>alone</td><td>93 (18%)</td><td>8.66</td><td>2.26-14.24</td><td>3.87</td></tr><tr><td></td><td>Composite<br>including<br>mortality</td><td>16 (3%)</td><td>9.11</td><td>3.20-25.98</td><td>12.91</td></tr><tr><td>Interpretation result</td><td>Other</td><td>401 (79%)</td><td>1</td><td></td><td>1</td></tr><tr><td></td><td>Significant<br>for benefit</td><td>233 (46%)</td><td>0.42</td><td>0.27-0.64</td><td>1.28</td></tr><tr><td></td><td>Significant<br>for harm</td><td>6 (1%)</td><td>1.04</td><td>0.19-5.79</td><td>0.38</td></tr><tr><td></td><td>Unsignificant</td><td>271 (53%)</td><td>1</td><td></td><td>1</td></tr><tr><td>Type of<br>intervention</td><td>Drug (Yes/<br>No)</td><td>213 (42%) / 297<br>(58%)</td><td>0.71</td><td>0.48-1.06</td><td>0.80</td></tr><tr><td>Country of<br>first author</td><td>USA (Yes/<br>No)</td><td>111 (22%) / 399<br>(78%)</td><td>1.34</td><td>0.84-2.13</td><td>1.31</td></tr><tr><td>Topic</td><td>Sepsis</td><td>84 (17%)</td><td>1</td><td></td><td>1</td></tr><tr><td></td><td>Cardiovascular</td><td>89 (18%)</td><td>1.62</td><td>0.81-3.28</td><td>1.26</td></tr><tr><td></td><td>Ventilation</td><td>130 (25%)</td><td>1.29</td><td>0.66-2.50</td><td>6.27</td></tr><tr><td></td><td>Miscellaneous</td><td>207 (40%)</td><td>1.39</td><td>0.75-2.58</td><td>5.10</td></tr></tbody><tfoot><tr><td colspan="6">OR: Odd Ratio, 95%CI: Confidence Interval at 95%, USA: United States of America</td></tr><tr><td colspan="6"><sup>a</sup> There was no missing data in the derivation cohort</td></tr></tfoot></table>
<table><caption>**Table 2** Study outcomes</caption><thead><tr><th>Variable</th><th></th><th>β parameter<sup>a</sup></th><th>Odd ratio</th><th>95% CI</th><th>P value</th></tr></thead><tbody><tr><td>Intercept</td><td></td><td>-5.83</td><td></td><td></td><td>&lt;0.001</td></tr><tr><td rowspan="4">Sample size</td><td>&lt;46</td><td>0</td><td></td><td></td><td></td></tr><tr><td>46-120</td><td>1.13</td><td>3.09</td><td>0.60-15.99</td><td>0.18</td></tr><tr><td>121-352</td><td>2.16</td><td>8.64</td><td>1.77-42.09</td><td>0.01</td></tr><tr><td>&gt;352</td><td>4.13</td><td>62.06</td><td>11.93-322.94</td><td>&lt;0.001</td></tr><tr><td rowspan="3">Centers</td><td>1</td><td>0</td><td></td><td></td><td></td></tr><tr><td>2-10</td><td>0.60</td><td>1.82</td><td>0.83-3.96</td><td>0.13</td></tr><tr><td>&gt;10</td><td>1.09</td><td>2.96</td><td>1.31-6.71</td><td>0.01</td></tr><tr><td rowspan="3">Primary endpoint</td><td>Mortality</td><td>1.28</td><td>3.60</td><td>1.81-7.15</td><td>&lt;0.001</td></tr><tr><td>Composite including mortality</td><td>2.50</td><td>12.23</td><td>3.04-49.28</td><td>&lt;0.001</td></tr><tr><td>Other</td><td>0</td><td></td><td></td><td></td></tr><tr><td rowspan="4">Topic</td><td>Sepsis</td><td>0</td><td></td><td></td><td></td></tr><tr><td>Cardiovascular</td><td>0.29</td><td>1.33</td><td>0.50-3.55</td><td>0.57</td></tr><tr><td>Ventilation</td><td>1.88</td><td>6.58</td><td>2.51-17.26</td><td>&lt;0.001</td></tr><tr><td>Miscellaneous</td><td>1.69</td><td>5.40</td><td>2.17-13.44</td><td>&lt;0.001</td></tr><tr><td>International</td><td></td><td>0.53</td><td>1.70</td><td>0.82-3.53</td><td>0.15</td></tr></tbody></table>

OR: Odd Ratio, 95%CI: Confidence Interval at 95%

<sup>a</sup> β parameters: coefficients from the logistic regression model

Internal validation of the SCOTI score on the validation cohort indicated high discrimination with model AUC at 0.84 (95% CI 0.80-0.88) (Fig. 2d). The calibration plot for the validation cohort showed that the model calibration line is very close to the ideal calibration line (Additional file 1: Figure S5). After internal validation by bootstrap, the AUC of the simplified score was at 0.85 (95% CI 0.82-0.89). Distribution of the SCOTI score in the validation cohort is presented in Additional file 1: Figure S6.

Figure 3b presents the predicted probabilities versus the observed probability of publication in a high-impact general journal on both cohorts in function of the SCOTI score. Figure 3c presents the observed probability of publication in a high-impact general journal according to three ranges of SCOTI score. A SCOTI score between 0 and 45 results in a low probability of publication in a high-impact general journal (6%). A SCOTI score between 50 and 80 results in an intermediate probability of publication in a high-impact general journal (47%). A SCOTI score above 80 results in a high probability of publication in a high-impact general journal (86%). Additional file 1: Table S6 presents the number of RCTs published by each journal per SCOTI score category. NEJM publishes significatively less RCTs with a SCOTI score between 0 and 45 than The Lancet or The JAMA (P<0.001, Kruskal-Wallis rank sum test).

A cutoff at 50 was determined by ROC analyses, to allow an excellent negative predictive value and a good

positive predictive value (i.e., a probability of publication in a high-impact general journal compared to a high-impact critical care journal over 20%). In the validation cohort, positive and negative predictive values (95% CI) were 49% and 93%, respectively, with a sensitivity of 81% and a specificity of 75%.

## Discussion

In this predictive cohort of RCTs performed in the field of critical care over 20 years and published in the three highest impact general journals and the three highest impact critical care journals, sample size, number of centers, the primary endpoint, and the topic of the RCT were independent predictive factors of successful publication in a high-impact general journal in multivariate analysis, compared to a high-impact critical care journal. We designed and internally validated by split-sample and bootstrap a model and a simplified score (SCOTI score for Sample size, Centers, Outcome, Topic, and International score). This study shows that a simple easy-to-use model can predict the probability of an RCT being published in a high-impact general journal.

All of the decisive factors of the predictive model and the SCOTI score are determined at trial conception, before inclusion of the first patient [3, 4]. The result of the study does not seem to influence its chances of being published in a high-impact general journal, leading to think that there are few publication bias when it comes to well-designed RCTs [25]. This means that high-impact
<figure><img src="image_6.png" /><figcaption>**Fig. 2** Receiver Operating Characteristic curves and Areas Under the Curve to predict publication in high-impact general journals, compared to high-impact critical care journals: a final model in development cohort. b Final model in validation cohort. c SCOTI score in development cohort. d SCOTI score in validation cohort. AUC: Area under the curve, 95%CI: Confidence Interval at 95%</figcaption></figure>

general journals have taken into account concerns regarding publication bias raised in the 2000s [25]. It is worth noting that it also suggests that critical care RCTs published in NEJM, The Lancet, or the JAMA are specifically designed to address the high requirements of these journals [18, 26]. High-impact general journals have already disclosed their publication policy as focusing on large, international RCTs with solid endpoints.

Indeed, high-impact general journals attach importance to the methodological construction of an RCT: its sample size, its multicentric and international character, as well as an objective and pertinent outcome, assessing mortality either as primary endpoint or as a component of the primary endpoint [18]. These results are consistent with a previous study focusing on manuscripts submitted to the British Medical Journal, The Lancet and

(See figure on next page.)

**Fig. 3** SCOTI score. **a** SCOTI score calculation worksheet. SCOTI score: sample size, centers, outcome, topic, and international score. **b** Probability of publication in high-impact general journals compared to high-impact critical care journals according to the SCOTI score: individual prediction. The plot shows predicted (orange curve) and observed (blue dots) probabilities of publication in high-impact general journals according to the SCOTI score. **c** Probability of publication in high-impact medical journals compared to high-impact critical care journals according to the SCOTI score: three categories. The plot shows observed probabilities of publication in high-impact general journals according to three groups of SCOTI score: "Low probability of success", "Intermediate probability of success" and "High probability of success"
A

<table><thead><tr><th>Variable</th><th></th><th>Points</th></tr></thead><tbody><tr><td rowspan="4">Sample size</td><td>&lt;46</td><td>0</td></tr><tr><td>46-120</td><td>10</td></tr><tr><td>121-352</td><td>25</td></tr><tr><td>&gt;352</td><td>40</td></tr><tr><td rowspan="3">Centers</td><td>1</td><td>0</td></tr><tr><td>2-10</td><td>10</td></tr><tr><td>&gt;10</td><td>15</td></tr><tr><td>Outcome: Primary endpoint</td><td>Mortality<br>(alone or in a composite outcome)</td><td>20</td></tr><tr><td rowspan="4">Topic</td><td>Sepsis</td><td>0</td></tr><tr><td>Cardiovascular</td><td>5</td></tr><tr><td>Ventilation</td><td>20</td></tr><tr><td>Other</td><td>20</td></tr><tr><td>International</td><td></td><td>5</td></tr><tr><td>Total</td><td></td><td>100</td></tr></tbody></table>

B

<figure><img src="image_7.png" /></figure>

C

<figure><img src="image_8.png" /><figcaption>**Fig. 3** (See legend on previous page.)</figcaption></figure>
Annals of Internal Medicine, that found that manuscripts with higher methodology scores were more likely to be published in those journals [27]. In this paper either, the result of a study was not associated with its acceptance. The impact factors of the critical care journals have substantially improved during the studied period, which has probably modified the quality of the RCTs published in these journals.

It is worth noting that the RCTs included in this study were published before the COVID-19 crisis, which may have altered the topics highlighted by high-impact general journals [28]. However, in the studied period sepsis seems to be equally studied in both critical care and general journals, while ventilation trials more often attract general journals.

Our study is to our knowledge the first to highlight independent predictors of publication in a high-impact general journal in the field of critical care. Moreover, it is to our knowledge the largest database of critical care RCTs published to this date, with 968 RCTs published over 20 years in 6 high-impact journals. In this study, we were able to develop and internally validate by split sample and bootstrap a simplified score with excellent discrimination and calibration qualities (Fig. 2, Additional file 1: Figures S3, S4). We could speculate that the SCOTI score presents three major points of interest for research clinicians and journal editors. First, it could help to optimize the methodology used to design an RCT according to the selected criteria which constitute the SCOTI score (Fig. 3) when these criteria are achievable [29]. Second, the SCOTI score might help to accelerate knowledge transfer by assessing pre-submission probability of acceptance. It is an ethical issue to disclose research results to the medical community as fast as possible after trial completion. Multiple submissions to general journals of an RCT with a low probability of acceptance predicted by the SCOTI score could be limited, to avoid waste of time. Third, the SCOTI score might be useful for authors to better understand how the highest impact general journals assess critical care RCTs for possible publication. However, such a score must not be the only component of a submission process, since an RCT with a low SCOTI score but with a revolutionary concept deserves to be granted funding and wide medical diffusion. Some researchers will probably continue to submit their research to general journals rather than directly submit to critical care journals even if their SCOTI scores were low.

Notwithstanding the fact that our model was developed on critical care RCTs, its consequences might be insightful to researchers of other specialties. An external validation cohort on a different field (such as emergency medicine or cardiology) could be complementary

to our analysis. Finally, although the score was built over 20 years, no effect of time was identified, neither as fixed nor random effect.

Our study also has limitations. First, we could not assess the manuscripts rejected by each journal over the period. Since our study hypothesizes that every RCT would have been published in a high-impact general journal if it had been accepted by one of them and that every RCT published in a high-impact critical care journal would have been rejected by high-impact general journals, it would have been an added value to compare both submitted and published RCTs to verify this hypothesis. Second, since there is little to no literature on this subject, variables collected and assessed for the model development were selected according to an expert panel (AD, GC, EA, EF, NM, SJ) and data on similar subjects [18, 30, 31]. Noteworthy, we chose to dichotomize the country of the study in United States of America versus other, while other classifications might have been used (such as English-speaking countries versus other). Moreover, we did not study in this paper the potential “human factors” (the number of publications of the first or last author, the endorsement by a large consortium or trial group, gender of authors...) influencing the publication process, since it would have highly increased the risk of collinearity with other variables [32]. Similarly, we did not focus on other methodological aspects, such as the absence of bias or the robustness of the statistical analysis. These human and methodological factors should be specifically evaluated in future works [33]. Third, we chose to include in our study the journals with the highest recent Impact Factors. From 2007 at least, data from Harhay and al. indicate that the 6 studied journals are the journals that publish the largest number of RCTs of their respective categories [18]. Likewise, we did not include in our analysis the sistership journals of the high-impact general journals (such as Lancet Respiratory Medicine), since their impact factors is closer to critical care journals than to high-impact general journals. Moreover, their existence is recent, and they cannot be evaluated from 1999 to 2018. Other methods of journals selection might have led to other journals being included. Fourth, since no linearity hypothesis could be done on any of the continuous variables, we had to categorize those variables. We split the variables into categories according to quartiles, since it is a valid and reproducible method. Fifth, we assessed high-impact general journals as a whole, but they might have different policies from one another. Sixth, comparing the characteristics of the trials accepted in the 6 included journals to journals with lower Impact Factors might have been very interesting, since publishing a paper in one of the 3 highest Impact Factor critical care journals is becoming more and more challenging.
## Conclusions

In this predictive cohort of critical care RCTs, the sample size, the number of centers, the primary endpoint, and the topic of an RCT were independent predictive factors of successful publication in a high-impact general journal in multivariate analysis, compared to a high-impact critical care journal. The SCOTI score was developed and internally validated by split sample and bootstrap. It predicts the probability of a critical care RCT being published in a high-impact general journal.

## Abbreviations

RCTs: Randomized controlled trials; ICM: Intensive Care Medicine; AJRCCM: The American Journal of Respiratory and Critical Care Medicine; CCM: Critical Care Medicine; NEJM: New England Journal of Medicine; JAMA: Journal of the American Medical Association; AIC: Akaike Information Criteria; ROC: Receiver operating characteristic; AUC: Area under the curve; SCOTI: Sample size, Centers, Outcome, Topic, and International score.

## Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s13613-021-00954-x.

**Additional file 1. Appendix S1.** Search strategy. **Table S1.** Description of derivation and validation cohorts. **Table S2.** Multivariate model to assess time effect. **Table S3.** Multivariate model to assess interaction between outcome and interpretation result and interaction between number of centers and sample size. **Table S4.** Univariate and multivariate analysis on the development cohort. **Table S5.** Univariate and multivariate analysis on the validation cohort. **Table S6.** Number of RCTs per SCOTI score category per journal. **Figure S1.** Calibration plot for the final model on the derivation cohort. **Figure S2.** Calibration plot for the final model on the validation cohort. **Figure S3.** Calibration plot for the SCOTI score on the derivation cohort. **Figure S4.** Distribution of the SCOTI score in the derivation cohort. **Figure S5.** Calibration plot for the SCOTI score on the validation cohort. **Figure S6.** Distribution of the SCOTI score in the validation cohort.

## Acknowledgements

None.

## Authors' contributions

JP, ADJ, NM and SJ designed the study. JP and ADJ collected the data. JP, ADJ, NM and SJ analyzed the data. All authors were involved in the data interpretation. JP, ADJ, NM, and SJ wrote the manuscript. All authors read and approved the final manuscript.

## Funding

The funding source had no role in the conception, design, or conduct of the trial, nor did their representatives participate in the collection, management, analysis, interpretation, or presentation of the data or in the preparation, review, or approval of the manuscript.

## Availability of data and materials

Research data and other material (e.g., study protocol and statistical analysis plan) will be made available to the scientific community, immediately on publication, with as few restrictions as possible. All requests should be submitted to the corresponding author who will review with the other investigators for consideration. A data use agreement will be required before the release of participant data and institutional review board approval as appropriate.

## Declarations

### Ethics approval, consent to participate and consent to publication

Not applicable.

### Competing interests

Pr. Jaber reports receiving consulting fees from Drager, Medtronic, Baxter, Fresenius-Xenios, and Fisher & Paykel. Pr. Azoulay has received fees for lectures from Gilead, Pfizer, Baxter, and Alexion. His research group has been supported by Ablynx, Fisher & Paykel, Jazz Pharma, and MSD. Dr. De Jong reports receiving consulting fees from Medtronic. No conflict of interests is reported for other authors.

### Author details

¹Department of Anesthesia and Intensive Care Unit, Regional University Hospital of Montpellier, St-Eloi Hospital, University of Montpellier, PhyMedExp, INSERM U1046, CNRS UMR, 9214 CEDEX 5 Montpellier, France. ²Department of Peri-Operative Medicine, CHU Clermont-Ferrand, 63000 Clermont-Ferrand, France. ³Médecine Intensive et Réanimation, Groupe FAMIREA, Hôpital Saint-Louis, Université de Paris, Paris, France. ⁴IDESP, INSERM, Univ Montpellier, CHU Montpellier, Montpellier, France. ⁵Université de Montpellier, Montpellier, Languedoc-Roussillon, France. ⁶Département d'Anesthésie Réanimation B (DAR B), 80 Avenue Augustin Fliche, 34295 Montpellier, France.

Received: 29 August 2021 Accepted: 15 November 2021

Published online: 27 November 2021

## References

1. Group E-BMW. Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA. 1992;268:2420–5.

2. Gattinoni L, Giomarelli P. Acquiring knowledge in intensive care: merits and pitfalls of randomized controlled trials. Intensive Care Med. 2015;41:1460–4.

3. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. Jama. 1996;276:637–9.

4. Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001;285:1987–91.

5. Montori VM, Guyatt GH. Progress in evidence-based medicine. JAMA. 2008;300:1814–6.

6. Santacruz CA, Pereira AJ, Celis E, Vincent JL. Which multicenter randomized controlled trials in critical care medicine have shown reduced mortality? A systematic review. Crit Care Med. 2019;47:1680–91.

7. Latronico N, Botteri M, Minelli C, Zanotti C, Bertolini G, Candiani A. Quality of reporting of randomised controlled trials in the intensive care literature. A systematic analysis of papers published in Intensive Care Medicine over 26 years. Intensive Care Med. 2002;28:1316–23.

8. Pensier J, de Jong A, Hajjej Z, Molinari N, Carr J, Belafia F, Chanques G, Futier E, Azoulay E, Jaber S. Effect of lung recruitment maneuver on oxygenation, physiological parameters and mortality in acute respiratory distress syndrome patients: a systematic review and meta-analysis. Intensive Care Med. 2019;45:1691–702.

9. Brochard L, Mancebo J, Wysocki M, Lofaso F, Conti G, Rauss A, Simonneau G, Benito S, Gasparetto A, Lemaire F, et al. Noninvasive ventilation for acute exacerbations of chronic obstructive pulmonary disease. N Engl J Med. 1995;333:817–22.

10. Papazian L, Forel JM, Gacouin A, Penot-Ragon C, Perrin G, Loundou A, Jaber S, Arnal JM, Perez D, Seghboyan JM, Constantin JM, Courant P, Lefrant JY, Guérin C, Prat G, Morange S, Roch A. Neuromuscular blockers in early acute respiratory distress syndrome. N Engl J Med. 2010;363:1107–16.

11. Jaber S, Lescot T, Futier E, Paugam-Burtz C, Seguin P, Ferrandiere M, Lasocki S, Mimoz O, Hengy B, Sannini A, Pottecher J, Abback PS, Riu B, Belafia F, Constantin JM, Masseret E, Beaussier M, Verzilli D, De Jong A, Chanques G, Brochard L, Molinari N. Effect of noninvasive ventilation on tracheal reintubation among patients with hypoxemic respiratory
failure following abdominal surgery: a randomized clinical trial. JAMA.
2016;315:1345-53.

12. Annane D, Sébille V, Charpentier C, Bollaert PE, François B, Korach JM, Capellier G, Cohen Y, Azoulay E, Troché G, Chaumet-Riffaud P, Bellissant E. Effect of treatment with low doses of hydrocortisone and fludrocortisone on mortality in patients with septic shock. JAMA. 2002;288:862-71.

13. Bellani G, Laffey JG, Pham T, Fan E, Brochard L, Esteban A, Gattinoni L, van Haren F, Larsson A, McAuley DF, Ranieri M, Rubenfeld G, Thompson BT, Wrigge H, Slutsky AS, Pesenti A. Epidemiology, patterns of care, and mortality for patients with acute respiratory distress syndrome in intensive care units in 50 Countries. JAMA. 2016;315:788-800.

14. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:n7594.

15. Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, Ioannidis JP, Straus S, Thorlund K, Jansen JP, Mulrow C, Catalá-López F, Gøtzsche PC, Dickersin K, Boutron I, Altman DG, Moher D. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med. 2015;162:777-84.

16. Garfield E. The history and meaning of the journal impact factor. JAMA.
2006;295:90-3.

17. Ospina-Tascón GABG, Vincent JL. Multicenter, randomized, controlled trials evaluating mortality in intensive care: doomed to fail? Crit Care Med. 2008;36:1311-22.

18. Harhay MO, Wagner J, Ratcliffe SJ, Bronheim RS, Gopal A, Green S, Cooney E, Mikkelsen ME, Kerlin MP, Small DS, Halpern SD. Outcomes and statistical power in adult critical care randomized trials. Am J Respir Crit Care Med. 2014;189:1469-78.

19. De Jong A, Molinari N, Terzi N, Mongardon N, Arnal JM, Guittou C, Allaouchiche B, Paugam-Burtz C, Constantin JM, Lefrant JY, Leone M, Papazian L, Asehnoune K, Maziers N, Azoulay E, Pradel G, Jung B, Jaber S. Early identification of patients at risk for difficult intubation in the intensive care unit: development and validation of the MACOCHA score in a multicenter cohort study. Am J Respir Crit Care Med. 2013;187:832-9.

20. Leisman DE, Harhay MO, Lederer DJ, Abramson M, Adjei AA, Bakker J, Ballas ZK, Barreiro E, Bell SC, Bellomo R, Bernstein JA, Branson RD, Brusasco V, Chalmers JD, Chokroverty S, Citerio G, Collop NA, Cooke CR, Crapo JD, Donaldson G, Fitzgerald DA, Grainger E, Hale L, Herth FJ, Kochanek PM, Marks G, Moorman JR, Ost DE, Schatz M, Sheikh A, Smyth AR, Stewart I, Stewart PW, Swenson ER, Szymusiak R, Teboul JL, Vincent JL, Wedzicha JA, Maslove DM. Development and Reporting of Prediction Models: Guidance for Authors From Editors of Respiratory, Sleep, and Critical Care Journals. Crit Care Med. 2020;48:623-33.

21. Elliott J, Bodinier B, Bond TA, Chadeau-Hyam M, Evangelou E, Moons KGM, Dehghan A, Muller DC, Elliott P, Tzoulaki I. Predictive Accuracy of a

Polygenic Risk Score-Enhanced Prediction Model vs a Clinical Risk Score for Coronary Artery Disease. JAMA. 2020;323:636-45.

22. Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. BMJ. 2009;338:604.

23. Steyerberg EW, Harrell FE Jr, Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54:774-81.

24. Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol. 2005;58:475-83.

25. Murray SB, Heathers JA, Schauer RM, Griffiths S, Mitchison D, Mond JM, Nagata JM. Postpublication metrics of randomized clinical trials with and without null findings. JAMA. 2019;321:1825-6.

26. Latronico N, Metelli M, Turin M, Piva S, Rasulo FA, Minelli C. Quality of reporting of randomized controlled trials published in Intensive Care Medicine from 2001 to 2010. Intensive Care Med. 2013;39:1386-95.

27. Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, Bero LA. Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust. 2006;184:621-6.

28. Bauchner H, Fontanarosa PB. Randomized Clinical Trials and COVID-19: Managing Expectations. JAMA. 2020;323:2262-3.

29. Harhay MO, Casey JD, Clement M, Collins SP, Gayat É, Gong MN, Jaber S, Laterre PF, Marshall JC, Matthay MA, Monroe RE, Rice TW, Rubin E, Self WH, Mebazaa A. Contemporary strategies to improve clinical trial design for critical care research: insights from the First Critical Care Clinical Trialists Workshop. Intensive Care Med. 2020;46:930-42.

30. Duffett M, Choong K, Hartling L, Menon K, Thabane L, Cook DJ. Randomized controlled trials in pediatric critical care: a scoping review. Crit Care (London, England). 2013;17:R256.

31. Gaudry S, Messika J, Ricard JD, Guillo S, Pasquet B, Dubief E, Boukertouta T, Dreyfuss D, Tubach F. Patient-important outcomes in randomized controlled trials in critically ill patients: a systematic review. Ann Intensive Care. 2017;7:28.

32. Fraser GE, Stram DO. Regression calibration in studies with correlated variables measured with error. Am J Epidemiol. 2001;154:836-44.

33. Okike K, Kocher MS, Mehlman CT, Heckman JD, Bhandari M. Nonscientific factors associated with acceptance for publication in The Journal of Bone and Joint Surgery (American Volume). J Bone Joint Surg Am. 2008;90:2432-7.

**Publisher's Note**

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Submit your manuscript to a **SpringerOpen** journal and benefit from:

▶ Convenient online submission

▶ Rigorous peer review

▶ Open access: articles freely available online

▶ High visibility within the field

▶ Retaining the copyright to your article

Submit your next manuscript at ► springeropen.com