# Evaluation of the survival-inferred fragility index to assess the robustness of the estimated treatment effect on survival endpoints

Roxane Couturier, Sylvie Chevret, Jérôme Lambert

► To cite this version:

Roxane Couturier, Sylvie Chevret, Jérôme Lambert. Evaluation of the survival-inferred fragility index to assess the robustness of the estimated treatment effect on survival endpoints. Contemporary Clinical Trials, 2025, 153, pp.107906. 10.1016/j.cct.2025.107906. hal-05512173

HAL Id: hal-05512173

https://hal.science/hal-05512173v1

Submitted on 16 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

# Evaluation of the survival-inferred fragility index to assess the robustness of the estimated treatment effect on survival endpoints

Roxane Couturier<sup>a, *,</sup> Sylvie Chevret<sup>a, b</sup>, Jérôme Lambert<sup>a</sup>

<figure><img src="image_3.png" /></figure>

## ARTICLE INFO

**Keywords:**

Phase III

Randomized clinical trial

Fragility index

Robustness

Survival endpoints

Simulation

## ABSTRACT

**Background/aims:** Phase III randomized clinical trials (RCTs) aim to evaluate the benefits of a new treatment. When conducted in patients with malignancies, most RCTs use a right censored endpoint, with conclusions regarding efficacy based on the *p* value of the logrank test. Recently, the survival inferred fragility index (SIFI) has been proposed as a measure of the robustness of the treatment effect. Applied to real RCTs, the reported SIFI values were very low. We hypothesized that such values relied on the contamination of the trial. Methods: We performed a simulation study of individuals enrolled in an RCT, generating survival times under several realistic scenarios differing in treatment effects, sample sizes and amount of censoring. Contamination of the sample by individuals with a very specific prognosis was studied.

**Results:** Surprisingly, under the null of no treatment effect, the standard SIFI exhibited very low values, poorly sensitive to the sample size. However, under both the null and the alternative hypotheses, the *p* value and the amount of censoring influenced the value. By contrast, randomly selected patients, rather than those selected at the survival tails, widely modified the results, notably with an impossibility of finding value under the null in a large proportion of cases. When contaminating the sample with individuals with very poor or good outcomes, results were close to those of the standard.

**Conclusions:** The SIFI should not be used as a measure of robustness of survival trials, as it relates to a very specific group of individuals. At least, a random selection of patients should be used in its calculation.

## 1. Background

In oncology, phase III randomized clinical trials (RCTs) enable assessment of the benefits of a new treatment on a right censored endpoint, with analysis of trial data restricted to the study period. To assess the robustness of the reported trial benefits due to a nonnegligible proportion of incomplete data [1,2], it has been recommended to set up sensitivity analyses [3]. To this end, the fragility index (FI) has been proposed as a statistical metric used to assess the extent to which the conclusion of the statistical test assessing the treatment effect can be altered by a change in the outcome of a few individuals. For binary outcomes, it was defined as the number of additional failures in the group with the best outcome that would result in a loss of statistical significance in an exact Fisher test [4]. The smaller the fragility index, the more fragile the result of the study. Conversely, the solidity index (also called the reverse FI) is the minimum number of failures to add to the worst group to cause a statistically nonsignificant result to become

significant [5]. Such indices were then derived for survival endpoints, as the survival inferred fragility index (SIFI) [6]. The SIFI was defined as the minimal number of the longest observations (whether censored or not) that must be switched to the other treatment group to change the trial conclusions based on the *p* value of the logrank test. Four versions of the SIFI that either reassign ('Flip') or copy ('Clone') observations to the opposite group based on the best survivors from the experimental group or the worst survivors from the control group, and conversely, in the case of a nonsignificant trial ('negative SIFI'), have been defined. Retrospectively applied to Phase III oncology RCT findings, only less than 1% of the trial population may overturn the trial conclusions of antiPD1 drugs in oncology [6,7] and of supporting FDA approved anticancer medicines [8], among others, the Flip version using the best survivors producing the smallest values [6].

In the face of these results, we aimed to gain further insights into the SIFI from a statistical perspective, as Potter did regarding the FI [9]. First, no clear justification was given for the selection of these

<footnote><sup>a</sup> ECSTRRA Team IRSL, INSERM UMR 1342, Université Paris Cité, 1 Ave Claude Vellefaux, Paris 75010, France</footnote>

<footnote><sup>b</sup> Institut Universitaire de France (IUF), Paris, France</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail addresses: roxane-.couturier@inserm.fr (R. Couturier), sylvie.chevret@u-paris.fr (S. Chevret), jerome.lambert@u-paris.fr (J. Lambert).</footnote>
<caption>**Table 1**</caption>

GRAAPH-R Trial data: SIFI estimates, using the 4 versions (flip, clone, delete and random), applied to the PFS (significant logrank test favouring A, *p* value = 0.029) or the OS (not significant logrank test, *p* value = 0.26).

<table><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="2">PFS</th><th colspan="2">OS</th></tr><tr><th>SIFI</th><th><i>p</i> value<sup>a</sup></th><th>Negative SIFI</th><th><i>p</i> value<sup>a</sup></th></tr></thead><tbody><tr><td colspan="6"><strong>Available SIFI</strong></td></tr><tr><td rowspan="2">Flip</td><td>Best</td><td>2</td><td>0.053</td><td>-10</td><td>0.042</td></tr><tr><td>Worst</td><td>1</td><td>0.056</td><td>-3</td><td>0.038</td></tr><tr><td rowspan="2">Clone</td><td>Best</td><td>4</td><td>0.053</td><td>-23</td><td>0.046</td></tr><tr><td>Worst</td><td>2</td><td>0.062</td><td>-7</td><td>0.045</td></tr><tr><td colspan="6"><strong>Alternative measures</strong></td></tr><tr><td rowspan="2">Delete SIFI</td><td>Best</td><td>4</td><td>0.052</td><td>-16</td><td>0.049</td></tr><tr><td>Worst</td><td>2</td><td>0.05</td><td>-5</td><td>0.037</td></tr><tr><td rowspan="4">Random SIFI<sup>b</sup></td><td>Flip</td><td>3</td><td></td><td>-52.5<sup>c</sup></td><td></td></tr><tr><td>Clone</td><td>9<sup>d</sup></td><td></td><td>-35<sup>e</sup></td><td></td></tr><tr><td>Delete</td><td>14<sup>f</sup></td><td></td><td>-56<sup>g</sup></td><td></td></tr></tbody></table>

a *p* value of the logrank test at conclusion change.

b Median values from 10,000 replications.

c SIFI not reached in 63.12 % of cases.

d SIFI not reached in 0.007 % of cases.

e SIFI not reached in 99.9 % of cases.

f SIFI not reached in 0.016 % of cases.

g SIFI not reached in 64.7 % of cases.

individuals. In addition to the use of right censored rather than binary outcome data, there are several differences between the SIFI and the FI that deserve attention. Contrary to the FI, which modifies only one cell of a two by two table while the sample sizes of both groups are fixed, the SIFI modifies the structure of the groups by switching individuals from one group to the other. Moreover, given the time scale of the endpoints, modifying one observation from one group could affect all the risk sets from the other group. Therefore, the SIFI could first be considered as a way to assess influential observations on the trial conclusion. However, it does not assess the influence of any individual from the sample as is typically done by the Cook's distance, from the Cox model or flexible survival regression models. By contrast, the SIFI uses only the log-rank test statistics, which measures the extent to which the attribution of statistical significance is subject to individual influences. However, individual influence is not applied randomly, but only to indolent (the best survivors) or aggressive (the worst survivors) forms of the disease, quantifying how much the conclusion changes when switching such individuals from one group to the other. We thus hypothesized that low observed values of SIFI in the medical literature exemplify the individual influence of survival ranks [10], while further assuming trial contamination of those particular patients from one arm to the other.

In this work, we first focused our interests on evaluating the properties of the SIFI. As a secondary objective, we proposed other measures of individual influence, based either on the Cook's distance or alternative measures derived from the SIFI, as specific ways of detecting how imbalances in trial 'contamination' by indolent patients may affect the trial conclusions.

## 2. Motivating example

The interest in measuring the robustness of survival estimated effect was exemplified in the setting of the GRAAPH 2014 trial (ClinicalTrials.gov ID, NCT02611492) (submitted). This RCT aimed to demonstrate the noninferiority of nilotinib combined with a reduced intensity chemotherapy over an intensive chemotherapy arm in terms of molecular response rate for front line treatment of younger adults with Philadelphia positive acute lymphoblastic leukemia (ALL).

A total of 155 patients were allocated to either the intensive (A) or reduced intensity (B) chemotherapy groups. We focused on two secondary endpoints, namely, progression free survival (PFS), that is, time to progression or death, whichever occurred first, and overall survival

(OS). There was a significant difference between both groups in terms of PFS, with an estimated hazard ratio (HR) of event at 0.54 (95 %CI, 0.31; 0.95, *p* = 0.029), with no evidence of any benefit in terms of survival (HR = 0.68, 95 %CI, 0.35; 1.33, *p* = 0.26) (Supplementary Fig. S1).

Standard versions of the SIFI for the PFS endpoint ranged from 1 to 4, while negative SIFI values for OS ranged from 3 to 23 (Table 1). As reported in the literature, the SIFI was lower when using the Flip rather than the Clone approach.

We then considered whether these results could be explained by the influence of individuals in the tails of the survival distribution.

## 3. Methods

Consider a phase III RCT conducted with a sample of *n* patients randomly allocated in a ratio 1:1 to the experimental arm (A) or the control (B). The logrank statistics can be defined as

$$Z_i = \sum_{j=1,...,J} (O_{ij} - E_{ij}) \quad (1)$$

where *i* is the indicator of the group, *j* is the rank of event time, and *O*<sub>*ij*</sub> and *E*<sub>*ij*</sub> are the observed and expected (under the null) numbers of events at the time of the *j*th death, respectively. Under the null, *O*<sub>*ij*</sub> is a hypergeometric variable with mean *E*<sub>*ij*</sub> = *O*<sub>*ij*</sub>*n*<sub>*ij*</sub>/*n*<sub>*j*</sub> and variance *Var*<sub>*ij*</sub> = *E*<sub>*ij*</sub>((*n*<sub>*j*</sub> − *O*<sub>*ij*</sub>)/*n*<sub>*j*</sub>)((*n*<sub>*j*</sub> − *n*<sub>*ij*</sub>)/(*n*<sub>*j*</sub> − 1)), where *n*<sub>*ij*</sub> is the number of exposed individuals at that time in group *i* and *n*<sub>*j*</sub> is the total number of exposed individuals at that time.

### 3.1. Toy example

Let consider a small sample of *n* individuals, with *n*/2 randomly allocated to each group. In the absence of censoring, the *n* survival times can be ordered (assuming no ties), and the logrank statistic reduces to a sum involving 10 terms defined by (*O*<sub>*ij*</sub> − *E*<sub>*ij*</sub>).

Under the null, the occurrence of death does not depend on the group. For simplicity, consider that deaths occurred at distinct times alternatively in arm A, then in arm B, and define the logrank statistic in arm A. For instance, if *n* = 10, the first death occurring in arm A results in a first term given by *E*<sub>A1</sub>=(5/10), and *Var*<sub>A1</sub>=0.5(10−1)/10(10−5) / (10−1) = 0.25. Modifying the allocation of that dead patient by flipping him(her) to the other arm results in modifying only the first term, which becomes null given no death is longer observed in arm A, while *E*<sup>*</sup><sub>A1</sub>=0.4 with *Var*<sup>*</sup><sub>A1</sub>=0.24. Indeed, at time 2, the number of exposed patients in group A is still decreased by 1 while that in group B is unchanged (given the additional death at time 1 has also impacted its sample size at that time). Therefore, all other terms remain unchanged, and the difference in the logrank statistic is given by the difference in the first terms, where *O*<sub>A1</sub> becomes null and *E*<sub>A1</sub> is decreased by (*O*<sub>1</sub>/*n*<sub>1</sub>); similarly, the denominator is impacted by only the differences in *V*<sub>A1</sub>, that is, *E*<sub>A1</sub> and *n*<sub>A1</sub> (given all other terms remain unchanged), with *V*<sub>A1</sub> decreased by *o*<sub>1</sub>/*n*<sub>1</sub> × ((*n*<sub>1</sub> − *o*<sub>1</sub>)/*n*<sub>1</sub> × (2*n*<sub>A1</sub> − *n*<sub>1</sub> − 1)/(*n*<sub>1</sub> − 1)).

The logrank statistics after this single flip (from A to B) can be given by

$$Z_{A_{flip1}} = Z_A - (O_{A1} - E_{A1}) = Z_A - (O_A/n_1 - 1) \quad (2)$$

where *Z*<sub>A</sub> refers to the logrank statistic based on the original data, with its variance given by $V_{A_{flip1}} = V_A + O_1/n_1 \times (n_1 - O_1)/n_1 \times (2n_{A1} - n_1 - 1)/(n_1 - 1)$.

Let *K* be the maximal number of deaths observed in the original sample, with ordered times *t*<sup>(1)</sup> < *t*<sup>(1)</sup> < ... < *t*<sup>(K)</sup>, and let *I*<sub>A</sub> be the indicator that death observed at time *t*<sup>(k)</sup> occurred in arm A. The difference in the logrank statistics could be extended to reach:
$$Z_{A_{flip}} - Z_A = \sum_{k=1}^{SIFI_{A}=1} \left( \frac{O_k}{n_k} - 1 \right) \quad (3)$$

where *k* is the rank of the observed death among all the observed deaths.

To change the conclusion of the logrank test, this means that the modified statistic $Z_{A_{flip}} > q_{1-\alpha/2}V_{A_{flip}}$ where $q_{1-\alpha}$ is the $1 - \alpha$ quantile of the standard normal distribution. It appears that this value mostly depends on the values of *n* and $n_A$ and the number of deaths in arm A over time. The probability of observing a number of switches needed to change the results appears not straightforward to derive; indeed, it depends on the survival distribution, and on the rank of the observed events. In addition, the underlying assumptions depend heavily on how the SIFI is calculated, by selecting only the worst or best survivors, ignoring the treatment group. The relevance of these assumptions may be questioned.

### 3.2. Analogues to Cook's distance

In the setting of right censored data, Cook's analogue methods have been proposed [10–12] when fitting the Cox model $\lambda(t_i|A) = \lambda_0(t)\exp(\beta A_i)$ where $i = 1, \dots, n$ is the index of the *n* individuals.

We applied Cook's distance to standardized martingale residuals, as proposed recently by Tehzeeb Kausar. [13].

### 3.3. Derived SIFI measures

We then proposed alternative ways of detecting individual observations using a modified SIFI framework.

We first considered a Delete approach that consisted of removing rather than flipping or cloning the observations with the longest observation times. The resulting delete SIFI (dSIFI) was defined as the minimal number of deleted observations from one group that achieves the logrank statistic to cross the significance threshold.

Then, we defined the random SIFI (rSIFI) as the number of randomly selected patients from the experimental arm, whatever the rank of their survival times, required to be switched, copied or deleted, to modify the trial conclusion. It was estimated by replicating *N* random samples and then taking the median of the observed values.

## 4. Simulation study

### 4.1. Data generation

We considered a randomized clinical trial with 2 balanced (1:1) parallel arms, A and B, where A denotes the experimental arm and B indicates the control arm.

Let *T* be the right censored endpoint. We generated individual times from a Weibull ($k, \lambda$) distribution, where *k* is the shape parameter and $\lambda$ is the scale parameter.

Several scenarios were considered (Supplementary Table S1) that differ in treatment effect, as measured by the hazard ratio ($HR = \lambda_A/\lambda_B$) and the proportionality of hazards (PH) (allowing $k_A$ and $k_B$ to differ). Fitting a Weibull model to the GRAAPH-R trial data resulted in selecting $\lambda_B=210$ (corresponding to a 60 month survival rate of 0.75) and $\lambda_A=370$ (corresponding to 60 month survival rate of 0.85), while the $k_A$ and $k_B$ values varied according to the underlying assumption of constant hazards in both arms or in only one arm (resulting in non PH).

Specifically, Scenario 1 is based on a treatment effect that is constant over time, Scenario 2a is based on an early treatment effect, and Scenario 2b is based on a delayed treatment effect (Supplementary Table S1). We placed ourselves under either the null, $H_0$, where $\lambda_A = \lambda_B = 210$, or the alternative, $H_1$, where $\lambda_A = 370$ and $\lambda_B = 210$. We fixed the number of events to 131, enabling detection of the HR of 210/370 = 0.57, with the type I and a statistical power set at 90% in Scenario 1a, while likely decreased in situations of nonproportional hazards

<caption>**Table 2**</caption>

Uncensored data: Summary statistics of the SIFI under the null ($H_0$) and the alternative ($H_1$) hypotheses, *n*=132.

<table><thead><tr><th></th><th>SIFI</th><th></th><th></th><th></th><th></th></tr><tr><th></th><th>RH0<sup>a</sup></th><th>Median (IQR)</th><th>Mean (SD)</th><th>Prop-SIFI</th><th>Hazard ratio</th></tr></thead><tbody><tr><td colspan="6"><strong>H<sub>0</sub> Constant hazards, Scenario 1a</strong></td></tr><tr><td>Best</td><td>5.3</td><td>-3</td><td>-2.871 (1.168)</td><td>0.0217</td><td rowspan="10">1.00</td></tr><tr><td>Flip</td><td></td><td>(-4,-2)</td><td></td><td></td></tr><tr><td>Best</td><td>5.3</td><td>-6</td><td>-6.032 (2.248)</td><td>0.0457</td></tr><tr><td>Clone</td><td></td><td>(-8,-4)</td><td></td><td></td></tr><tr><td>Best</td><td>5.3</td><td>-5</td><td>-5.185 (2.5963)</td><td>0.0393</td></tr><tr><td>Delete</td><td></td><td>(-7,-3)</td><td></td><td></td></tr><tr><td>Worst</td><td>5.3</td><td>-9</td><td>-8.172 (3.203)</td><td>0.0619</td></tr><tr><td>Flip</td><td></td><td>(-11,-6)</td><td></td><td></td></tr><tr><td>Worst</td><td>5.3</td><td>-19</td><td>-18.44 (7.730)</td><td>0.1397</td></tr><tr><td>Clone</td><td></td><td>(-25,-13)</td><td></td><td></td></tr><tr><td>Worst</td><td>5.3</td><td>-15</td><td>-14.77 (6.010)</td><td>0.1119</td></tr><tr><td>Delete</td><td></td><td>(-20,-10)</td><td></td><td></td></tr><tr><td colspan="6"><strong>H<sub>1</sub> Constant hazards, Scenario 1a</strong></td></tr><tr><td>Best</td><td>88.9</td><td>2 (1,3)</td><td>2.342 (1.204)</td><td>0.0177</td><td rowspan="10">0.564</td></tr><tr><td>Flip</td><td></td><td></td><td></td><td></td></tr><tr><td>Best</td><td>88.9</td><td>3 (2,4)</td><td>3.231 (1.650)</td><td>0.0245</td></tr><tr><td>Clone</td><td></td><td></td><td></td><td></td></tr><tr><td>Best</td><td>88.9</td><td>6 (4,10)</td><td>7.072 (4.296)</td><td>0.0536</td></tr><tr><td>Delete</td><td></td><td></td><td></td><td></td></tr><tr><td>Worst</td><td>88.9</td><td>8 (5,11)</td><td>8.422 (4.5644)</td><td>0.0638</td></tr><tr><td>Flip</td><td></td><td></td><td></td><td></td></tr><tr><td>Worst</td><td>88.9</td><td>15 (9,22)</td><td>16.35 (9.6990)</td><td>0.1238</td></tr><tr><td>Clone</td><td></td><td></td><td></td><td></td></tr><tr><td>Worst</td><td>88.9</td><td>17 (10,24)</td><td>17.15 (9.1496)</td><td>0.1299</td></tr><tr><td>Delete</td><td></td><td></td><td></td><td></td></tr><tr><td colspan="6"><strong>Early events, Scenario 1b</strong></td></tr><tr><td>Best</td><td>61</td><td>1 (1,2)</td><td>1.661 (0.881)</td><td>0.0126</td><td rowspan="10">0.67</td></tr><tr><td>Flip</td><td></td><td></td><td></td><td></td></tr><tr><td>Best</td><td>61</td><td>2 (1,3)</td><td>2.233 (1.288)</td><td>0.0169</td></tr><tr><td>Clone</td><td></td><td></td><td></td><td></td></tr><tr><td>Best</td><td>61</td><td>4 (2,6)</td><td>4.52 (3.229)</td><td>0.0342</td></tr><tr><td>Delete</td><td></td><td></td><td></td><td></td></tr><tr><td>Worst</td><td>61</td><td>5 (3,8)</td><td>5.561 (3.655)</td><td>0.0930</td></tr><tr><td>Flip</td><td></td><td></td><td></td><td></td></tr><tr><td>Worst</td><td>61</td><td>9 (4,15)</td><td>10.39 (7.515)</td><td>0.0787</td></tr><tr><td>Clone</td><td></td><td></td><td></td><td></td></tr><tr><td>Worst</td><td>61</td><td>10 (5,16)</td><td>11.31 (7.590)</td><td>0.0857</td></tr><tr><td>Delete</td><td></td><td></td><td></td><td></td></tr><tr><td colspan="6"><strong>Delayed events, Scenario 1c</strong></td></tr><tr><td>Best</td><td>100</td><td>6 (5,8)</td><td>6.493 (1.607)</td><td>0.0492</td><td rowspan="10">0.32</td></tr><tr><td>Flip</td><td></td><td></td><td></td><td></td></tr><tr><td>Best</td><td>100</td><td>8 (7,10)</td><td>8.415 (1.839)</td><td>0.0638</td></tr><tr><td>Clone</td><td></td><td></td><td></td><td></td></tr><tr><td>Best</td><td>100</td><td>22 (18,26)</td><td>22.13 (5.841)</td><td>0.1677</td></tr><tr><td>Delete</td><td></td><td></td><td></td><td></td></tr><tr><td>Worst</td><td>100</td><td>23 (19,26)</td><td>22.63 (4.784)</td><td>0.1714</td></tr><tr><td>Flip</td><td></td><td></td><td></td><td></td></tr><tr><td>Worst</td><td>100</td><td>48 (40,56)<sup>b</sup></td><td>47.4 (10.523)</td><td>0.3247</td></tr><tr><td>Clone</td><td></td><td></td><td></td><td></td></tr><tr><td>Worst</td><td>100</td><td>43 (38,48)</td><td>42.47 (7.213)</td><td>0.3217</td></tr></tbody></table>

(scenarios 1b and 1c).

First, no censoring was considered. Then, random non informative censoring was simulated according to a uniform distribution [0, $\tau$], where $\tau$ was fixed to reach an average censoring percentage varying from 0% to 80%, while increasing the sample size accordingly (Table S1). We also considered informative censoring, which was generated by randomly selecting one half of the censored observations in the right tail of the survival distribution. Last, we considered a situation where the population is contaminated by another population of either very good or very poor outcomes. For the sake of simplicity, we considered non censored data. Contaminants were generated from a binomial distribution with the overall proportion of contaminants ranging from 0 to 0.95 with a step size of 0.01. Survival time for each patient from this

<footnote><sup>a</sup> % Rejection of the null hypothesis.</footnote>

<footnote><sup>b</sup> SIFI not reached in 959 cases.</footnote>
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Uncensored data under H0: Distribution of the six SIFI versions as a function of *p* – value based on 10,000 replications of Scenario 1a (exponential survival times, constant hazards), sample size, *n*=132.</figcaption></figure>

contaminated population was then modified to mimic ‘indolent’ or ‘aggressive’ forms of the disease. Indolent forms were defined by a survival time randomly selected from a uniform distribution over the 3rd quartile of patient survival times and the maximum time for the best responders. The aggressive cases were generated from a uniform distribution from 0 to the first quartile of patient survival times. Sensitivity analyses were conducted to assess the impact of sample size *n* under the null hypothesis (from *n* = 40 to 640).

## 4.2. Computations of the SIFI

In each sample from each scenario, we recorded the *p* value of the logrank test, and the hazard ratio (HR). We then computed Cook’s distance applied to the standardized residuals, and the 3 versions (flip, clone and delete) of the SIFI, using either the best or the worst survivors, and the random SIFI. Last, we computed the number of contaminated patients required to change the conclusion of the logrank test. This was referred to as the contaminated SIFI (cSIFI).

A total of *N* = 10,000 independent replications were run, from which were obtained the SIFI, rSIFI and cSIFI distributions, summarized by the median, first (Q1) and third (Q3) quartiles, mean, and standard deviation (SD). The proportion of SIFI relative to the sample size (Prop SIFI = $\frac{\sum_{i=1}^{N} SIFI_i}{\sum_{i=1}^{N} n_i}$) and to the number of censored observations (PropCens SIFI) were also computed. For each scenario, the empirical estimate of power was also calculated by the proportion of rejections of the null based on the log-rank test. All simulation scripts have been reported in the Supplementary materials. The data and computer code used to analyze the motivating example, the computer code used to create and analyze the simulated data sets, and the computer code used to plot the figures and tables in the manuscript (including Additional files) are

available in the github repository: https://github.com/roxcouturier/SIFI_projet/.

## 4.3. Results of the standard survival inferred fragility index

### 4.3.1. No censoring

We first considered uncensored survival data.

The Cook’s distance indicated a considerable influence of subjects at the tails of the survival curve (Supplementary Fig. S3). Patients with the strongest individual influences were those who experienced an event the latest, followed by those who experienced it early, and finally by those who experienced it in the middle of the follow up.

Under the null, in Scenario 1a, the negative SIFI based on the flip of the best survivors resulted in the conclusions changing more quickly (with values down to 4 % of the sample, median between −3 and −6) than the clone and delete SIFI (Table 2).

As expected, the closer the *p* value to 0.05, the lower the absolute value of the negative SIFI (Fig. 1). When generating proportional hazards data from Weibull rather than exponential distributions (Scenario 1b or 1c), the results were unchanged (Supplementary Table S2). Similarly, the SIFI were weakly affected by sample size (Fig. 2). For example, when the best patients in the control arm were flipped into the experimental arm, the SIFI mostly ranged from −1 and −5. By contrast, the proportion SIFI decreased from 0.0596 (with *n* = 40) to 0.0067 (with *n* = 640). Table S3 highlights an increased variability in SIFI value depending on sample size. This variability becomes more pronounced as the sample size increases: with 40 patients, the standard deviation of SIFI values ranges from 1.030 to 5.336, while with 640 patients, it extends from 1.761 to 14.374; this is also observed in the interquartile range: with 40 patients, the interquartile range of SIFI values ranges from 1 to 8, while with 640 patients, it extends from 3 to 22.
<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Uncensored data under H0: Distribution of the six SIFI versions as a function of sample size *n* (from 40 to 640), based on 10,000 replications of Scenario 1a (exponential survival times, constant hazards).</figcaption></figure>

Under the alternative, the observed values of the SIFI were still low, with the best flip more likely to provide a quick change in the study conclusion, similarly to the scenario under the null (Table 2). When the proportional hazards assumption was violated, the number of patients required to change the test conclusion increased approximately twofold when delayed effects were observed and conversely, was divided by 2 in the case of early effects (Supplementary Table S4). In any case, regardless of the data, the standard deviation of the SIFI was somewhat large (Table 2).

### 4.3.2. With censoring

Under the null, the greater the proportion of censored observations, the higher the SIFI: the median ranged between -3 and -8 with 10% censoring and up to -20 and -48 with 80%. When expressed relative to the sample size, SIFI exhibited a similar and low proportion of observations (between 2 and 6%), regardless of the censoring percentage. On the other hand, the SIFI values for the methods with the worst responders showed similar values regardless of the percentage of censored observations but corresponding to a lower proportion relative to the sample size, (from 5.5 to 10% with 10% censoring versus 1.1 and 2.37 % with 80% censoring) (Table 3).

Under the alternative, the results were close: the SIFI increased with the proportion of censoring, was unaffected by the sample size, while the *propSIFI* decreased. Violation of the PH assumption led to an increase in the SIFI of the best responders in Scenario 2a; the SIFI of the worst responders increased in Scenario 2a, but decreased in Scenario 2b (Supplementary Table S8). When introducing informative censoring, results were only slightly modified (Supplementary Table S5).

### 4.4. Results of the random survival-inferred fragility index

Under the null, it was much more difficult to change the test

conclusion when selecting patients uniformly over the sample rather than from the tails of the survival distribution. In 53.7% to 97.3% of the tests, the conclusions could not even be reversed (Supplementary Table S6). The results were close whether starting with the experimental group or the control group. By contrast to the standard SIFI, the random-SIFI was impacted by the sample size (Fig. 3) even if, as stated above, in a large number of cases, the test conclusions did not change, especially when using the Clone version (SIFI was not reached in approximately 97 % of cases).

Under the alternative, the number of tests that achieved no change in conclusion was much lower, with 0.07% (Flip) to 23.3% (Clone) of cases for which the random-SIFI was not identified. However, when the random-SIFI could be found, the values obtained with *rSIFI* were expectedly higher than with the standard SIFI, with median between 22 and 48 (Supplementary Table S6).

### 4.5. Results of the contaminated survival-inferred fragility index

When the contaminated (c)-SIFI was computed under the null, results were of the same magnitude as those obtained with the standard SIFI for the best versions (median 4 compared with 3 to 6) and somewhat lower for the worst versions (median 6 compared with 9 to 19). Indeed, it takes approximately 4 contaminated patients with the best follow-up times or 6 contaminated patients with the poorest follow-up times to reach significance under the null hypothesis based on samples of 132 individuals with no censoring (Supplementary Table S7).

Under the alternative, in line with the results under the null, cSIFI was approximately at 3 for the best responders and 5 for the worst, whereas for standard versions, the median SIFI ranged between 2 and 6 for the best and 8 and 17 for the worst.
<caption>**Table 3**</caption>

Censored data: Estimated values of the six versions of the SIFI, in Scenario 1a (exponential survival times, constant hazards in both arms) under the null ($H_0$) or alternative ($H_1$) hypotheses.

<table><thead><tr><th></th><th>RHO<sup>a</sup></th><th>Median (IQR)</th><th>Mean (SD)</th><th>PropSIFI</th><th>PropCens</th></tr></thead><tbody><tr><td colspan="6"><strong>H<sub>0</sub></strong></td></tr><tr><td colspan="6"><strong>10 % censored data</strong></td></tr><tr><td>Best Flip</td><td>5.2</td><td>-3 (-4,-2)</td><td>-3.3763 (1.3612)</td><td>0.0228</td><td>0.1836</td></tr><tr><td>Best Clone</td><td>5.2</td><td>-8 (-10,-5)</td><td>-7.4062 (2.9838)</td><td>0.05</td><td>0.4027</td></tr><tr><td>Best Delete</td><td>5.2</td><td>-6 (-8,-4)</td><td>-5.9211 (2.841)</td><td>0.04</td><td>0.322</td></tr><tr><td>Worst Flip</td><td>5.2</td><td>-9 (-11,-6)</td><td>-8.0563 (3.1791)</td><td>0.0544</td><td>0.4381</td></tr><tr><td>Worst Clone</td><td>5.2</td><td>-20 (-25,-13)</td><td>-18.7228 (7.9458)</td><td>0.1265</td><td>2.2668</td></tr><tr><td>Worst Delete</td><td>5.2</td><td>-15 (-19,-10)</td><td>-14.4852 (5.9082)</td><td>0.0979</td><td>0.7876</td></tr><tr><td colspan="6"><strong>40 % censored data</strong></td></tr><tr><td>Best Flip</td><td>5.1</td><td>-6 (-7,-4)</td><td>-5.6804 (2.3151)</td><td>0.0258</td><td>0.0603</td></tr><tr><td>Best Clone</td><td>5.1</td><td>-13 (-17,-9)</td><td>-12.9944 (5.5371)</td><td>0.0591</td><td>0.1379</td></tr><tr><td>Best Delete</td><td>5.1</td><td>-10 (-13,-7)</td><td>-9.8276 (4.3522)</td><td>0.0447</td><td>0.1043</td></tr><tr><td>Worst Flip</td><td>5.1</td><td>-8 (-10,-6)</td><td>-7.7675 (2.9889)</td><td>0.0353</td><td>0.0824</td></tr><tr><td>Worst Clone</td><td>5.1</td><td>-18 (-23,-12)</td><td>-17.0521 (6.9444)</td><td>0.0775</td><td>0.181</td></tr><tr><td>Worst Delete</td><td>5.1</td><td>-15 (-18,-10)</td><td>-13.8644 (5.4848)</td><td>0.063</td><td>0.1471</td></tr><tr><td colspan="6"><strong>80 % censored data</strong></td></tr><tr><td>Best Flip</td><td>4.8</td><td>-20 (-16,-14)</td><td>-19.4862 (8.1327)</td><td>0.0295</td><td>0.0359</td></tr><tr><td>Best Clone</td><td>4.8</td><td>-48 (-62,-31)</td><td>-45.9601 (20.3516)</td><td>0.0696</td><td>0.0846</td></tr><tr><td>Best Delete</td><td>4.8</td><td>-35 (-45,-24)</td><td>-34.0617 (14.3401)</td><td>0.0516</td><td>0.0627</td></tr><tr><td>Worst Flip</td><td>4.8</td><td>-8 (-10,-5)</td><td>-7.3024 (2.7502)</td><td>0.0111</td><td>0.0134</td></tr><tr><td>Worst Clone</td><td>4.8</td><td>-16 (-21,-11)</td><td>-15.6218 (6.1343)</td><td>0.0237</td><td>0.0288</td></tr><tr><td>Worst Delete</td><td>4.8</td><td>-14 (-17,-9)</td><td>-12.9204 (5.0128)</td><td>0.0196</td><td>0.0238</td></tr><tr><td colspan="6"><strong>H<sub>1</sub></strong></td></tr><tr><td colspan="6"><strong>10 % censored data</strong></td></tr><tr><td>Best Flip</td><td>73</td><td>2(1,4)</td><td>2.6337 (1.5394)</td><td>0.0178</td><td>0.1394</td></tr><tr><td>Best Clone</td><td>73</td><td>4(2,6)</td><td>4.2354 (2.7152)</td><td>0.0286</td><td>0.2242</td></tr><tr><td>Best Delete</td><td>73</td><td>5 (3,8)</td><td>5.9427 (4.0184)</td><td>0.0402</td><td>0.3146</td></tr><tr><td>Worst Flip</td><td>73</td><td>6 (3,9)</td><td>6.7307 (4.2227)</td><td>0.0455</td><td>0.3564</td></tr><tr><td>Worst Clone</td><td>73</td><td>11 (6,18)</td><td>12.7564 (8.718)</td><td>0.0862</td><td>0.6754</td></tr><tr><td>Worst Delete</td><td>73</td><td>13 (7,20)</td><td>13.7953 (8.744)</td><td>0.0932</td><td>0.7304</td></tr><tr><td colspan="6"><strong>40 % censored data</strong></td></tr><tr><td>Best Flip</td><td>80.5</td><td>5 (3,7)</td><td>5.2839 (3.2928)</td><td>0.024</td><td>0.0582</td></tr><tr><td>Best Clone</td><td>80.5</td><td>9 (5,14)</td><td>10.4658 (7.1787)</td><td>0.0476</td><td>0.1154</td></tr><tr><td>Best Delete</td><td>80.5</td><td>9 (5,14)</td><td>10.1051 (6.48)</td><td>0.0459</td><td>0.1114</td></tr><tr><td>Worst Flip</td><td>80.5</td><td>7(4,10)</td><td>7.5106 (4.5526)</td><td>0.0341</td><td>0.0828</td></tr><tr><td>Worst Clone</td><td>80.5</td><td>13 (7,20)</td><td>14.0841 (9.1856)</td><td>0.064</td><td>0.1552</td></tr><tr><td>Worst Delete</td><td>80.5</td><td>14 (8,22)</td><td>15.552 (9.5521)</td><td>0.0707</td><td>0.1714</td></tr><tr><td colspan="6"><strong>80 % censored data</strong></td></tr><tr><td>Best Flip</td><td>85.5</td><td>18 (10,28)</td><td>20.2438 (12.8365)</td><td>0.0307</td><td>0.0379</td></tr><tr><td>Best Clone</td><td>85.5</td><td>41(22,67)</td><td>47.3182 (33.5186)</td><td>0.0717</td><td>0.0886</td></tr><tr><td>Best Delete</td><td>85.5</td><td>33 (19,49)</td><td>35.5837 (21.6625)</td><td>0.0539</td><td>0.0666</td></tr><tr><td>Worst Flip</td><td>85.5</td><td>7 (4,11)</td><td>7.9014 (4.5247)</td><td>0.012</td><td>0.0148</td></tr><tr><td>Worst Clone</td><td>85.5</td><td>13 (8,20)</td><td>14.3723 (8.623)</td><td>0.0218</td><td>0.0269</td></tr><tr><td>Worst Delete</td><td>85.5</td><td>16 (9,23)</td><td>16.6507 (9.8045)</td><td>0.0252</td><td>0.0312</td></tr></tbody></table>

## 5. Reanalysis of the GRAAPH 2014 trial data

The subjects with the earliest events were those with the greatest individual influence, as measured by the Cook's distance. Given the large amount of censoring at the end of the trial, subjects at the right tail of the curve have less influence. We then used these most influential patients to be flipped/cloned/deleted from the study one by one until the test conclusions changed. The results show that it takes the same number of patients to change the conclusions as it does when using SIFI flip/clone/delete (Supplementary Fig. S2).

The rSIFI for this trial was finally calculated based on 10,000 independent replications. Randomly selected patients modified the conclusions of the Flip (median = 3), Clone (median = 9), and Delete (median = 14) versions. However, a very large number of tests (62.42–99.8 %) did not change the test conclusion. In the remainder, rSIFI needed between 32 and 55 patients to change the test’s conclusions. These results coincide with the simulations that with the random SIFI method, the conclusions of a nonsignificant test could not easily be switched to significance.

## 6. Discussion and conclusion

There has recently been growing interest in providing robustness measures of treatment effects in RCT with the proposal of a fragility index (FI) for trials with binary endpoints. However, important drawbacks of the FI were recently noted as being confusingly reported as measures of trial robustness that inappropriately penalize small trials [9]. Similarly, we wondered which parameters influence the value of its extension to survival studies, the fragility inferred by the survival index (SIFI) [6], and to what extent it tells us about the robustness of a study and how to interpret its value. The results of the simulation study show that the SIFI was not affected by the treatment effect, either constant, early or delayed, but was strongly affected by the amount of right censoring; however, expressing the SIFI as a proportion of observations eliminated this effect.

We first questioned the meaning of robustness reached by the SIFI. Indeed, robustness is usually defined as the capacity of a procedure to produce unbiased results when small changes in the experimental conditions are made voluntarily [14]. However, there was no explicit mention of any assumptions to be checked by the SIFI. Nevertheless, the SIFI appears to be directed towards a very particular assumption

<footnote><sup>a</sup> % Rejection of the null hypothesis.</footnote>
<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Uncensored data under H0: Distribution of the three versions of the random-SIFI as a function of sample size *n* (from 40 to 640), based on 10,000 replications of Scenario 1a (exponential survival times, constant hazards).</figcaption></figure>

regarding individual influence. First, by switching or copying the patients who either died first or survived the longest, the SIFI evaluates the robustness of the logrank test to the presence of such patients. Second, by modifying the group of treatment of these patients, the SIFI further assumes that their outcomes are not affected by any treatment. We hypothesized that the SIFI could thus be referred to as a measure of robustness towards the ‘contamination’ of patients with the poorest (or the best, respectively) outcomes, irrespective of treatment actually given.

To confirm this hypothesis, we first considered the influential individuals, as detected by the Cook’s distance applied to right censored outcomes. In the absence of censoring, the SIFI reached low values, and we showed that these individuals were those with the largest Cook’s distance. In the presence of censoring, as observed in the Graaph study, the subjects with the earliest events were those with the greatest individual influence, explaining the very low SIFI values based on the worst survivors. This result confirmed that the patient’s rank in the survival distribution widely influences the SIFI, in line with previous results exhibiting the influence of best survivors in the Cox model [10]. The importance of survival rank on individual influence measures was finally confirmed by the simulation study.

As an alternative and simple way to measure the trial robustness towards individuals, we proposed a random SIFI. Expectedly, it reached much larger values than the original methods; notably, under the null, the absence of reversal of trial conclusion was observed in most cases (Supplementary Table S6). Note that in our dataset, the Flip random SIFI was still low; this was explained by the distribution of observations times, which were imbalanced towards the right due to the large amount of administrative censoring. In other words, random selection achieved an over-representation of long survivors in the selected samples, resulting in an rSIFI close to the standard Flip SIFI. This result warns against the use of such measures in the case of heavily administratively

censored data. There was no attempt to define which values of the random SIFI would be acceptable for a robust trial. Actually, the rSIFI depends on the trial settings, notably the sample size (Fig. 3) and/or *p* value. This contrasts with the standard version of the SIFI, which is deeply influenced by the *p* value of the original trial (Fig. 1) but increased very slowly with the sample size, in line with a previous report [6]. In addition to the sample size, we considered the influence of the proportional hazards assumption on the SIFI. As shown in Table 2, under the alternative, the lowest values of SIFI were observed in the case of early treatment effects, while delayed effects somewhat increased the SIFI value.

Some recommendations can be made. First, the SIFI measures only the extent to which the statistical significance of the log-rank test is subject to individual influences. Therefore, its use in contexts where statistical power is either very low or very high raises additional concerns. Moreover, when the results have no clinical relevance due to low power or small effect size, interpreting the robustness of the estimates raises difficulty. This is in line with the debate regarding the clinical relevance of treatment effect in comparison to statistical significance. Second, it was little affected by effect size or sample size, but strongly affected by the censoring rate, which may cause additional difficulties in its interpretation. Finally, the SIFI seems to address mainly the problem of study contamination by patients who have very poor or very good outcomes independent of the actual treatment effect. This seems to be a very restrictive setting compared to the broad “robustness” evaluation that was initially claimed.

Our study has some limitations. We considered only a few scenarios, mimicking our motivating trial, while other settings could have been studied. Our paper focused only on SIFI and did not provide insight into alternative robustness measures of the survival effect. This was mainly because other approaches addressed a specific feature of the data, either long-term assessments in the presence of extreme observations [15],
contaminants [16], or distributional shifts [17] In addition, most robust approaches used statistics derived from the Cox model or flexible parametric survival models, with criteria corresponding to classical - and D-optimality, rather than based on survival estimates or log-rank p-values. Other approaches, such as the tipping-point method, could also have been considered [18]. However, our aim was to obtain further insights into the SIFI rather than to compare various approaches that may address different aspects of robustness. Similarly, we did not consider the limitation of using the *p* value of a statistical test as a measure of robustness, while many papers have recently noted the limits of such measures of effect [19,20]. We considered only RCT data, whereas the robustness of treatment effects in survival analyses of observational data is also of interest. This would require further investigation, as the test should be applied to either matched or weighted data sets to address confounding by indication bias. Finally, no clearcut probabilistic interpretation of the SIFI can be derived, that limits the metric relevance and practical value of the SIFI.

In summary, we focused our interests on the proposed SIFI when measuring the robustness of survival trials, in the sense of the extent to which the conclusion of the study can be affected by a few individuals. This approach addresses an interesting area of research that has long been poorly considered in the survival setting. For all these reasons, we do not recommend using the SIFI as a measure of the robustness of the trial.

## CRediT authorship contribution statement

**Roxane Couturier:** Writing – review & editing, Writing – original draft, Visualization, Validation, Software, Methodology, Investigation, Formal analysis, Conceptualization.
**Sylvie Chevret:** Writing – review & editing, Writing – original draft, Supervision, Project administration, Methodology.
**Jérôme Lambert:** Writing – review & editing, Writing – original draft, Supervision, Conceptualization.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.cct.2025.107906.

## Data availability

The authors do not have permission to share data.

## References

[1] G.H. Guyatt, M. Briel, P. Glasziou, D. Bassler, V.M. Montori, Problems of stopping trials early, BMJ 344 (2012) e3863.

[2] Yu Huiyao Huang, Yue Yu Tang, Yu Anqi, Wu Dawei, Hong Fang, Shuhang Wang, Chao Sun, Xin Wang, Qi Fan, Yuan Fang, Qiyu Tang, Ning Jiang, Du Jingting, Huilei Miao, Ying Bai, Peiwen Ma, Shujun Xing, Dandan Cui, Shuangman Miao, Yale Jiang, Jingxiao Zhu, Qi Zhu, Ye Leng, Lan Wei Guo, Shanmei Liao, Yaguang Shao, Yinyin Song, Zeyuan Liu, Minghuang Hong, Suxia Luo, Xu Binghe, Gongtao Lan, Ning Li, The reliability and integrity of overall survival data based on follow-up records only and potential solutions to the challenges, Lancet Reg. Health West Pac. 31 (2023) 100624.

[3] European Union European Medicines Agency. Guideline on missing data in confirmatory clinical trials.

[4] Michael Walsh, Sadeesh K. Srinathan, Daniel F. McAuley, Marko Mrkobrada, Oren Levine, Christine Ribic, Amber O. Molnar, Neil D. Dattani, Andrew Burke, Gordon Guyatt, Lehana Thabane, Stephen D. Walter, Janice Pogue, P.J. Devereaux, The statistical significance of randomized controlled trial results is frequently fragile: a case for a fragility index, J. Clin. Epidemiol. 67 (6) (2014) 622–628.

[5] Muhammad Shahzeb Khan, Gregg C. Fonarow, Tim Friede, Noman Lateef, Safi U. Khan, Stefan D. Anker, Frank E. Harrell, Javed Butler, Application of the reverse fragility index to statistically nonsignificant randomized clinical trial results, JAMA Netw. Open 3 (8) (2020) e2012469.

[6] David Bomze, Nathanel Asher, Omar Hasan Ali, Lukas Flatz, Daniel Azoulay, Gal Markel, Tomer Meirson, Survival-inferred fragility index of phase 3 clinical trials evaluating immune checkpoint inhibitors, JAMA Netw. Open 3 (10) (2020) e2017675.

[7] Laith Al-Showbaki, Fahad A. Almugbel, Husam A. Alqaisi, Eitan Amir, Eric X. Chen, Optimism bias in the design of phase III randomized control trials evaluating PD-1/ PD-L1 targeting monoclonal antibodies, Oncologist 27 (6) (2022) 487–492.

[8] Joseph C. Del Paggio, Ian F. Tannock, The fragility of phase 3 trials supporting FDA-approved anticancer medicines: a retrospective analysis, Lancet Oncol. 20 (8) (2019) 1065–1069.

[9] Gail E. Potter, Dismantling the fragility index: a demonstration of statistical reasoning, Stat. Med. 39 (26) (2020-11-20) 3720–3731.

[10] Matthieu Resche-Rigon, Sylvie Chevret, Local influence for the subdistribution of a competing risk, Stat. Med. 25 (11) (2006) 1937–1947.

[11] K.C. Cain, N.T. Lange, Approximate case influence for the proportional hazards regression model with censored data, Biometrics 40 (2) (1984) 493–499.

[12] William E. Barlow, Global measures of local influence for proportional hazards regression models, Biometrics 53 (3) (1997) 1157–1162.

[13] Atif Akbar Tehzeeb Kausar, Muhammad Qasim, Influence diagnostics for the cox proportional hazards regression model: method, simulation and applications, J. Stat. Comput. Simul. 93 (10) (2023) 1580–1600.

[14] J.M. Bosque-Sendra, M. Nechar, L. Cuadros Rodríguez, Decision protocol for checking robustness with previous outlier detection in the validation of analytical methods, Fresenius J. Anal. Chem. 365 (6) (1999) 480–488.

[15] M. Sadiq, T. Mehmood, A flexible and robust approach to analyze survival systems in the presence of extreme observations, Math. Probl. Eng. 2021 (1) (2021) 9927377.

[16] M. Konstantinou, S. Biedermann, A. Kimber, Model robust designs for survival trials, Comput. Stat. Data Anal. 113 (2017) 239–250.

[17] S. Fan, R. Xu, Q. and Dong., Stable cox regression for survival analysis under distribution shifts, Nat. Mach. Intell. 6 (2024) 1525–1541.

[18] Bohdana Ratitch, Michael O’Kelly, Robert Tosiello, Missing data in clinical trials: from clinical assumptions to statistical analysis using pattern mixture models, Pharm. Stat. 12 (6) (2013) 337–347.

[19] Valentin Amrhein, Sander Greenland, Blake McShane, Scientists rise up against statistical significance, Nature 567 (2019) 305–307.

[20] Blakeley B. McShane, David Gal, Andrew Gelman, Robert Christian, Jennifer L. Tackett, Abandon statistical significance, Am. Stat. 73 (S1) (2019) 235–245.