# Combining green cards, telephone calls and postcards into an intervention algorithm to reduce suicide reattempt (algos): p-hoc analyses of an inconclusive randomized controlled trial

Antoine Messiah, Anne-Laure Demarty, Stéphane Duhem, Guillaume Vaiva

► To cite this version:

Antoine Messiah, Anne-Laure Demarty, Stéphane Duhem, Guillaume Vaiva. Combining green cards, telephone calls and postcards into an intervention algorithm to reduce suicide reattempt (algos): p-hoc analyses of an inconclusive randomized controlled trial. PLoS ONE, 2019, 14, pp.e0210778. 10.1371/journal.pone.0210778. [https://doi.org/10.1371/journal.pone.0210778](https://doi.org/10.1371/journal.pone.0210778). [hal-02439764](https:// HAL Id: )

Submitted on 27 Mar 2020

**HAL** is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire **HAL**, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
RESEARCH ARTICLE

# Combining green cards, telephone calls and postcards into an intervention algorithm to reduce suicide reattempt (AlgoS): P-hoc analyses of an inconclusive randomized controlled trial

Antoine Messiah<sup>1*</sup>, Charles-Edouard Notredame<sup>2,3</sup>, Anne-Laure Demarty<sup>4,5</sup>, Stéphane Duhem<sup>4,5</sup>, Guillaume Vaiva<sup>1,2,5</sup>, on behalf of AlgoS investigators<sup>¶</sup>

1 INSERM Research unit U-1178 "Mental Health and Public Health", Centre de recherche en Épidémiologie et Santé des Populations (CESP), INSERM, Université Paris-Sud, UVSQ, Université Paris-Saclay, Villejuif, France, 2 SCALab Laboratory, CNRS, UMR 9193, Universités de Lille, Lille, France, 3 Department of Child and Adolescent Psychiatry, Hôpital Fontan, CHRU Lille, Lille, France, 4 Inserm Clinical Investigation Center (CIC) 1403, CHRU and Universités de Lille, Lille, France, 5 Department of Adult Psychiatry, Hôpital Fontan, CHRU Lille, Lille, France

¶ AlgoS investigators affiliation and names are provided in the Acknowledgments

* antoine.messiah@inserm.fr

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

OPEN ACCESS

**Citation:** Messiah A, Notredame C-E, Demarty A-L, Duhem S, Vaiva G, on behalf of AlgoS investigators (2019) Combining green cards, telephone calls and postcards into an intervention algorithm to reduce suicide reattempt (AlgoS): P-hoc analyses of an inconclusive randomized controlled trial. PLoS ONE 14(2): e0210778. https://doi.org/10.1371/journal.pone.0210778

**Editor:** Natalie Riblet, VA Medical Center, UNITED STATES

**Received:** December 13, 2017

**Accepted:** January 1, 2019

**Published:** February 1, 2019

**Copyright:** © 2019 Messiah et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

**Data Availability Statement:** All relevant data are within the paper and its Supporting Information files.

**Funding:** This study received a Hospital Clinical Research Grant (PHRC 2009-1902) from the French Health Ministry. None of the funding sources had any involvement in the study design, the collection, analysis or interpretation of data, the

## Abstract

## Background

Brief contact interventions (BCIs) might be reliable suicide prevention strategies. BCI efficiency trials, however, gave equivocal results. AlgoS trial is a composite BCI that yielded inconclusive results when analyzed with Intention-To-Treat strategy. In order to elicit intervention strengths and weaknesses, post-hoc analyses of AlgoS data were performed.

## Methods

AlgoS was a randomized controlled trial conducted in 23 French hospitals. Suicide attempters were randomly assigned to either the intervention group (AlgoS) or the control group (Treatment as usual TAU). In the AlgoS arm, first-time suicide attempters received crisis cards; non first-time suicide attempters received a phone call, and post-cards if the call could not be completed, or if the participant was in crisis and/or non-compliant with the post-discharge treatment. An As Treated strategy, accounting for the actual intervention received, was combined with subgroup analyses.

## Results

1,040 patients were recruited and randomized into two groups of N = 520, from which 53 withdrew participation; 15 were excluded after inclusion/exclusion criteria reassessment. AlgoS first attempters were less likely to reiterate suicide attempt (SA) than their TAU counterparts at 6 and 13–14 months (RR [95% CI]: 0.46 [0.25–0.85] and 0.50 [0.31–0.81] respectively). AlgoS non-first attempters had similar SA rates as their TAU counterparts at 6 and
writing of the report or the decision to submit the
paper for publication.

**Competing interests:** The authors have declared
that no competing interests exist.

13–14 months (RR [95% CI]: 0.84 [0.57–1.25] and 1.00 [0.73–1.37] respectively). SA rates
were dissimilar within the AlgoS non-first attempter group.

Conclusions

This new set of analysis suggests that crisis cards could be efficacious to prevent new SA
attempts among first-time attempters, while phone calls were probably not significantly effi-
cacious among multi-attempters. Importantly, phone calls were informative of new SA risk,
thus a key component of future interventions.

**Introduction**

With a toll nearing one million per year, suicide is the 17th leading cause of death worldwide, and the 10th cause of death for adults 18–40 years old [1,2]. Suicide prevention is now consid-
ered a global public-health priority, and large-scale multimodal programs are deemed the
most efficient strategy to adopt [1]. Brief Contact Interventions (BCIs) in the immediate after-
math of suicide attempts (SAs) were identified as relevant pieces to integrate into such pro-
grams [3]. BCIs encompass a range of timely interventions designed to help patients in coping
with prospective suicide crisis, notably by maintaining connection with the health care system.
From a public health perspective, BCIs are particularly appropriate because (1) they target one
of the most highly at-risk population, since SA is the main predictor for subsequent suicide
behavior, with a maximum risk in the immediate aftermath of an index SA [2,4–7]; (2) rather
than a substitute, they are intended to complement usual treatment and augment its effective-
ness; and (3) as they can easily be initiated from emergency departments (EDs), they do not
require on-site mental-health specialist, and are therefore easily scalable up to an entire popu-
lation [8,9].

As shown by 3 recently published meta-analyses, results of efficacy trials of BCI are equivo-
cal, however. Focusing on ED-based preventive interventions, Inagaki et al. found active con-
tact and follow-up actions to significantly reduce the risk of suicide reattempt at 12 month
(combined RR = 0.83, 95% CI: 0.71–0.98), but the effect was not maintained at 24 months
(RR = 0.97, 95% CI: 0.76–1.22) [2]. Results obtained by Milner et al. are equally nuanced;
when compared to control, the interventions resulted in significantly reduced number of sui-
cide reattempts per person (incidence rate ratio IRR = 0.66, 95% CI:0.54–0.80), while the over-
all effects on suicide reattempt and suicide completion combined were not significant [10]. By
contrast, Riblet et al. found that BCI were associated with significantly lower odds of suicide
(OR = 0.20, 95% CI: 0.09–0.42)[3]; these interventions were WHO BCIs performed in low-
and middle-income countries.

Results from AlgoS, a multi-modal intervention strategy that combines BCIs to fit partici-
pants' profile through an algorithm, are in line with these meta-analyses [11]. AlgoS was built
upon the hypothesis that each type of BCIs would not have an equal effect on all participants.
Given the literature accumulated at that time, it assumed that the crisis card would be most
efficient for first attempters, whereas multiple attempters would rather need a phone call. Post-
cards would be used as a substitute if the participant was not reachable, and in complement to
the phone call for most serious cases (non-adherent to the post-discharge treatment, and/or in
suicidal crisis; see [9,11] and below for complete description). In the AlgoS trial, we indeed
found the intervention to significantly reduce suicide reattempts or loss to follow-up at 6
months (19.5% vs. 25.1%, p = 0.034), contrasting with a non-significant reduction of suicide
reattempts analyzed alone (12.8% vs. 17.2%, p = 0.056). It is noteworthy that those equivocal
results were obtained from Intent to Treat analysis (ITT), while Per Protocol (PP) analyses
yielded more consistent result on suicide reattempts alone (10.2% vs. 15.2%, p = 0.022) and on
suicide reattempts and loss to follow-up combined (17.6% vs. 24.8%, p = 0.007). This contrast,
together with the inconsistencies of the literature, raises questions about the appropriateness
of relying only on ITT analyses for the evaluation of BCIs in general, and of the AlgoS protocol
in particular. Indeed, the New England Journal of Medicine issued in 2016 a series of questions
that researchers should ask when an ITT analysis finds a "non-significant" treatment differ-
ence [12]. Having in mind that the PP analysis of AlgoS trial was favorable to the intervention,
four of these questions were particularly relevant to our case: "is there some indication of
potential benefit"; "was the population appropriate"; "do subgroup analysis elicit positive sig-
nals"; and "can alternative analysis help".

Both ITT and PP analyses of AlgoS trial indistinctly encompassed the global intervention,
without disentangling the effects of each component, which yet substantially differed by
patient's suicidal status (first vs. non-first attempter), and were further composite for non-first
attempters. Consequently, even if we assumed sufficient statistical power, the reasons for
inconclusive results couldn't be firmly decided between: (1) insufficient efficacy across the
board; (2) inefficacy among a particular sub-group or a particular aspect of the multifaceted
intervention; or (3) a combination of (1) and (2). Also, ITT analysis included some participants
that shouldn't have entered the protocol, because information given at inclusion turned out to
be erroneous and corresponded to exclusion criteria after correction. These participants were
high-end re-attempters (over 3 SA within 3 years); such a limit was set because AlgoS investi-
gators considered that BCIs would be insufficiently efficacious on high-end suicidal cases—for
them, more intensive care would be appropriate [4]. Therefore, their inclusion in the ITT anal-
ysis might have resulted in an artificial deflation of intervention efficacy. Furthermore, strict
ITT framework does not allow for post-hoc subgroup analyses, hence prevent complete evalu-
ation of some BCIs multiple benefits. For instance, the 10–21 days phone calls in AlgoS, pri-
marily intended to reach out to multiple attempters and reduce their reattempt risk, also
provided information about their clinical state, which could have a risk-assessment value *per
se*. Only fine-grain analyses, beyond a strict ITT framework applied to AlgoS as a whole, would
aim at eliciting such value. Finally, usual PP approach excludes patients that are “non-adher-
ent”. By contrast, AT approach keep these patients in the analysis, but in a separate group, in
order to evaluate the treatment effect among them. In our case “non-adherence” was peculiar:
(1) it consisted in cross-over between distinct type of BCI, not between BCI and non-BCI
groups; (2) it originated from the investigator (erroneous assessment of participant's suicide
attempt history).

For all the above reasons, we decided to reexamine AlgoS data, with a post-hoc As Treated (AT) approach, subgroup analyses, and exclusion of participants that turned out to be ineligible. The goal of this strategy was (1) to complement previous assessment of AlgoS efficacy in terms of reduction of SA reoccurrence, while taking into account the actual intervention received by the patients and, accordingly, patients' suicidal characteristics; (2) to assess the risk-assessment value of AlgoS 10-21 days phone calls among multiple attempters. This strategy was undertaken with acknowledgment of flaws it may generate, which we discuss herein.

Methods

The Algos trial was authorized by the French Ministry of Health, and approved by the Comité
de Protection des Personnes de Nord-Pas-de-Calais (Ethics Committee). It was registered with
ClinicalTrials.gov (NCT01123174).
## General procedure

The AlgoS intervention was evaluated through a multicenter RCT conducted in 23 French Emergency Departments (EDs) and psychiatric crisis centers. Eligible participants were randomized to receive either treatment as usual (TAU group) or BCIs, nature of which was predefined by the AlgoS algorithm (AlgoS group). Randomization was performed with a 1:1 group ratio. All patients gave written informed consent before randomization, and were blinded to their group attribution. A complete description of the protocol has been published elsewhere [9]; we give here details that are relevant to the present analysis.

## Study participants

SA was defined after Silverman et al. nomenclature [13] as a “self-inflicted, potentially injurious behavior (self-injury or self-poisoning) with a nonfatal outcome for which there is evidence (either explicit or implicit) of intent to die” [13]. This definition was clarified to all emergency physicians. Adult patients (≥ 18 years old) were offered to participate in the study if they presented at the ED within 7 days of an SA, provided that their total number of SA was no more than 3 over the 3 past years. For follow-up purposes, they had to be reachable by phone for 14 months. Patients who were homeless or under legal guardianship were not invited to participate.

## Intervention

Participants of the TAU group received standard care for a SA as defined by the guidelines of the emergency center where the patients were attended. This included referral to outpatient clinics or ad-hoc post-crisis appointments. In accordance to AlgoS algorithm (Fig 1), patients assigned to the intervention group received the following intervention in addition to TAU:

* First-attempters: if the current SA was patient's first lifetime SA, the patient received a crisis card at discharge, showing a toll-free number that could be called 24/7.

* Multi-attempters: if the current SA was not patient's first SA, the patient received a phone call by a dedicated team of trained psychologists, on behalf of the initial attending team, between day 10 and day 21 after the SA. This phone contact aims were (1) to provide psychological support and asses patient's mental health state, (2) to evaluate the patient's adherence to post-discharge healthcare plan, and (3) to encourage the patient to make new contacts with his/her healthcare providers. At this point, the algorithm further split into 3 branches:

* The patient could be reached, was neither in suicidal crisis nor in distress, and was adherent to the post-discharge treatment: no further intervention was undertaken (apart from TAU).

* The patient could not be reached after 3 attempts at 3 different days and times: post-cards were sent at months 2, 3, 4 and 5.

* The patient could be reached, but was in distress or suicidal crisis, and/or did not adhere to the post-discharge treatment: post-cards were sent at months 2, 3, 4 and 5; for patients in distress or suicidal crisis, an emergency consultation was set-up within 24 hours, at the center where they had been attended for their SA.

## Data collection

Baseline data were collected during a semi-structured ED visit led by an attending physician. They included demographics, basic information about the index SA (medication/drug
<figure><img src="image_4.png" /><figcaption>Fig 1. Study flowchart.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0210778.g001

overdose, use of alcohol), and information regarding current treatment for anxiety, depression, alcohol/drug related disorder, or eating disorder.

The outcome consisted in occurrence of at least one suicide reattempt during the follow-up period. Follow-up assessments were conducted at 6 months, and 13–14 months by a trained psychologist, blinded to patients’ group. The 13–14 month window for the second follow-up was chosen to avoid the 12-month SA anniversary date. To limit the recall bias, the patient status with respect to SA attempts was verified by cross-matching patients’ interview, reviews of medical records, and communication with patient’s GP, psychiatrists and caregivers.

## Data analysis

To complete and refine the results from the ITT assessment design of ALGOS, we performed post-hoc AT analysis and subgroup analysis on the same data set. Since the AT strategy privileges efficacy assessment as a function of actual intervention (or absence thereof) received by patients, groups were divided beyond the AlgoS vs. TAU dichotomy. Consequently, the plan of analysis was (1) to compare AlgoS first-attempters with their TAU counterparts; (2) to compare AlgoS multi-attempters with their TAU counterparts; then, in order to assess the 10–21 days phone call predictive value, AlgoS multi-attempters were compared by phone call outcome, using reachable and compliant patients as the reference group. AlgoS multi-attempters
who received a crisis card instead of a phone call by mistake were also compared with reach-
able and compliant multi-attempters.

In line with the AT approach, patients who were unduly enrolled in the RCT were excluded post-hoc from analysis, in particular patients for whom the number of past SA was found post-randomization to exceed 3 in the 3 past years; AlgoS interventions are not expected to be efficacious on these patients.

The two outcomes were the existence of a SA reoccurrence within 6 months and within 13-
14 months (binary yes/no). The main independent variable was the intervention actually
received. Given the subgroups defined above, three sets of analyses were undertaken: (1)
among firs-time attempters, crisis card and treatment as usual vs. treatment as usual alone
(TAU); (2) among non first-time attempters, phone call and/or post cards and TAU vs. TAU
alone; (3) within non first-time attempters assigned to AlgoS, comparisons by intervention
received: crisis card, phone call alone (patient adherent to post-discharge treatment and not in
crisis), post cards alone (patient non reachable), phone call and post cards (patient not adher-
ent to post-discharge treatment and/or in crisis). The same sets of analyses were performed for
baseline characteristics to elicit which of them could be significant confounders. We used chi-
square tests with continuity correction or Fisher's exact test to perform baseline comparisons.
Six-month and 13-14-month new SA were analyzed by computing bivariate relative risks and
their 95% confidence interval. Correction for confounding bias was then achieved by comput-
ing a multivariate logistic model with 6-months and 13-months new SA as outcome, and sub-
groups of intervention as a predictor. To identify potential confounders, variables that differed
between groups at baseline with a p-value <0.15 were submitted to binomial logistic regres-
sions. Variables were retained for the final model whenever the Wald Chi2 test reached the
0.10 level of significance. This model-building strategy complied with the Hosmer and Leme-
show guidelines [14]. All statistical analysis was done using the R statistical software version
3.3.2 [15].

**Results**

Study sample

The inclusion and analysis flow chart is presented on Fig 2. One thousand and forty patients were included and randomized into two groups of equal size (N = 520 each). Among them, 53 patients (27 in the AlgoS group, 26 in the control group) dropped out by consent withdraw, and 15 patients (13 in the AlgoS group, 2 in the control group) were excluded because they were reassessed with a history of more than 3 SA in the past 3 years. The final AT analysis thus included 480 individuals in the intervention group (248 first attempters and 232 multiple attempters) and 492 individuals in the TAU group (270 first attempters and 222 multiple attempters). Since AlgoS consisted in pro-active intervention originating from investigators, no member of the TAU group received it. In the AlgoS arm, the prescribed intervention was implemented among all 248 first-attempters, and among 214 out of 232 (92%) multiple attempters. The 18 multiple attempters who erroneously received a crisis card instead of a phone call were analyzed as a separate group, as per the AT strategy. The other 214 multiple attempters formed another group, which was further analyzed by phone call outcome (Fig 2).

**Baseline characteristics**

Baseline comparisons are shown in Tables 1 and 2. AlgoS first-attempters did not differ from their TAU counterparts, except for SA by medication/drug overdose, which was slightly more frequent. AlgoS multi-attempters did not differ from their TAU counterparts. AlgoS multi-attempters subgroups differed on SA involving the use of alcohol, and on being treated or
<figure><img src="image_5.png" /><figcaption>**Fig 2.** Analysis groups for the As Treated strategy. Number of participants (N) are those at baseline. N at 6-months and 13–14 months analysis are in Table 3.
https://doi.org/10.1371/journal.pone.0210778.g002</figcaption></figure>
<table><caption>Table 1. Baseline characteristics of patients by group of analysis (part 1): First attempters vs. multi-attempters.</caption><thead><tr><th rowspan="2"></th><th colspan="3">First attempters</th><th colspan="3">Multi-attempters</th></tr><tr><th>AlgoS N (%)</th><th>TAU N (%)</th><th>p*</th><th>AlgoS N (%)</th><th>TAU N (%)</th><th>p*</th></tr></thead><tbody><tr><td>Age (years):</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>18–35</td><td>125 (50)</td><td>126 (47)</td><td></td><td>81 (35)</td><td>86 (39)</td><td></td></tr><tr><td>36–55</td><td>105 (42)</td><td>116 (43)</td><td>.401</td><td>116 (50)</td><td>110 (50)</td><td>.492</td></tr><tr><td>&gt; 55</td><td>18 (8)</td><td>28 (10)</td><td></td><td>35 (15)</td><td>26 (12)</td><td></td></tr><tr><td>Gender: men (vs. women)</td><td>97 (39)</td><td>115 (43)</td><td>.421</td><td>78 (34)</td><td>64 (29)</td><td>.270</td></tr><tr><td>Living alone (vs. in couple)</td><td>135 (54)</td><td>143 (53)</td><td>.737</td><td>118 (51)</td><td>119 (54)</td><td>.556</td></tr><tr><td>Employed (vs. unemployed)</td><td>172 (70)</td><td>182 (68)</td><td>.673</td><td>130 (56)</td><td>135 (61)</td><td>.328</td></tr><tr><td>SA by medication overdose</td><td>235 (95)</td><td>242 (90)</td><td>.030</td><td>221 (95)</td><td>214 (96)</td><td>.642</td></tr><tr><td>SA with alcohol</td><td>97 (39)</td><td>113 (42)</td><td>.776</td><td>109 (47)</td><td>98 (44)</td><td>.788</td></tr><tr><td>Treated or followed for:</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Depressive disorder</td><td>83 (34)</td><td>90 (34)</td><td>.996</td><td>121 (52)</td><td>118 (53)</td><td>.829</td></tr><tr><td>Anxiety disorder</td><td>101 (41)</td><td>109 (41)</td><td>.988</td><td>121 (52)</td><td>124 (57)</td><td>.367</td></tr><tr><td>Alcohol/drug related disorder</td><td>29 (12)</td><td>36 (13)</td><td>.564</td><td>59 (26)</td><td>42 (19)</td><td>.111</td></tr><tr><td>Eating disorder</td><td>7 (3)</td><td>6 (2)</td><td>.782</td><td>16 (7)</td><td>18 (8)</td><td>.623</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0210778.t001

followed for alcohol/drug related disorders, which were more frequent among those not reach- able by phone and those non-adherents or in crisis.

## AlgoS efficacy on SA attempts

In the bivariate analysis (Table 3), AlgoS first-attempters had significantly lower risk of SA attempts than their TAU counterparts both at 6 months and 13–14 months (RR = 0.46, 95% CI: 0.25–0.85, and RR = 0.50, 95% CI: 0.31–0.81, respectively). AlgoS multi-attempters did not differ significantly from their TAU counterparts, either at 6 months or at 13–14 months.

Since significant results shown in Table 3 could be the results of differences between groups at baseline characteristics (Tables 1 and 2), we performed logistic regressions (LR) to adjust for potential confounders (Tables 4 and 5). SA attempts at 6 and at 13–14 months were the dependent variables (two distinct sets of LR), AT intervention groups were the independent variables as shown in Tables 1 and 2, and baseline variables from Tables 1 and 2 with p<0.15 were potential confounders. These confounders were kept in final models if their corresponding p-value were below 0.10 in the LR, and dropped otherwise (Table 4). Results of final LR, detailed in Table 5, were in line with results of bivariate analyses: AlgoS first-attempters had significantly lower odds of SA attempts than their TAU counterparts, at 6 months and 13–14 months follow-ups (no adjustments needed).

## Risk assessment value of the 10–21 days phone call

AlgoS multi-attempter subgroup bivariate analysis showed that, with reference to participants who were reachable, adherents, and not in crisis: (1) those not reachable did not significantly differ; (2) those non-compliant and/or those in crisis had significantly higher risk of SA attempts at 13–14 months (RR = 1.83, 95% CI 1.09–3.07); (3) those who received a crisis card

<footnote>*chi-square test if no more than 20% of the expected frequencies are less than 5 and none is less than 1, Fisher's exact test otherwise</footnote>

<footnote>Abbreviations: TAU = Treatment As Usual, SA = suicide attempt.</footnote>
<table><caption>Table 2. Baseline characteristics of patients by group of analysis (part 2): Actual intervention among multi-attempters.</caption><thead><tr><th rowspan="2"></th><th colspan="3">Multi-attempters</th><th rowspan="2">Crisis card<br>N (%)</th><th rowspan="2">p-value *</th></tr><tr><th>Adherent, not in crisis<br>N (%)</th><th>Phone call<br>Not reachable<br>N (%)</th><th>Non-adherent, or in crisis<br>N (%)</th></tr></thead><tbody><tr><td>Age (years):</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>18–35</td><td>35 (38)</td><td>20 (33)</td><td>15 (24)</td><td>11 (61)</td><td></td></tr><tr><td>36–55</td><td>43 (47)</td><td>33 (54)</td><td>34 (55)</td><td>6 (33)</td><td>.112</td></tr><tr><td>&gt; 55</td><td>13 (14)</td><td>8 (13)</td><td>13 (21)</td><td>1 (6)</td><td></td></tr><tr><td>Gender: men<br>(vs. women)</td><td>29 (32)</td><td>24 (39)</td><td>18 (29)</td><td>7 (39)</td><td>.609</td></tr><tr><td>Living alone<br>(vs. in couple)</td><td>42 (46)</td><td>33 (54)</td><td>36 (59)</td><td>7 (39)</td><td>.294</td></tr><tr><td>Employed<br>(vs. unemployed)</td><td>55 (60)</td><td>32 (52)</td><td>29 (48)</td><td>14 (78)</td><td>.099</td></tr><tr><td>SA by medication overdose</td><td>86 (95)</td><td>56 (92)</td><td>61 (98)</td><td>18 (100)</td><td>.265</td></tr><tr><td>SA with alcohol</td><td>34 (37)</td><td>36 (59)</td><td>33 (53)</td><td>6 (33)</td><td>.027</td></tr><tr><td>Treated or followed for:</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Depressive disorder</td><td>44 (48)</td><td>32 (52)</td><td>38 (61)</td><td>7 (41)</td><td>.331</td></tr><tr><td>Anxiety disorder</td><td>50 (55)</td><td>26 (43)</td><td>33 (53)</td><td>12 (67)</td><td>.296</td></tr><tr><td>Alcohol/drug related<br>disorder</td><td>15 (17)</td><td>23 (38)</td><td>19 (31)</td><td>2 (11)</td><td>.010</td></tr><tr><td>Eating disorder</td><td>7 (8)</td><td>4 (7)</td><td>4 (6)</td><td>1 (6)</td><td>.976</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0210778.t002

instead of a phone call had significantly higher risk of SA attempts at 6 months (RR = 2.58, 95% CI: 1.10–6.06). These results did not change substantially after adjusting for potential founders in multivariate analysis (Table 5): those non-compliant and/or those in crisis had higher odds of SA attempts, with corresponding OR significantly different from 1 at 13–14 months; and those receiving a crisis card had higher odds of SA attempts, with corresponding OR significantly different from 1 at 6 months.

# Discussion

## Significance of findings

The post-hoc Analyses showed a significant risk reduction of SA attempts among first-time attempters by delivery of crisis cards thus adding to results found by Evans et al. [16,17]. According to relative risk 95% confidence intervals, risk reduction might range from one fifth in the worst-case scenario to three quarters in the best-case scenario. Crisis card delivery is an easy, low-cost and relatively safe intervention to implement. We believe that the present results reduce the literature ambiguities and invite considering the scale-up of such minimal action for first attempters to a population-size prevention strategy. Alternatively, due to unknown bias introduced by subgroup and AT analyses, one might want to wait for further research results before embarking on such scaling-up. In addition, it might be challenging in an ED environment to ascertain whether an individual has actually no past history of suicide attempt. Therefore, future research on the efficacy of crisis card in the multi-attempters population, probably as a supplement to other interventions, is needed; if proved efficacious, crisis

<footnote>*chi-square test if no more than 20% of the expected frequencies are less than 5 and none is less than 1, Fisher's exact test otherwise</footnote>

<footnote>Abbreviations: SA = suicide attempt.</footnote>
<table><caption>Table 3. Suicide reattempts at 6 months and 13–14 months follow-ups: Rates (per 100) and corresponding relative risks.</caption><thead><tr><th rowspan="2"></th><th colspan="3">6 months follow-up</th><th colspan="3">13–14 months follow-up</th></tr><tr><th>N</th><th>suicide reattempt (per 100)</th><th>RR [95% CI]</th><th>N</th><th>suicide reattempt (per 100)</th><th>RR [95% CI]</th></tr></thead><tbody><tr><td colspan="7"><strong>First attempters</strong></td></tr><tr><td>TAU</td><td>241</td><td>13.3</td><td>ref.</td><td>215</td><td>19.5</td><td>ref.</td></tr><tr><td>AlgoS</td><td>227</td><td>6.2</td><td>0.46<br>[0.25–0.85]</td><td>215</td><td>9.8</td><td>0.50<br>[0.31–0.81]</td></tr><tr><td colspan="7"><strong>Multi-attempters, AlgoS vs. TAU</strong></td></tr><tr><td>TAU</td><td>204</td><td>21.0</td><td>ref.</td><td>187</td><td>28.3</td><td>ref.</td></tr><tr><td>AlgoS</td><td>219</td><td>17.8</td><td>0.84<br>[0.57–1.25]</td><td>201</td><td>28.3</td><td>1.00<br>[0.73–1.37]</td></tr><tr><td colspan="7"><strong>Multi-attempters, within AlgoS</strong></td></tr><tr><td colspan="7"><strong>Phone call and its outcome</strong></td></tr><tr><td>Adherent, not in crisis<br>→ no more action</td><td>85</td><td>12.9</td><td>ref.</td><td>74</td><td>23.0</td><td>ref.</td></tr><tr><td>Non reachable<br>→ post cards</td><td>57</td><td>12.3</td><td>0.95<br>[0.39–2.30]</td><td>52</td><td>17.3</td><td>0.75<br>[0.36–1.56]</td></tr><tr><td>Non-adherent / in crisis<br>→ post cards</td><td>59</td><td>25.4</td><td>1.96<br>[0.97–3.97]</td><td>57</td><td>42.1</td><td>1.83<br>[1.09–3.07]</td></tr><tr><td>Crisis card</td><td>18</td><td>33.3</td><td>2.58<br>[1.10–6.06]</td><td>18</td><td>38.9</td><td>1.69<br>[0.82–3.45]</td></tr></tbody></table>

Abbreviations: TAU = Treatment As Usual

https://doi.org/10.1371/journal.pone.0210778.t003

card delivery could then be proposed as a basic intervention towards any type of suicide attempter.

By contrast, phone calls, possibly complemented with or substituted by post-cards, did not show significant reduction of SA attempts among multi-attempters. Although neither lack of power nor bias can be ruled out (AlgoS was powered to be evaluated as a whole, not by sub-groups), those negative results are coherent with a recent meta-analysis [18] and provide a

<table><caption>Table 4. Logistic regressions (LR) of suicide reattempt at 6 and 13–14 months by As Treated (AT) group or subgroup: Selection of baseline variables as potential confounders.</caption><thead><tr><th rowspan="2">Main independent variable: AT group or subgroup</th><th rowspan="2">Other independent variables: potential confounders</th><th colspan="2">6 months follow-up</th><th colspan="2">13–14 months follow-up</th></tr><tr><th>p-value initial LR</th><th>Included (I) / Excluded (E) in final LR</th><th>p-value initial LR</th><th>Included (I) / Excluded (E) in final LR</th></tr></thead><tbody><tr><td>First attempters, AlgoS (ref: TAU)</td><td>SA by medication overdose</td><td>.401</td><td>E</td><td>.446</td><td>E</td></tr><tr><td rowspan="4">Multi-attempters, within AlgoS subgroup * (ref: phone call, patient is adherent and not in crisis)</td><td>Age</td><td>.603</td><td>E</td><td>.560</td><td>E</td></tr><tr><td>Employed</td><td>.567</td><td>E</td><td>.296</td><td>E</td></tr><tr><td>SA with alcohol</td><td>.035</td><td>I</td><td>.151</td><td>E</td></tr><tr><td>Treated/Followed for alcohol/drug disorder</td><td>.017</td><td>I</td><td>.001</td><td>I</td></tr></tbody></table>

* Subgroups are

(1) phone call, patient is adherent and not in crisis → no more action

(2) phone call, patient is not reachable → post cards

(3) phone call, patient is not adherent and/or in crisis → post cards (and consultation if in crisis)

(4) crisis card (instead of a phone call)

https://doi.org/10.1371/journal.pone.0210778.t004
<table><caption>Table 5. Logistic regressions (LR) of suicide reattempt at 6 and 13–14 months by As Treated (AT) group or sub-group: Odds-ratios of SA reattempt adjusted for potential confounders.</caption><thead><tr><th rowspan="2">Main independent variable: AT group or subgroup</th><th colspan="2">6 months follow-up</th><th colspan="2">13–14 months follow-up</th></tr><tr><th>OR</th><th>95% CI</th><th>OR</th><th>95% CI</th></tr></thead><tbody><tr><td colspan="5"><strong>First attempters</strong></td></tr><tr><td>TAU</td><td>1 (ref.)</td><td></td><td>1 (ref.)</td><td></td></tr><tr><td>AlgoS</td><td>0.43</td><td>0.22–0.83</td><td>0.45</td><td>0.25–0.78</td></tr><tr><td colspan="5"><strong>Multi-attempters, within AlgoS</strong></td></tr><tr><td>Phone call: Adherent, not in crisis → no more action</td><td>1 (ref.)</td><td></td><td>1 (ref.)</td><td></td></tr><tr><td>Phone call: Non reachable<br/>→ post cards</td><td>0.56</td><td>0.19–1.65</td><td>0.49</td><td>0.19–1.32</td></tr><tr><td>Phone call: Non-adherent / in crisis → post cards</td><td>1.59</td><td>0.64–4.01</td><td>2.23</td><td>1.01–4.94</td></tr><tr><td>Crisis card</td><td>3.97</td><td>1.16–13.6</td><td>2.51</td><td>0.81–7.73</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0210778.t005

plausible explanation for the inconclusive overall effect of AlgoS found with the ITT analysis. A possible reason for such insufficient efficacy is the modesty of the intervention: only one phone call (over three attempts) in the immediate aftermath of the SA. Likewise, the number of post-card sent was small, and the efficacy of post-cards is still equivoque in the literature [4,16,19–21]. In other studies, multiple attempters were found to show higher intentionality in their attempts, higher levels of psychopathology, and higher risk of reattempt than first-time attempters [22,23].

Phone calls had a risk-assessment utility, since their outcome was predictive of SA reattempt: participants found to be in crisis and/or non-compliant were more likely to reiterate a SA than their compliant and not-in-crisis counterparts. In spite of their apparent insufficient efficacy, phone calls could therefore be kept within a multimodal strategy, as a surveillance component, results of which could trigger more aggressive actions—in AlgoS, subsequent actions were limited to sending post-cards.

The current literature does not give unequivocal directions about how to optimize surveillance systems for multiple attempters. Heterogeneity might be one of the most prominent characteristic of suicidal populations, and pathways to SA are highly complex [4,24–26]. There is a growing recognition that suicide prevention needs a comprehensive, systemic approach, involving a complex set of interventions that are likely to complement each other [24–29], even in the face of underwhelming results (for example [6]). Consequently, AlgoS’ multiple components should be further developed into a finer-grain algorithm. Depending on the outcome of short-term clinical assessments, interventions could be diversified in nature, frequency, and intensity. In addition, BCIs could incorporate new technologies such as mobile phone text messages [30], emails, or social media, in complement to traditional postcards or phone calls.

## Study strengths and limitations

Our analysis gave priority to the intervention actually received by patients, and performed post-hoc subgroups analysis, thus unveiling aspects of the intervention that would otherwise stay obliterated. It found efficacy discrepancies between subgroup of patients and, in turn, informed aspects of the overall intervention that could be improved or changed. We must acknowledge, however, that exclusion of participants who were erroneously included can create bias and generate misleading results, and the same is true for the AT approach [12]. If we call ITTe the intervention effect that is estimated with ITT rules, all randomized subjects ought to be analyzed, whether rightly or wrongly included. Indeed, ITTe is a good estimate of the
intervention effectiveness, because in real world erroneous treatment assignment occurs, as well as imperfect adherence to treatment. In scenarios where the true treatment effect is null or small, ITTe is also the least biased estimate of efficacy; in general, ITTe provides a more conservative estimate of treatment efficacy, and therefore will avoid overstating an effect when one doesn't exist. But when the true effect is moderate or large, however, ITTe turns out to be the most biased estimate of treatment efficacy, towards the null [31–33], hence with a tendency of finding an absence of effect when one exists. Nevertheless, one must bear in mind that, depending on the real unknown magnitude of the intervention effect, and depending on quantitative and qualitative characteristics of all departures from the protocol, an AT estimate of efficacy can also be biased, most likely overinflated. Our results, then, should be regarded as optimistic.

By the very nature of the intervention among multi-attempters, based on the result of a phone call that was made for the AlgoS group only, subgroup AlgoS vs. TAU comparisons were not possible. We made subgroup comparisons within the AlgoS group, taking the group with the fewest expected SA attempts as reference, in order to elicit subgroup differences that could unveil the possible risk-assessment value of the phone call outcome. This outcome is very likely to correspond to unmeasured psychopathology. Indeed, it can be hypothesized that presenting with severe mental health problems increases the risk of suicide reattempt and the probability of being found unreachable or in crisis at 10–21 days. Rigorously collecting psychopathology, however, is labor intensive and requires psychiatric expertise, both of which are barely compatible with the routine activity of an ED physician. As far as AlgoS is meant to be scaled up to an entire population, where availability of psychiatric time and expertise is not ubiquitous, such assessment can only be deferred after the day 10–21 call, let alone assessment inaccuracies in the wake of a suicidal crisis. Indeed, algorithm-based BCIs have to compose with uncertainties, and rely on clinical proxies; in view of our results, the outcome of the 10–21 days call can be one of these proxies.

## Conclusions

This study suggests that crisis cards (green cards) might be a useful tool to prevent SA attempts among first-time attempters, and that phone calls are unlikely to be efficacious among multi-attempters, but are informative of SA reattempt risk. We believe that a more elaborated and vigorous BCI, capitalizing on the results of the current analyses, deserve investigation.

## Supporting information

S1 File. Dataset. Dataset containing all variables used to produce results of this article.
(XLSX)

## Acknowledgments

AlgoS investigators affiliation and names are (in alphabetical order of city):

CHU Angers: JB Garré

CHU Brest: M Walter, D Goetghebeur

CHBoulogne-sur-mer: D Hanique, F. Teissiere

CHU Caen: F Chastang, E Roupie

CHU Clermont-Ferrand: G Brousse

CH Douai: O Brochart, B Lacrampe

CH Dunkerque: F Vileyn, JB Campagne

Polyclinique d’Henin-Beaumont: C Michaut, P Lestavel
CHU Lille: F Ducrocq, E Wiel, V Jardon, V Boss, F Warembourg

CHU Marseille: JM Henry

CH Montauban: L Cailhol

CHU Montpellier: P Courtet

CHU Nancy: C Pichene, L Nace

CHU Nantes: V Lagrée

CHU Nice: F Jover, C Thiery, V Buisse

CHU Paris HEGP: E Guillibert, R Masmoudi

CHU Paris H Mondor: F Bellivier, B Renaud

H Quimper: C Chanudet, G Mehu

CHU Rennes: D Travers, J Bouget

CH Roubaix: MA Babe, F Secousse

CHU Toulouse: P Birmes, A Mathur, D Lauque

CH Tourcoing: C Thevenon-Gignac, H Moussouni

CH Vannes: D. Houdet

France.

## Author Contributions

**Conceptualization:** Antoine Messiah, Guillaume Vaiva.

**Data curation:** Antoine Messiah, Anne-Laure Demarty.

**Formal analysis:** Antoine Messiah, Guillaume Vaiva.

**Funding acquisition:** Guillaume Vaiva.

**Investigation:** Anne-Laure Demarty, Stéphane Duhem, Guillaume Vaiva.

**Methodology:** Antoine Messiah, Anne-Laure Demarty, Guillaume Vaiva.

**Project administration:** Stéphane Duhem.

**Resources:** Anne-Laure Demarty, Stéphane Duhem, Guillaume Vaiva.

**Software:** Antoine Messiah.

**Supervision:** Anne-Laure Demarty, Stéphane Duhem, Guillaume Vaiva.

**Validation:** Antoine Messiah, Anne-Laure Demarty, Stéphane Duhem, Guillaume Vaiva.

**Visualization:** Antoine Messiah, Guillaume Vaiva.

**Writing – original draft:** Antoine Messiah.

**Writing – review & editing:** Antoine Messiah, Charles-Edouard Notredame, Stéphane Duhem, Guillaume Vaiva.

## References

1. World Health Organization. WHO | Suicide data [Internet]. WHO. 2016 [cited 2016 Dec 12]. Available from: http://www.who.int/mental_health/prevention/suicide/suicideprevent/en/

2. Inagaki M, Kawashima Y, Kawanishi C, Yonemoto N, Sugimoto T, Furuno T, et al. Interventions to prevent repeat suicidal behavior in patients admitted to an emergency department for a suicide attempt: a meta-analysis. J Affect Disord. 2015; 175:66–78. https://doi.org/10.1016/j.jad.2014.12.048 PMID: 25594513

3. Riblet NBV, Shiner B, Young-Xu Y, Watts BV. Strategies to prevent death by suicide: meta-analysis of randomised controlled trials. Br J Psychiatry. 2017; 210(6):396–402. https://doi.org/10.1192/bjp.bp. 116.187799 PMID: 28428338
4. Hawton K, Witt KG, Taylor Salisbury TL, Arensman E, Gunnell D, Hazell P, et al. Psychosocial interventions for self-harm in adults. In: Cochrane Database of Systematic Reviews [Internet]. John Wiley & Sons, Ltd; 2016 [cited 2016 Dec 21]. Available from: http://cochranelibrary-wiley.com/doi/10.1002/14651858.CD012189/full

5. Finkelstein Y, Macdonald EM, Hollands S, Sivilotti MLA, Hutson JR, Mamdani MM, et al. Risk of Suicide Following Deliberate Self-poisoning. JAMA Psychiatry. 2015; 72(6):570–5. https://doi.org/10.1001/jamapsychiatry.2014.3188 PMID: 25830811

6. Beautrais AL. Intervening to prevent suicide. Lancet Psychiatry. 2014; 1(3):165–7. https://doi.org/10.1016/S2215-0366(14)70304-9 PMID: 26360713

7. Kawanishi C, Aruga T, Ishizuka N, Yonemoto N, Otsuka K, Kamijo Y, et al. Assertive case management versus enhanced usual care for people with mental health problems who had attempted suicide and were admitted to hospital emergency departments in Japan (ACTION-J): a multicentre, randomised controlled trial. Lancet Psychiatry. 2014; 1(3):193–201. https://doi.org/10.1016/S2215-0366(14)70259-7 PMID: 26360731

8. Cooper J, Steeg S, Gunnell D, Webb R, Hawton K, Bennewith O, et al. Variations in the hospital management of self-harm and patient outcome: A multi-site observational study in England. J Affect Disord. 2015; 174:101–5. https://doi.org/10.1016/j.jad.2014.11.037 PMID: 25496757

9. Vaiva G, Walter M, Al Arab AS, Courtet P, Bellivier F, Demarty AL, et al. ALGOS: the development of a randomized controlled trial testing a case management algorithm designed to reduce suicide risk among suicide attempters. BMC Psychiatry. 2011; 11:1–7. https://doi.org/10.1186/1471-244X-11-1 PMID: 21194496

10. Milner AJ, Carter G, Pirkis J, Robinson J, Spittal MJ. Letters, green cards, telephone calls and post-cards: Systematic and meta-analytic review of brief contact interventions for reducing self-harm, suicide attempts and suicide. Br J Psychiatry. 2015; 206(3):184–90. https://doi.org/10.1192/bjp.bp.114.147819 PMID: 25733570

11. Vaiva G, Berrouiguet S, Walter M, Courtet P, Ducrocq F, Jardon V, et al. Combining Postcards, Crisis Cards, and Telephone Contact Into a Decision-Making Algorithm to Reduce Suicide Reattempt: A Randomized Clinical Trial of a Personalized Brief Contact Intervention. J Clin Psychiatry. 2018; 79 (6):17m11631. https://doi.org/10.4088/JCP.17m11631 PMID: 30256552

12. Pocock SJ, Stone GW. The Primary Outcome Fails—What Next? Drazen JM, Harrington DP, McMurray JJV, Ware JH, Woodcock J, editors. N Engl J Med. 2016; 375(9):861–70. https://doi.org/10.1056/NEJMra1510064 PMID: 27579636

13. Silverman MM, Berman AL, Sanddal ND, O’carroll PW, Joiner TE. Rebuilding the tower of Babel: a revised nomenclature for the study of suicide and suicidal behaviors. Part 2: Suicide-related ideations, communications, and behaviors. Suicide Life Threat Behav. 2007; 37(3):264–77. https://doi.org/10.1521/suli.2007.37.3.264 PMID: 17579539

14. Hosmer DW, Lemeshow S. Applied Logistic Regression. John Wiley & Sons; 2004. 397 p.

15. R Development Core Team. R: The R Project for Statistical Computing [Internet]. 2018. Available from: http://www.r-project.org/

16. Kapur N, Cooper J, Bennewith O, Gunnell D, Hawton K. Postcards, green cards and telephone calls: therapeutic contact with individuals following self-harm. Br J Psychiatry. 2010; 197(1):5–7. https://doi.org/10.1192/bjp.bp.109.072496 PMID: 20592425

17. Evans J, Evans M, Morgan HG, Hayward A, Gunnell D. Crisis card following self-harm: 12-month follow-up of a randomised controlled trial. Br J Psychiatry. 2005; 187(2):186–187.

18. Noh D, Park Y-S, Oh EG. Effectiveness of telephone-delivered interventions following suicide attempts: A systematic review. Arch Psychiatr Nurs. 2016; 30:114–9. https://doi.org/10.1016/j.apnu.2015.10.012 PMID: 26804512

19. Beautrais AL, Gibb SJ, Faulkner A, Fergusson DM, Mulder RT. Postcard intervention for repeat self-harm: randomised controlled trial. Br J Psychiatry. 2010; 197(1):55–60. https://doi.org/10.1192/bjp.bp.109.075754 PMID: 20592434

20. Carter GL, Clover K, Whyte IM, Dawson AH, D’Este C. Postcards from the EDge project: randomised controlled trial of an intervention using postcards to reduce repetition of hospital treated deliberate self poisoning. BMJ. 2005; 331(7520):805–9. https://doi.org/10.1136/bmj.38579.455266.E0 PMID: 16183654

21. Motto JA, Bostrom AG. A randomized controlled trial of postcrisis suicide prevention. Psychiatr Serv. 2001; 52(6):828–833. https://doi.org/10.1176/appi.ps.52.6.828 PMID: 11376235

22. Liu Y, Zhang J, Sun L. Who are likely to attempt suicide again? A comparative study between the first and multiple timers. Compr Psychiatry. 2017; 78:54–60. https://doi.org/10.1016/j.comppsych.2017.07.007 PMID: 28803042
23. Miranda R, Scott M, Hicks R, Wilcox HC, Harris Munfakh JL, Shaffer D. Suicide Attempt Characteristics, Diagnoses, and Future Attempts: Comparing Multiple Attempters to Single Attempters and Ideators. J Am Acad Child Adolesc Psychiatry. 2008; 47(1):32–40. https://doi.org/10.1097/chi.0b013e31815a56cb PMID: 18174823

24. Turecki G. Preventing suicide: where are we? Lancet Psychiatry. 2016; 3(7):597–8. https://doi.org/10.1016/S2215-0366(16)30068-2 PMID: 27289304

25. Turecki G, Brent DA. Suicide and suicidal behaviour. The Lancet. 2016; 387(10024):1227–39.

26. Zalsman G, Hawton K, Wasserman D, van Heeringen K, Arensman E, Sarchiapone M, et al. Suicide prevention strategies revisited: 10-year systematic review. Lancet Psychiatry. 2016; 3(7):646–59. https://doi.org/10.1016/S2215-0366(16)30030-X PMID: 27289303

27. Krysinska K, Batterham PJ, Tye M, Shand F, Calear AL, Cockayne N, et al. Best strategies for reducing the suicide rate in Australia. Aust N Z J Psychiatry. 2016; 50(2):115–8. https://doi.org/10.1177/0004867415620024 PMID: 26698822

28. Wasserman D, Rihmer Z, Rujescu D, Sarchiapone M, Sokolowski M, Titelman D, et al. The European Psychiatric Association (EPA) guidance on suicide treatment and prevention. Eur Psychiatry. 2012; 27(2):129–41. https://doi.org/10.1016/j.eurpsy.2011.06.003 PMID: 22137775

29. Mann JJ, Apter A, Bertolote J, Beautrais A, Currier D, Haas A, et al. Suicide Prevention Strategies: A Systematic Review. JAMA. 2005; 294(16):2064–74. https://doi.org/10.1001/jama.294.16.2064 PMID: 16249421

30. Berrouiguet S, Alavi Z, Vaiva G, Courtet P, Baca-García E, Vidailhet P, et al. SIAM (Suicide intervention assisted by messages): the development of a post-acute crisis text messaging outreach for suicide prevention. BMC Psychiatry. 2014; 14:294. https://doi.org/10.1186/s12888-014-0294-8 PMID: 25404215

31. Ye C, Beyene J, Browne G, Thabane L. Estimating treatment effects in randomised controlled trials with non-compliance: a simulation study. BMJ Open. 2014; 4(6):e005362. https://doi.org/10.1136/bmjopen-2014-005362 PMID: 24939814

32. Hernan MA, Hernandez-Diaz S. Beyond the intention-to-treat in comparative effectiveness research. Clin Trials. 2012; 9(1):48–55. https://doi.org/10.1177/1740774511420743 PMID: 21948059

33. Bang H, Davis CE. On estimating treatment effects under non-compliance in randomized clinical trials: are intent-to-treat or instrumental variables analyses perfect solutions? Stat Med. 2007; 26(5):954–64. https://doi.org/10.1002/sim.2663 PMID: 16900559