# Using Discretely Integrated Condition Event Simulation To Construct Quantitative Benefit-Risk Models: The Example of Rotavirus Vaccination in France

Hugo Arlegui, G. Nachbaur, N. Praet, Bernard Begaud, J. J. Caro

► To cite this version:

Hugo Arlegui, G. Nachbaur, N. Praet, Bernard Begaud, J. J. Caro. Using Discretely Integrated Condition Event Simulation To Construct Quantitative Benefit-Risk Models: The Example of Rotavirus Vaccination in France. Clinical Therapeutics, 2020, 42 (10), pp.1983-1991 e2. 10.1016/j.clinthera.2020.08.013. hal-03167420

HAL Id: hal-03167420

https://hal.science/hal-03167420

Submitted on 12 Mar 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

Check for
updates

# Using Discretely Integrated Condition Event Simulation To Construct Quantitative Benefit–Risk Models: The Example of Rotavirus Vaccination in France

Hugo Arlegui, PharmD, MSc, PhD<sup>a,b,c</sup>; Gaëlle Nachbaur, MSc<sup>c</sup>; Nicolas Praet, DVM, MSc, PhD<sup>d</sup>; Bernard Bégaud, MD, PhD<sup>a,b</sup>; and J. Jaime Caro, MDCM, FRCPC<sup>e,f,g</sup>

<sup>a</sup>University of Bordeaux, Bordeaux, France; <sup>b</sup>Pharmacoepidemiology Team, INSERM, Bordeaux Population Health Research Centre, Bordeaux, France; <sup>c</sup>GlaxoSmithKline, Rueil, Malmaison, France; <sup>d</sup>GlaxoSmithKline, Wavre, Belgium; <sup>e</sup>Department of Epidemiology and Biostatistics, McGill University, Montreal, Quebec, Canada; <sup>f</sup>Eidera, Waltham, MA, United Kingdom; and <sup>g</sup>London School of Economics and Political Science, London, United Kingdom

## ABSTRACT

**Purpose:** Although quantitative benefit-risk models (qBRms) are indisputably valuable tools for gaining comprehensive assessments of health care interventions, they are not systematically used, probably because they lack an integrated framework that provides methodologic structure and harmonization. An alternative that allows all stakeholders to design operational models starting from a standardized framework was recently developed: the discretely integrated condition event (DICE) simulation. The aim of the present work was to assess the feasibility of implementing a qBRm in DICE, using the example of rotavirus vaccination.

**Methods:** A model of rotavirus vaccination was designed using DICE and implemented in spreadsheet software with 3 worksheets: Conditions, Events, and Outputs. *Conditions* held the information in the model; this information changed at *Events*, and *Outputs* were special Conditions that stored the results collected during the analysis. A hypothetical French birth cohort was simulated for the assessment of rotavirus vaccination over time. The benefits were estimated for up to 5 years, and the risks in the 7 days following rotavirus vaccination versus no vaccination were assessed, with the results expressed as benefit-risk ratios.

**Findings:** This qBRm model required 8 Events, 38 Conditions, and 9 Outputs. Two Events cyclically updated the rates of rotavirus gastroenteritis (RVGE) and intussusception (IS) according to age. Vaccination

occurred at 2 additional Events, according to the vaccination scheme applied in France, and affected the occurrence of the other Events. Outputs were the numbers of hospitalizations related to RVGE and to IS, and related deaths. The entire model was specified in a small set of tables contained in a 445-KB electronic workbook. Analyses showed that for each IS-related hospitalization or death caused, 1613 (95% credible interval, 1001–2800) RVGE-related hospitalizations and 787 (95% credible interval, 246–2691) RVGE-related deaths would be prevented by vaccination. These results are consistent with those from a published French study using similar inputs but a very different modeling approach.

**Implications:** A limitation of the DICE approach was the extended run time needed for completing the sensitivity analyses when implemented in the electronic worksheets. DICE provided a user-friendly integrated framework for developing qBRms and should be considered in the development of structured approaches to facilitate benefit-risk assessment. (Clin Ther. 2020;42:1983–1991) © 2020 GlaxoSmithKline Biologicals SA. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

Accepted for publication August 21, 2020

https://doi.org/10.1016/j.clinthera.2020.08.013

0149-2918/$ - see front matter

© 2020 GlaxoSmithKline Biologicals SA. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
**Key words:** DICE, discretely integrated condition event, benefit-risk model, simulation method, rotavirus.

## INTRODUCTION

The benefits of any health care intervention have to be balanced with its risks.<fnref n="1" />⁻³ To this end, conducting a comprehensive benefit-risk assessment (BRA) is essential and is the core of health care intervention conceptions and regulatory review processes.⁴ Although qualitative judgments have been used in BRAs of health interventions, quantitative benefit-risk models (qBRms) may provide better consistency, transparency, and predictability of decision making.⁵⁻⁷ These qBRms integrate evidence from multiple sources to quantify and put into perspective the benefits and risks of a health intervention.⁴,⁶,⁸ Several modeling approaches, such as decision trees,⁹,¹⁰ cohort models (also called Markov models),¹¹⁻¹⁴ discrete-event simulations,¹⁵,¹⁶ and dynamic transmission models,¹⁷,¹⁸ have been used, each presenting its own strengths and weaknesses.⁴,⁸ Despite efforts to promote their use, qBRms are not systematically used for BRAs, probably owing to their perceived complexity and the absence of an integrated framework providing methodologic structure and harmonization.⁸,¹⁹⁻²¹ The latter is crucial for ensuring transparency, with the aim of supporting informed decision making.⁴,⁸

Recently, an alternative for designing and structuring decision-analytic models was developed: the *discretely integrated condition event* (DICE) simulation.²² DICE brings together the common modeling approaches into a standardized framework that can be used by any stakeholder to structure a decision problem. A DICE model is conveniently specified in a series of tables that follow a consistent format, and the simulation can be fully implemented in spreadsheet software, making design, analysis, review, and validation straightforward.²³

Although DICE simulation is increasingly applied in health economics,²⁴⁻²⁸ it has not yet been used for BRA. It was tempting to challenge this novel approach with actual problems of BRA. The aim of the work was to test the use of DICE to implement a qBRm and examine its advantages and limitations.

This was done using the example of rotavirus vaccination.

## MATERIALS AND METHODS

### DICE Approach

In DICE, a model is specified in terms of conditions, events, and outputs. *Conditions* represent all information in the model, such as demographics, treatment characteristics, processes, the environment, the timing of events, model controls, random number, run settings and other inputs. They are tabulated using spreadsheet software (Excel 2016; Microsoft Corporation, Redmond, Washington). *Events* are points in time at which the values of some conditions change. These changes in event times, accumulating outputs, or other conditions are formulated as text expressions that respect Excel syntax but do not have the leading equals sign. These event consequences are also tabulated in an Events worksheet. *Outputs* are special conditions that store results and are tabulated on a corresponding worksheet. The events are executed by a simple open-source macro (Visual Basic for Applications; Microsoft) that loops through each Events table, row by row, sequentially inserting an equals sign in front of each text expression to convert it into an active Excel formula.²²,²⁹

### Rotavirus Vaccination in France

Rotavirus gastroenteritis (RVGE) in young children can lead to hospitalization and even death.³⁰⁻³² Rotavirus vaccines have led to significant declines in RVGE-related morbidity and mortality.³³⁻³⁶ Although rotavirus vaccines have proved to be effective, a small increased risk for intussusception (IS)—a rare, but potentially serious, condition in which a segment of the intestine invaginates into an adjacent segment³⁷⁻³⁹)—has been observed, typically within 7 days of vaccination.³⁹⁻⁴²

Two live, attenuated oral rotavirus vaccines, Rotarix®<fnref n="1" /> and RotaTeq®,<fnref n="2" /> are licensed for use in France, with neither recommended over the other. In 2015, after a report of 2 IS-related fatalities in

<footnote n="1">¹Trademark owned by or licensed to the GSK group of companies.</footnote>

<footnote n="2">²Trademark of Merck & Co Inc.</footnote>
French infants following vaccination, the Haut Conseil de la Santé Publique suspended the previous recommendation on rotavirus vaccination from the French Technical Committee for Pharmacovigilance.43,44 In such a situation, when new safety data become available, it is important to put them into the perspective of the benefit-risk profile of the intervention. To this end, qBRms can be particularly helpful as they allow for the comparison of quantitative estimates of the benefits (eg, vaccine-preventable RVGE-related hospitalizations and deaths) with the risks (eg, vaccine-induced IS-related hospitalizations and deaths) and allow for extensive sensitivity and scenario analyses. A qBRm of the Rotarix® vaccine in France was designed and implemented using DICE.

## Case Study

The case study to test the utility of DICE in the implementation of a qBRm using the Rotarix® vaccine was conducted by using inputs available from the literature. The Rotarix® vaccination course consists of 2 oral doses, with the first given at 6 weeks of age, the 2 doses separated by at least 4 weeks, and the vaccination course being completed before age 24 weeks.45 The benefits considered were the numbers of vaccine-preventable RVGE-related hospitalizations and deaths, and the risks considered were the numbers of vaccine-induced IS-related hospitalizations and deaths. A theoretic cohort aged from birth to 5 years was modeled because this range defines the period of immunization benefits,30–32,45 most commonly used for measuring RVGE burden. Furthermore, infants are vaccinated during their first year, and vaccine-induced IS occurs shortly after vaccination.39,40,42

## RESULTS

The DICE qBRm stores the required information on 38 conditions (see Supplemental Table S1 in the online version at https://doi.org/10.1016/j.clinthera.2020.08.013) covering the course without vaccination (hospitalization and case-fatality rates related to RVGE, and to IS) and with the vaccination (age at vaccination, compliance with second dose, effectiveness against RVGE, and relative risk for IS after doses 1 and 2). Other conditions helped to structure the model, such as simulation time, the time horizon signaling the end of the simulation, duration

of cycles (time to the next occurrence of cyclical events), and intervention strategy (no vaccination versus vaccination).

Eight events (see Supplemental Table S2 in the online version at https://doi.org/10.1016/j.clinthera.2020.08.013) handled disease occurrences (RVGE and IS), vaccine administration (dose 1: Vaccinate1; dose 2: Vaccinate2), onset of vaccine effectiveness 2 weeks after vaccination (VaccineEff), increased risk for IS at 7 days postvaccination (ISinduced), and the mandatory modeling events (Start and End).

Nine outputs specified hospitalizations and deaths related to RVGE and to IS for each strategy, as well as the number of replications to run (see Supplemental Table S3 in the online version at https://doi.org/10.1016/j.clinthera.2020.08.013).

The general DICE macro (https://www.evidera.com/dice) triggered Start first, in which: (1) the intervention strategy was assigned (no vaccination vs vaccination) and (2) time to Vaccinate1 was derived (if vaccination strategy). Then, RVGE event rates were applied every month, and IS rates, every week. During each RVGE and IS event: (1) the number of hospitalizations was estimated by applying the age-specific hospitalization rate obtained using parametric functions46; (2) the number of deaths was computed by applying the case-fatality rate to the hospitalizations; and (3) deaths were subtracted from the cohort. During each Vaccinate event: (1) coverage was updated and (2) ISinduced and VaccineEff events were scheduled. ISinduced occurs daily for 7 days after vaccination, with these consequences: (1) relative risk for IS was updated; (2) number of IS hospitalizations induced by vaccination was computed by applying the relative risk to the age-specific IS hospitalization rate in the proportion of the cohort covered; (3) number of IS deaths induced by vaccination was calculated by applying the IS case-fatality rate; and (4) deaths were subtracted from the cohort. At VaccineEff, which occurs 2 weeks after vaccination: (1) effectiveness was updated; and (2) this affected the numbers of hospitalizations and deaths in the following RVGE events by decreasing the age-specific RVGE-related hospitalization rate. At every event, age was updated, and the next event was triggered based on its scheduled time. At the end of the time horizon (5 years), the End event was executed. The final accrual of outputs is reported in the Figure 1.
<figure><img src="image_3.png" /></figure>

The differences between the 2 strategies, in numbers of hospitalizations and deaths related to RVGE and to IS, were used for determining the hospitalization and death benefit-risk ratios (BRRs). Appropriate distributions were fitted to inputs with parameter uncertainty (Table I). These distributions were used for random sampling in probabilistic sensitivity analyses, with up to $10^4$ replications, to evaluate how uncertainty in these parameters would affect outcomes.

The entire model was specified in a small set of tables contained in a 445-KB Excel workbook. In the absence of a rotavirus vaccination program, the French birth cohort (mean births, 791,183 per annum during 2006–2014) would be expected to experience a mean (95% credible interval) of 15,010 (12,058–18,390) RVGE-related hospitalizations and 10.13 (4.57–19.38) RVGE-related deaths over 5 years, as well as 326 (258–401) IS-related hospitalizations and 0.45 (0.18–0.97) IS-related deaths in infants aged <1 year (Table II).

Vaccination would be expected to prevent 12,553 (9949–15,580) RVGE-related hospitalizations and 8.47 (3.81–16.22) deaths, but to cause 7.76 (4.70–11.86) IS-related hospitalizations and 0.0107 (0.0038–0.0251) IS-related deaths. A BRR for hospitalizations of 1613 (1001–2800) was estimated, meaning that for each IS-related hospitalization caused, 1613 RVGE-related hospitalizations would be prevented by vaccination; with regard to deaths, the BRR was 787 (246–2691). The DICE results confirm that the 2-dose schedule of rotavirus vaccination would prevent a substantial number of RVGE cases at the cost of a small increase in the number of IS cases. These findings are consistent with those published in a 2018 study that used similar inputs but in a quite different model framework,⁴⁶ with absolute differences of <1% in rates without vaccination, and of ~10% in RVGE cases prevented and IS induced following vaccination. The BRR
<table><caption>Table I. Inputs subject to parameter uncertainty used in probabilistic sensitivity analysis in the DICE qBRm.</caption><thead><tr><th>Input</th><th>Median (95% CrI)*</th><th>Distribution†</th></tr></thead><tbody><tr><td><strong>Demography</strong></td><td></td><td></td></tr><tr><td>Age at 1st dose, wk</td><td>9.47 (6.17–22.21)</td><td>γ (2861; 0.000871)</td></tr><tr><td>Delay before 2nd dose, wk</td><td>4.78 (3.38–10.87)</td><td>γ (2239; 0.000566)</td></tr><tr><td>Age at 2nd dose, wk</td><td>14.73 (11.03–28.16)</td><td>Derived from 2 distributions above</td></tr><tr><td>Compliance with 2nd dose</td><td>0.92 (0.72–0.99)</td><td>β (14.43, 1.60)</td></tr><tr><td><strong>Benefit</strong></td><td></td><td></td></tr><tr><td>Baseline hospitalization rate related to RVGE/10<sup>5</sup> children/y</td><td></td><td></td></tr><tr><td>&nbsp;&nbsp;&nbsp;&nbsp;&lt;3 y</td><td>604.0 (526.9–688.4)</td><td>γ (216; 35,711)</td></tr><tr><td>&nbsp;&nbsp;&nbsp;&nbsp;&lt;5 y</td><td>380.7 (305.9–467.1)</td><td>γ (86; 22,534)</td></tr><tr><td>Mortality rate related to RVGE/10<sup>6</sup> children/y</td><td></td><td></td></tr><tr><td>&nbsp;&nbsp;&nbsp;&nbsp;&lt;3 y</td><td>4.08 (1.94–7.42)</td><td>γ (9; 2,125,000)</td></tr><tr><td>&nbsp;&nbsp;&nbsp;&nbsp;&lt;5 y</td><td>2.56 (1.17–4.92)</td><td>Derived from 3 distributions above</td></tr><tr><td>RVGE fatality rate/10<sup>5</sup> hospitalizations</td><td>67.5 (31.7–125.0)</td><td>Derived from 2 distributions above</td></tr><tr><td>Vaccine effectiveness against RVGE D2</td><td>90% (81–95)</td><td>1-log-normal (Ln[0.10]; 0.341)</td></tr><tr><td>Ratio of vaccine effectiveness against RVGE after a 1-dose vs 2-dose schedule</td><td>0.85 (0.62–0.97)</td><td>β (13.6, 2.73)</td></tr><tr><td>Vaccine effectiveness against RVGE D1</td><td>75% (55–88)</td><td>Product of the 2 distributions above</td></tr><tr><td><strong>Risk</strong></td><td></td><td></td></tr><tr><td>Baseline hospitalization rate related to IS/10<sup>5</sup> infants (aged &lt;1 y)/y</td><td>40.85 (32.52–50.50)</td><td>γ (80;195,005)</td></tr><tr><td>Mortality rate related to IS/10<sup>6</sup> infants (aged &lt;1 y)/y</td><td>0.597 (0.24–1.11)</td><td>γ (7;11,731,635)</td></tr><tr><td>IS fatality risk/10<sup>5</sup> IS hospitalizations</td><td>139.17 (57.32–283.30)</td><td>Derived from 2 distributions above</td></tr><tr><td><strong>Relative risk for IS</strong></td><td></td><td></td></tr><tr><td>&nbsp;&nbsp;&nbsp;&nbsp;D1</td><td>5.39 (3.94–7.38)</td><td>Log-normal</td></tr><tr><td>&nbsp;&nbsp;&nbsp;&nbsp;SD: 0.16</td><td>SD: 0.16</td><td></td></tr><tr><td>&nbsp;&nbsp;&nbsp;&nbsp;D2</td><td>1.81 (1.32–2.48)</td><td>Log-normal</td></tr><tr><td>&nbsp;&nbsp;&nbsp;&nbsp;SD: 0.16</td><td>SD: 0.16</td><td></td></tr></tbody></table>

Crl = credible interval; D1 = post dose 1; D2 = post dose 2; DICE, discretely integrated condition event; IS = intussusception; RVGE, rotavirus gastroenteritis.

Data from Ledent et al.⁴⁴

*The limits of 95% CrIs were determined using the 2.5%–97.5% range.

†Distribution parameters are β (α; β), γ (shape; rate), and log-normal (mean; SD).
<table><caption>Table II. Results of the simulation comparing the Ledent et al⁴⁴ with DICE model.</caption><thead><tr><th>Outcome</th><th>Ledent et al<sup>44</sup><br>Median (95% CrI)</th><th>DICE<br>Median (95% CrI)</th><th>Difference, %</th></tr></thead><tbody><tr><td><strong>RVGE &lt; 5 y</strong></td><td></td><td></td><td></td></tr><tr><td>Hospitalizations</td><td></td><td></td><td></td></tr><tr><td>Baseline</td><td>15,059 (12,100–18,476)</td><td>15,010 (12,058–18,390)</td><td>0.33</td></tr><tr><td>Prevented</td><td>11,132 (7842–14,408)</td><td>12,553 (9949–15,580)</td><td>-11.31</td></tr><tr><td>Deaths</td><td></td><td></td><td></td></tr><tr><td>Baseline</td><td>10.13 (4.64–19.46)</td><td>10.13 (4.57–19.38)</td><td>0.05</td></tr><tr><td>Prevented</td><td>7.43 (3.27–14.68)</td><td>8.47 (3.81–16.22)</td><td>-12.23</td></tr><tr><td><strong>IS &lt; 1 y</strong></td><td></td><td></td><td></td></tr><tr><td>Hospitalizations</td><td></td><td></td><td></td></tr><tr><td>Baseline</td><td>323 (257–400)</td><td>326 (258–401)</td><td>-0.76</td></tr><tr><td>Excess</td><td>6.86 (2.25–38.37)</td><td>7.76 (4.70–11.86)</td><td>-11.58</td></tr><tr><td>Deaths</td><td></td><td></td><td></td></tr><tr><td>Baseline</td><td>0.45 (0.19–0.88)</td><td>0.45 (0.18–0.97)</td><td>0.41</td></tr><tr><td>Excess</td><td>0.0099 (0.0024–0.060)</td><td>0.0107 (0.0038–0.0251)</td><td>-6.92</td></tr><tr><td><strong>BRR</strong></td><td></td><td></td><td></td></tr><tr><td>Hospitalization</td><td>1624 (240–5243)</td><td>1613 (1001–2800)</td><td>0.72</td></tr><tr><td>Death</td><td>743 (93–3723)</td><td>787 (246–2691)</td><td>-5.50</td></tr></tbody></table>

BRR = benefit-risk ratio; CrI = credible interval; DICE = discretely integrated condition event; IS = intussusception; RVGE = rotavirus gastroenteritis.

estimates for hospitalizations are similar (<1% difference), and for deaths, slightly higher (by ~5%).

## DISCUSSION

Several health authorities are currently considering the adoption of structured BRAs for facilitating regulatory decisions.⁴⁷,⁴⁸ Despite efforts to promote their use, qBRms are not systematically used, probably owing to their perceived complexity and the absence of an integrated framework that provides methodologic structure and harmonization.⁸,¹⁹⁻²¹ In the present article, we examined the feasibility of implementing a qBRm by using a simpler, standardized modeling framework: DICE. The qBRm was easily implemented in Excel using DICE, requiring no special programming and specifying the entire model in a few simple tables. The consistency of the results with those from previous analyses carried out by programming in SAS software⁴⁶ is reassuring.

Decision makers have shown a marked preference for models implemented in spreadsheets because reviewing them does not require programming

skills.⁴⁹ A DICE model is fully implemented in spreadsheets using a standard framework and terminology. The structuring choices that define the model are straightforward and, thus, easy to communicate and to understand by physicians, modelers, reviewers, decision makers, and other stakeholders.²²

Another advantage of implementing a model in DICE is that it can accommodate any aspect in a single template, from basic models to vast, complex structures.²⁸ The framework could be easily modified, without necessitating additional specific programming skills. These changes could notably include: (1) transposing the cohort model to a patient-level model in order to address the heterogeneity in the determinants of the course (eg, age, sex, genetic makeup)⁵⁰ or RVGE⁵¹,⁵² and IS³⁷,³⁸,⁵³ risk factors; (2) estimating the impact of vaccination on outpatient visits¹⁴; (3) taking into account the herd immunity effects of rotavirus vaccination⁵⁴; and/or (4) updating the cohort by considering deaths attributable to causes other than RVGE and IS
during the time periods considered in the simulation. Therefore, the availability of DICE may encourage a broader group of modelers to use more sophisticated, realistic approaches.

A limitation of DICE, when implemented in Excel, is the extended run time required to complete the probabilistic sensitivity analysis: ~8 h for 10,000 simulations. In the basic DICE version, the repeated interactions between the macro code and the spreadsheets, slow down execution. Minimizing the number of times the macro interacts with the worksheets cuts run time substantially. However, this limitation is currently obsolete since a faster version that uses a compiled macro for reducing run time by > 300-fold is now available.²⁸ Another limitation of using Excel for these simulations was that some statistical functions, such as the Dirichlet-multinomial distribution, were not available. However, these distributions can be implemented as user-defined functions.

## CONCLUSIONS

This study implemented a qBRm using DICE methodology. DICE simulation allows for designing and implementing a decision-analytic model without the need for customized programming. Indeed, the formulation is user friendly, as the entire model is specified in simple tables that can be easily updated and modified as needed. For these reasons, the use of DICE should be considered in the development of structured methodologies to facilitate BRA.

## AUTHORS CONTRIBUTIONS

All of the authors participated in the preparation of the article and approved the final version of the manuscript. H.A. and J.J.C. created the DICE version and ran the analyses. B. Bégaud acted as the academic PhD supervisor of this study.

## DISCLOSURES

Financial support for this study was provided as part of a PhD program funded by GlaxoSmithKline Biologicals SA and the Association Nationale pour la Recherche et la Technologie, Paris, France. The funding agreement ensured the authors' independence in designing the study, interpreting the data, and writing and publishing the report.

H. Arlegui was employed under a PhD program by the GlaxoSmithKline group of companies at the time of

the study. G. Nachbaur and N. Praet are employed by and hold stock options in the GlaxoSmithKline group of companies, the producer of Rotarix®. J.J. Caro is employed by a consultancy that works with GlaxoSmithKline and other companies. The authors have indicated that they have no other financial and non-financial conflicts of interest with regard to the content of this article.

## ACKNOWLEDGMENTS

The authors thank Jörgen Möller (Evidera) for technical assistance and Emmanuelle Ghys (XPE Pharma and Science) for editorial assistance and publication coordination on behalf of GlaxoSmithKline.

## REFERENCES

1. Bonhoeffer J, Black S, Izurieta H, Zuber P, Sturkenboom M. Current status and future directions of post-marketing vaccine safety monitoring with focus on USA and Europe. *Biologicals*. 2012;40:393–397.

2. Greenberg M, Simondon F, Saadatian-Elahi M. Perspectives on benefit-risk decision-making in vaccinology: conference report. *Hum Vaccin Immunother*. 2016;12:176–181.

3. Schosser R. Risk/benefit evaluation of drugs: the role of the pharmaceutical industry in Germany. *Eur Surg Res*. 2002;34: 203–207.

4. Mt-Isa S, Hallgreen CE, Wang N, et al. Balancing benefit and risk of medicines: a systematic review and classification of available methodologies. *Pharmacoepidemiol Drug Saf*. 2014;23:667–678.

5. Luteijn JM, White BC, Gunnlaugsdottir H, et al. State of the art in benefit-risk analysis: medicines. *Food Chem Toxicol*. 2012;50:26–32.

6. Mt-Isa S, Ouwens M, Robert V, Gebel M, Schacht A, Hirsch I. Structured benefit-risk assessment: a review of key publications and initiatives on frameworks and methodologies. *Pharm Stat*. 2016;15:324–332.

7. Raju GK, Gurumurthi K, Domike R. Benefit-risk analysis for decision-making: an approach. *Clin Pharmacol Ther*. 2016;100:654–671.

8. Phillips LD, Fasolo B, Zafiropoulos N, Beyer A. Benefit-risk Methodology Project Work Package 2 Report: Applicability of Current Tools and Processes for Regulatory Benefit-Risk Assessment. London, UK: European Medicines Agency; 2012.

9. Chawla AJ, Mytelka DS, McBride SD, et al. Estimating the incremental net health benefit of requirements for cardiovascular risk evaluation for diabetes therapies. *Pharmacoepidemiol Drug Saf*. 2014;23:268–277.
10. Phillips LD, Fasolo B, Zafiropoulous N, et al. Modelling the risk-benefit impact of H1N1 influenza vaccines. *Eur J Public Health*. 2013;23:674–678.

11. Cho BH, Clark TA, Messonnier NE, Ortega-Sanchez IR, Weintraub E, Messonnier ML. MCV vaccination in the presence of vaccine-associated Guillain-Barre syndrome risk: a decision analysis approach. *Vaccine*. 2010;28:817–822.

12. Clark A, Jit M, Andrews N, Atchison C, Edmunds WJ, Sanderson C. Evaluating the potential risks and benefits of infant rotavirus vaccination in England. *Vaccine*. 2014;32:3604–3610.

13. Clark M, Cameron DW. The benefits and risks of Bacille Calmette-Guerin vaccination among infants at high risk for both tuberculosis and severe combined immunodeficiency: assessment by Markov model. *BMC Pediatr*. 2006;6:5.

14. Desai R, Cortese MM, Meltzer MI, et al. Potential intussusception risk versus benefits of rotavirus vaccination in the United States. *Pediatr Infect Dis J*. 2013;32:1–7.

15. Lynd LD, Marra CA, Najafzadeh M, Sadatsafavi M. A quantitative evaluation of the regulatory assessment of the benefits and risks of rofecoxib relative to naproxen: an application of the incremental net-benefit framework. *Pharmacoepidemiol Drug Saf*. 2010;19:1172–1180.

16. Lynd LD, Najafzadeh M, Colley L, et al. Using the incremental net benefit framework for quantitative benefit-risk analysis in regulatory decision-making—a case study of alosetron in irritable bowel syndrome. *Value Health*. 2010;13:411–417.

17. Maro JC, Fryback DG, Lieu TA, Lee GM, Martin DB. Responding to vaccine safety signals during pandemic influenza: a modeling study. *PLoS One*. 2014;9:e115553.

18. Nokes DJ, Anderson RM. Vaccine safety versus vaccine efficacy in mass immunisation programmes. *Lancet*. 1991;338(8778):1309–1312.

19. Benefit-Risk Balance for Marketed Drugs. Evaluating Safety Signals. Geneva, Switzerland: Council for International Organizations of Medical Sciences; 1998.

20. Arlegui H, Bollaerts K, Salvo F, et al. A systematic review of methodological approaches for benefit-risk assessment of vaccines. *Pharmacoepidemiol Drug Saf*. 2017;26(Suppl 2):3–636.

21. Arlegui H, Nachbaur G, Praet N, Begaud B. Quantitative benefit-risk models used for rotavirus vaccination: a systematic review. *Open Forum Infect Dis*. 2020;7: ofaa087.

22. Caro JJ. Discretely integrated condition event (DICE) simulation for pharmacoeconomics. *Pharmacoeconomics*. 2016;34:665–672.

23. Microsoft Office website. Excel; 2016. Available at: https://products.office.com/fr/excel. Accessed July 24, 2020.

24. Deniz B, Ambavane A, Yang S, et al. Treatment sequences for advanced renal cell carcinoma: a health economic assessment. *PLoS One*. 2019;14, e0215761.

25. Pennington B, Filby A, Owen L, Taylor M. Smoking cessation: a comparison of two model structures. *Pharmacoeconomics*. 2018;36:1101–1112.

26. French Health Authority (HAS). *Health Economic Evaluation of Biological Treatments in the Management of Rheumatoid Arthritis*; 2019. Available at: https://www.has-sante.fr/jcms/c_2580906/fr/evaluation-medico-economique-des-traitements-de-fond-biologiques-dans-la-prise-en-charge-de-la-polyarthrite-rhumatoide.

27. Caro JJ, Moller J. Adding events to a Markov model using DICE simulation. *Med Decis Making*. 2018;38:235–245.

28. Moller J, Davis S, Stevenson M, Caro JJ. Validation of a DICE simulation against a discrete event simulation implemented entirely in code. *Pharmacoeconomics*. 2017;35: 1103–1109.

29. Microsoft Office website. Visual Basic for Applications (VBA); 2016. Available at: https://docs.microsoft.com/en-us/office/vba/api/overview/#vba-programming-in-office. Accessed July 24, 2020.

30. Clark A, Black R, Tate J, et al. Estimating global, regional and national rotavirus deaths in children aged <5 years: current approaches, new analyses and proposed improvements. *PLoS One*. 2017;12, e0183392.

31. Lanata CF, Fischer-Walker CL, Olascoaga AC, et al. Global causes of diarrheal disease mortality in children <5 years of age: a systematic review. *PLoS One*. 2013;8, e72788.

32. Troeger C, Khalil IA, Rao PC, et al. Rotavirus vaccination and the global burden of rotavirus diarrhea among children younger than 5 years. *JAMA Pediatr*. 2018;172:958–965.

33. World Health Organization. *Summary of Key Characteristics of Currently WHO-Pre-Qualified Rotavirus Vaccines*; 2019. Available at: https://www.who.int/ immunization/diseases/rotavirus/ WHO_Summary_xtics_PQ'd_rota_ vaccines.PDF?ua=1. Accessed July 24, 2020.

34. Deen J, Lopez AL, Kanungo S, et al. Improving rotavirus vaccine coverage: can newer-generation and locally produced vaccines help? *Hum Vaccin Immunother*. 2018;14:495–499.

35. Ruiz-Palacios GM, Perez-Schael I, Velazquez FR, et al. Safety and efficacy of an attenuated vaccine against severe rotavirus gastroenteritis. *N Engl J Med*. 2006;354:11–22.

36. Vesikari T, Matson DO, Dennehy P, et al. Safety and efficacy of a pentavalent human-bovine (WC3) reassortant rotavirus vaccine. *N Engl J Med*. 2006;354:23–33.

37. Bines JE, Patel M, Parashar U. Assessment of postlicensure safety of
rotavirus vaccines, with emphasis on
intussusception. *J Infect Dis.*
2009;200(Suppl 1):S282–S290.

38. Jiang J, Jiang B, Parashar U, Nguyen T, Bines J, Patel MM. Childhood intussusception: a literature review. *PLoS One*. 2013;8: e68482.

39. Rosillon D, Buyse H, Friedland LR, Ng SP, Velazquez FR, Breuer T. Risk of intussusception after rotavirus vaccination: meta-analysis of postlicensure studies. *Pediatr Infect Dis J.* 2015;34:763–768.

40. Dong R, Yang Y-F, Chen G, Shen Z, Zheng S. Risk of intussusception after rotavirus vaccination: a meta-analysis. *Int J Clin Exp Med*. 2016;9: 1306–1313.

41. Rha B, Tate JE, Weintraub E, et al. Intussusception following rotavirus vaccination: an updated review of the available evidence. *Expert Rev Vaccin.* 2014;13:1339–1348.

42. Stowe J, Andrews N, Ladhani S, Miller E. The risk of intussusception following monovalent rotavirus vaccination in England: a self-controlled case-series evaluation. *Vaccine*. 2016;34:3684–3689.

43. Haut Conseil de la Santé Publique (HCSP). *Rotavirus Infections: Suspension of Infant Vaccination Recommendations*; 2015. Available at: http://www.hcsp.fr/explore.cgi/avisrapportsdomaine?clefr=501. Accessed July 24, 2020.

44. Agence Nationale de Sécurité des Médicaments et des Produits de Santé (ANSM). *Meeting Report, Meeting of the Pharmacovigilance Technical Committee*; 2015. Available at: http://ansm.sante.fr/content/download/75433/958819/version/2/file/CR-CT-Pharmacovigilance-012015023?rotavirus.pdf. Accessed July 24, 2020.

45. European Medicines Agency. Rotarix: a summary of the European public assessment report. Available at:

http://www.ema.europa.eu/ema/
index.jsp?curl=pages/medicines/
human/medicines/000639/human_
med_001043.
jsp&mid=WC0b01ac058001d124.
[Accessed 24 July 2020]

46. Ledent E, Arlegui H, Buyse H, et al. Benefit versus risk assessment of rotavirus vaccination in France: a simulation and modeling analysis. *BioDrugs*. 2018;32:139–152.

47. European Medicines Agency.
*Committee for Medicinal Products for Human Use (CHMP). Work Plan 2018.*
*Benefit/Risk Methodology*; 2018.
Available at: http://www.ema.europa.eu/docs/en_GB/document_library/Work_programme/2011/01/WC500101505.pdf. Accessed July 24, 2020.

48. US Dept Health and Human Services, Food and Drug Administration. *Benefit-Risk Assessment in Drug Regulatory Decision-Making*. Draft *PDUFA VI Implementation Plan (FY 2018-2022)*; 2018. Available at: https://www.fda.gov/downloads/ForIndustry/UserFees/PrescriptionDrugUserFee/UCM602885.pdf. Accessed July 24, 2020.

49. Caro JJ, Moller J. Advantages and disadvantages of discrete-event simulation for health economic analyses. *Expert Rev Pharmacoecon Outcomes Res*. 2016;16:327–329.

50. Zaric GS. The impact of ignoring population heterogeneity when Markov models are used in cost-effectiveness analysis. *Med Decis Making*. 2003;23:379–396.

51. Adlhoch C, Hoehne M, Littmann M, et al. Rotavirus vaccine effectiveness and case-control study on risk factors for breakthrough infections in Germany, 2010-2011. *Pediatr Infect Dis J.* 2013;32:e82–e89.

52. Arena C, Amoros JP, Vaillant V, et al. Acute diarrhea in adults consulting a

general practitioner in France during
winter: incidence, clinical
characteristics, management and risk
factors. *BMC Infect Dis*. 2014;14:574.

53. Tate JE, Simonsen L, Viboud C, et al. Trends in intussusception hospitalizations among US infants, 1993-2004: implications for monitoring the safety of the new rotavirus vaccination program. *Pediatrics*. 2008;121:e1125–e1132.

54. Standaert B, Strens D, Alwan A, Raes M. Medium- to long-term impact of rotavirus vaccination on hospital care in Belgium: a 7-year follow-up of the rotavirus Belgium impact study (RotaBIS). *Infect Dis Ther*. 2016;5:31–44.

Address correspondence to: Hugo Arlegui, PharmD, MSc, PhD, 146 Rue Léo Saignat, 33076, Bordeaux Cedex, France. E-mail: hugo.arlegui@gmail.com
APPENDIX

<table><caption>Table S1. List of Conditions used in the rotavirus vaccination DICE qBRm</caption><thead><tr><td>Name</td><td>Definition</td></tr></thead><tbody><tr><td>ID</td><td>Sequential number for each profile (auto)</td></tr><tr><td>Time</td><td>Mandatory system condition that tracks simulation time</td></tr><tr><td>TimeHorizon</td><td>User-selected duration of simulation</td></tr><tr><td>IntervNum</td><td>System, used when multiple interventions</td></tr><tr><td>StrategyVacc</td><td>Index of the current treatment (non-vaccination versus vaccination)</td></tr><tr><td>Age</td><td>Age at any time during execution</td></tr><tr><td>BsAgeDose1</td><td>Age at vaccine dose 1</td></tr><tr><td>Cohort</td><td>Birth cohort followed 5 years</td></tr><tr><td>CycleRVGE</td><td>User-selected duration of RVGE cycle</td></tr><tr><td>BsRvgeHospRate</td><td>Baseline RVGE hospitalisation rate</td></tr><tr><td>RvgeHospRate</td><td>RVGE hospitalisation rate according to time</td></tr><tr><td>NumHospRvge</td><td>Number of hospitalisations due to RVGE</td></tr><tr><td>NumDeathRvge</td><td>Number of deaths due to RVGE</td></tr><tr><td>RvgeFatalityRate</td><td>Case fatality rate for RVGE</td></tr><tr><td>CycleIS</td><td>User-selected duration of IS cycle</td></tr><tr><td>BsIsHospRate</td><td>Baseline IS hospitalisation rate</td></tr><tr><td>IsHospRate</td><td>IS hospitalisation rate according to time</td></tr><tr><td>NumHospIs</td><td>Number of hospitalisations due to IS</td></tr><tr><td>NumDeathIs</td><td>Number of deaths due to IS</td></tr><tr><td>IsFatalityRate</td><td>Case fatality rate due to IS</td></tr><tr><td>BsCoverage1</td><td>Vaccine coverage dose 1 at baseline</td></tr><tr><td>CurCoverage1</td><td>Vaccine coverage dose 1 according to time</td></tr><tr><td>Dose</td><td>Index of the current dose injection (Dose 1 versus Dose 2)</td></tr><tr><td>BsIsRR1</td><td>Relative risk of IS following dose 1 of rotavirus vaccination at baseline</td></tr><tr><td>CurlsRR1</td><td>Relative risk of IS following dose 1 of rotavirus vaccination according to time</td></tr><tr><td>BsRvgeRR1</td><td>Vaccine Effectiveness following dose1 of rotavirus vaccination at baseline</td></tr><tr><td>CurRvgeRR1</td><td>Vaccine Effectiveness following dose1 of rotavirus vaccination according to time</td></tr><tr><td>BsAgeDelayD1D2</td><td>Delay between dose 1 and dose 2</td></tr><tr><td>BsAgeDose2</td><td>Age at vaccine dose 2</td></tr><tr><td>BsRvgeRR2</td><td>Vaccine Effectiveness following dose2 of rotavirus vaccination at baseline</td></tr><tr><td>CurRvgeRR2</td><td>Vaccine Effectiveness following dose2 of rotavirus vaccination according to time</td></tr><tr><td>BsIsRR2</td><td>Relative risk of IS following dose 2 of rotavirus vaccination at baseline</td></tr><tr><td>CurlsRR2</td><td>Relative risk of IS following dose 2 of rotavirus vaccination according to time</td></tr><tr><td>RiskPeriod</td><td>Risk period of vaccine-induced IS following vaccination</td></tr><tr><td>BsCoverage2</td><td>Vaccine coverage dose 2 at baseline</td></tr><tr><td>CurCoverage2</td><td>Vaccine coverage dose 2 according to time</td></tr><tr><td>NextEventTime</td><td>System, keeps track of time for next event</td></tr><tr><td>NextEvent</td><td>System, keeps track of which event is next</td></tr></tbody></table>

IS, intussusception; RVGE, rotavirus gastroenteritis; DICE, discretely integrated condition event; qBRm, quantitative benefit-risk model.
<table><caption>Table S2. List of Events used in the rotavirus vaccination DICE qBRm</caption><thead><tr><td>Name</td><td>Definition</td></tr></thead><tbody><tr><td>Start</td><td>Initialize execution for intervention</td></tr><tr><td>RVGE</td><td>Occurrence of the RVGE</td></tr><tr><td>Intussusception</td><td>Occurrence of the IS</td></tr><tr><td>Vaccinate1</td><td>Initiate rotavirus vaccination</td></tr><tr><td>VaccineEff</td><td>Initiate vaccine effectiveness of rotavirus vaccination</td></tr><tr><td>VaccineRisk</td><td>Initiate IS risk following rotavirus vaccination</td></tr><tr><td>Vaccinate2</td><td>Complete rotavirus vaccination</td></tr><tr><td>End</td><td>End execution for intervention</td></tr></tbody></table>

IS, intussusception; RVGE, rotavirus gastroenteritis; DICE, discretely integrated condition event; qBRm, quantitative benefit-risk model.

<table><caption>Table S3. List of Outputs for each strategy used in the rotavirus vaccination DICE qBRm</caption><thead><tr><td>Name</td><td>Definition</td></tr></thead><tbody><tr><td>ctHospRvgeVacc</td><td>Number of RVGE hospitalisations following vaccination strategy</td></tr><tr><td>ctDeathRvgeVacc</td><td>Number of RVGE deaths following vaccination strategy</td></tr><tr><td>ctHospIsVacc</td><td>Number of IS hospitalisations following vaccination strategy</td></tr><tr><td>ctDeathIsVacc</td><td>Number of IS deaths following vaccination strategy</td></tr><tr><td>ctHospRvgeNoVacc</td><td>Number of RVGE hospitalisations following no vaccination strategy</td></tr><tr><td>ctDeathRvgeNoVacc</td><td>Number of RVGE deaths following no vaccination strategy</td></tr><tr><td>ctHospIsNoVacc</td><td>Number of IS hospitalisations following no vaccination strategy</td></tr><tr><td>ctDeathIsNoVacc</td><td>Number of IS deaths following no vaccination strategy</td></tr><tr><td>RepNum</td><td>System; Replication number</td></tr></tbody></table>

IS, intussusception; RVGE, rotavirus gastroenteritis; DICE, discretely integrated condition event; qBRm, quantitative benefit-risk model.