<figure><img src="image_1.png" /></figure>

# Infectious disease modelling to inform policy

G.C. Smith* (1), R.R. Kao (2) & M. Walker (3)

(1) National Wildlife Management Centre, Animal and Plant Health Agency, Sand Hutton, York, YO41 1LZ, United Kingdom

(2) Roslin Institute, University of Edinburgh, Easter Bush Campus, Midlothian, EH25 9RG, United Kingdom

(3) Department of Pathobiology and Population Sciences, Royal Veterinary College, Hawkshead Lane, North Mymms, Hatfield, Hertfordshire, AL9 7TA, United Kingdom

*Corresponding author: graham.smith@apha.gov.uk

## Summary

With modelling becoming increasingly important in helping to inform decisions about animal diseases, it is essential that the process be optimised to gain the maximum benefit for the decision-maker. Here, the authors set out ten steps that can improve this process for all concerned. Four steps describe initialisation to ensure that the question, answer and timescale are defined; two steps describe the modelling process and quality assurance; and four steps describe the reporting stage. The authors believe that this greater emphasis at the beginning and end of a modelling project will increase the relevance of the work and understanding of the results, and thus contribute towards better decision-making.

## Keywords

Decision-making – Modelling process – Quality assurance.

## Introduction

In recent decades, models have increasingly been used to influence decision-making, particularly in the policy area. The Department for Environment, Food and Rural Affairs in the United Kingdom (UK) has published requirements for evidence-based decision-making in animal health [1] and bovine tuberculosis (bTB) [2] that clearly state that mathematical/ simulation models should be used to inform the development of policy. Evidence plans for public health (zoonoses) [3] and biodiversity [4] also suggest that modelling may help to pull evidence together to inform future decision-making. In addition, the UK Treasury has reviewed quality assurance of analytical models that inform policy decisions across government [5], and provides guidance for quality assurance in analysis that can be used for modelling [6]. There is also international guidance for conducting modelling studies in animal health [7].

Historically, the UK's 2001 foot and mouth disease epidemic quickly spawned a variety of models [8], all testing different control strategies and potentially supplying information that could be used for decision-making. Modelling has also been used extensively over the past two decades to evaluate different approaches to the control of bTB, both in the UK [9, 10, 11, 12, 13] and elsewhere [14, 15, 16]. More recently,

the emergence and spread of African swine fever in Eastern Europe has prompted a burgeoning of policy-orientated mathematical modelling studies [17, 18, 19, 20], while decision-making on strategies to achieve eradication of pests des petits ruminants is also being supported by modelling [21]. Reviews of modelling have also been produced for viral zoonoses [22], wildlife disease generally [23] and rabies [24] specifically.

A recent review has shown that theory is increasingly being incorporated into publications on the management of wildlife and livestock diseases, with more than two-thirds of the reviewed case studies involving a government agency [7, 25]. It can be inferred from these that a large proportion of publications were used at some level to inform decision-making, although the use of modelling was not necessarily involved. What is not clear from many scientific publications, including those using models, is whether the original study's results were presented to the decision-makers responsible for the issue, or whether they influenced any change in management. Commonly, the actual decision-maker or policy official is not versed in modelling so may be unable to evaluate the weight that should be applied to the model output, since they cannot evaluate the importance of the assumptions and data limitations that are necessarily applied to the model.
Suggestions for documenting the model process to improve transparency already exist in a variety of contexts [26, 27], although there is little evidence that these have had a substantial impact. A standard protocol has been suggested for documenting individual-based and agent-based models [28], which is making headway and may become a standard in model documentation. Guidance on good software practices for maintaining information generated from modelling also exists [29, 30]. In this paper, the authors concentrate on the modelling process with the aim of using the results to inform a decision. This requires the modeller to document the process in such a way that a non-modeller can well understand and make best use of the information. There are some recent guidelines for this, with five principles identified for neglected tropical diseases [27] and four more set out in the general context of evidence synthesis [31]. These principles should apply to any disease modelling and emphasise the importance of screening questions about a model's purpose, assumptions and evidence [32] before relying on its output.

In this paper, the authors assume that a model will be used for predicting different outcomes of disease management decisions, so the purpose is predictive rather than academic or exploratory. They do not discuss the choice of model, but rather provide a toolkit to enhance the whole modelling process.

## Outline

The modelling process should follow a series of predefined steps. Wrong decisions can be very costly in disease control, so maintaining quality and an audit trail is important. These steps can be broken down into three areas:

- initialisation

- production

- reporting.

## Initialisation

Initialisation consists of the following four steps:

1. Clearly define the question to be answered and the choice of output metric (e.g. probability of success, time to achieve eradication, etc.).

2. Agree the timescale of the model process and the output (e.g. results, recommendations and model).

3. Agree the extent of the system under study and options for management.

4. Define and agree the structure(s) of the system.

## Production

This stage consists of the following:

5. Produce the model and decide its parameters.

6. Carry out quality assurance, involving:

- verification

- validation.

## Reporting

Reporting involves the following:

7. Produce a preliminary output and report back.

8. Carry out a sensitivity analysis.

9. Produce a full evaluation of the output.

10. Write a report.

## Details

### 1. The question

Modelling is a formal abstract representation of reality and producing a model is often instructive since one is forced to be precise about the relationships of all components. The decision-maker and other stakeholders should be involved [27] as this builds confidence that the model includes all the necessary parameters. It is also vital that the decision-maker and the modeller both understand what output is required.

The authors highly recommend that this be achieved by drawing a hypothetical output (e.g. a graph of resources versus reduction in disease, or a map showing areas of increased risk) and agreeing how uncertainty should be represented. They suggest this to present not only the 'best' option, but also an estimate of the probability of a particular decision being the best option. The latter requires at least the incorporation of stochastic processes but also integration of parametric uncertainty, which may, in turn, be associated with other variables (e.g. geographic/spatial variability in parameters). For example, culling may be preferred to vaccination for disease control in a specific model and population, but if this is true in only 60% of that model's simulations, then the choice of management strategy may depend more heavily on other factors and may vary among populations.

### 2. The timescale and output

Decision-makers often have a strict timetable. Extending this may not be possible, and decision-makers generally understand that these decisions are made with imperfect information. It is therefore important to decide the timescale in advance, and initial results need to be available early, as secondary questions often occur. Thus, model construction and presentation are often only the first stage.

Requests for 'what if' scenarios inevitably arise. It is important for everyone to be aware of the available resources within any project since iterative reporting commonly leads to 'project creep': boundless additional scenarios and questions being asked within the current project. The scope of additional questions/scenarios needs to be made clear at the outset, akin to terms of reference. Establishing this is also crucial to avoid 'model creep': where the model is being used for situations for which it was not originally
designed. The modeller needs to be very clear when this
may occur and to state that the current model does not al-
low for the suggested scenarios. Using a model outside its
validated working 'envelope' will lead to loss of credibility in
the output.

It is also important to determine cases where models should
not be used for decision-making. In early outbreak situations
with very limited reliable data, models can be constructed
and examined. But if the uncertainty in model input is very
high, no weight should be given to the output at this stage,
and any such model must be adjusted in real time [33]. In
these situations, it may be better to delay producing model
output for decision-making, and concentrate on collecting
suitable data [34]. However, models that are light on data
may be very useful in exploring output sensitivity and helping
to determine which field parameters are the most important
to collect. Indeed, collecting data during the outbreak re-
sponse, including data on efforts to manage the disease thus
far, should be prioritised in such situations, even if it slows
down progress slightly, as it will vastly improve the ability to
manage later outbreaks.

3. Extent and options

It is important to define the scope of the system under
study and to identify which options are being considered
by the decision-maker. This should include a 'business as
usual' scenario as a baseline (or, as an equivalent, a coun-
terfactual scenario to any management/control/interven-
tion scenarios being considered). Requested management
options may be restricted for logistical, practical or political
reasons, so the range of options to model must be defined
at an early stage, but the full range of realistic management
options should be examined for completeness and should
not be otherwise restricted. The output that the modeller
wants is evidence-based policy and not policy-based evi-
dence (i.e. limited options investigated that would likely fa-
vour a specific outcome), although at times there may be
pressure for the latter approach. Since models are generally
more reliable when making comparative predictions (rather
than exact predictions), this full inclusion will make results
more reliable.

4. System structure

The system should then be defined and agreed with the
decision-maker. For complex models, different parts of the
system should be checked or developed in collaboration
with different experts from relevant disciplines. For exam-
ple, testing cattle for bTB can involve different tests that
may be performed in different ways during an outbreak.
Where the process is relatively complex, it is important that
the decision-maker, or other suitable experts, be able to
agree with the structure to be modelled, for example, with a
flow chart.

It is quite common that the exact process is not fully known.
In this case, several different model structures can be pro-
duced and examined, with or without using different param-
eters. Where two or more models appear to be equally 'good'
and it is not possible to choose between them, it is prefer-
able to run them all and consider the combined output for
decision-making (i.e. ensemble modelling). Where there is
a quantitative assessment of model weight, the number of
simulations in the combined output may be chosen by this
weighting. This multi-model approach is used for simulating
climate change scenarios [35], as well as infectious diseases
[36, 37, 38, 39].

In such situations, it is important that decisions be made
based on the accumulated results and not on a subjective
choice of any preferred model based on its output. Indeed,
ensemble disease modelling has started to gain ground in
recent years, with the potential to improve decision-making
and reduce output uncertainty [40, 41].

5. Produce the model

It is not the authors' intention here to define any specific mod-
elling approach. The choice of model and its level of detail
should depend on the exact question, the time available and
the data. The authors do recommend that any model used
for decision-making take account of uncertainty, although
this could be done through stochastic, parametric or struc-
tural means, or a suitable combination that can clearly and
transparently state the appropriate bounds of uncertainty.

Any model coding should be annotated with explanatory
comments. The model should be well documented [28] so
that others can follow the logic, and it should be subject to
version control, with a master list of all versions and their
changes recorded on a single document. The sources of all
data used should be documented, whether peer reviewed,
grey literature or just educated guesses. All assumptions
should be explicitly stated in the documentation, and the
model version used in the final decision should also be ex-
plicitly identified. Where possible, the source code could also
be made available.

6. Quality assurance

It is often assumed that there are two levels of quality assur-
ance, verification and validation, although there is a grey area
between them. Verification means checking that the model is
doing what it was created to do. This can be done by check-
ing programming code, by exploratory sensitivity analysis,
by running the model for known input and output values, or
– in the case of complex models – by creating a second inde-
pendent model and comparing the outputs.

Model validation and improved parameter estimation may be
carried out by using the model to predict an output not used
in its creation, or by predicting secondary, emergent outputs at different scales [42]. This can be an iterative process: the model being updated as new data become available. However, in certain circumstances, such as when modelling disease outbreaks or long-term prospects for disease elimination, it may not be possible to validate a model as similar scenarios may not exist. In these cases, it may still be possible to validate certain sections of the model.

## 7. Preliminary output

Preliminary output of the chosen management/control/inter- vention options should be performed promptly. It is useful to present this output to the decision-maker as this maintains communication, avoids surprises and clarifies the required out- put. It is also an opportunity to check the time frame for delivery and whether the policy requirement has changed in the interim.

## 8. Sensitivity analysis

In terms of decision-making, sensitivity analysis (or parametric uncertainty) can be used to explore the range of possible outcomes. There is no point in changing all the parameters by 10% if some vary by substantially more than others. One can use the bounds of variability (e.g. 80th percentiles) or the bounds of uncertainty in a parameter value. Performing a sensitivity analysis and then reporting the most important driving factors can also help to:

a) identify those parameters with the most important uncertainty where further data collection would be most beneficial;

b) identify those factors that can be managed by the decision-maker to influence the system.

## 9. Provide output

This is the last point at which one can step backwards to adjust the model. At this stage, the work should just be 'cranking the handle' and repeating the model for each set of scenarios. However, production of the results should never be the last step. The output needs to be interpreted by the modeller, preferably in combination with the

decision-maker or another suitable expert. A statistical difference is not necessarily an important biological difference between scenarios. Results should be interpreted with relation to the caveats and assumptions used in the model, but not in relation to the use of the results (i.e. there should be no political agenda or subjectivity in interpretation). Ideally this should be done as an 'honest broker' rather than from an academic ivory tower, or as an advocate [43]. Models can be used to examine scenarios that are not publicly or politically acceptable. They can be used to examine unethical choices. The modeller's role is to provide the output and its interpretation and impose no weight on one set of outcomes over another. Thus, the model does not define the decision, but merely informs the decision-maker on the likely consequences of each choice.

However, during a disease outbreak, it may be useful and even necessary to re-run the model with updated parameter estimates that include the management responses to date [33].

## 10. Report

While there is a clear difference in the production of an academic paper and the requirements of a report for decision-makers, it is often possible to produce a single report that is easily transferred into an academic paper, and thus suitable for peer review, which may enhance credibility. Decision-makers may find it useful to see a flow diagram of the model but may not find the mathematics and/or model code accessible. While these details must be documented (e.g. in an appendix), they are not essential for a wider audience. Similarly, only key results of principal interest and major assumptions need to be included in the main body of a report or paper, with the remaining results and assumptions appended. However, it is important to avoid obfuscating uncertainty or blurring the range of potential outcomes, and to be aware that 'headline' results can be misinterpreted.

## Acknowledgements

The authors would like to thank Richard Budgey for comments on the text.
# La modélisation des maladies infectieuses en appui de l'élaboration des politiques

G.C. Smith, R.R. Kao & M. Walker

## Résumé

Compte tenu de l'importance croissante de la modélisation pour documenter les décisions sur les maladies animales, il est essentiel d'optimiser le processus afin de le rendre le plus profitable possible pour les personnes décisionnaires. Les auteurs définissent dix étapes permettant d'améliorer le processus pour tous les intervenants. Quatre étapes concernent la phase de démarrage et visent à s'assurer que les questions posées, les réponses obtenues et le calendrier sont bien définis ; les deux étapes suivantes portent sur le processus de modélisation et sur l'assurance qualité ; les quatre dernières décrivent la phase d'élaboration des rapports. Les auteurs estiment que cette attention particulière accordée aux phases de démarrage et d'achèvement d'un projet de modélisation rend l'exercice plus pertinent et améliore la compréhension des résultats, ce qui contribue à une meilleure prise de décisions.

## Mots-clés

Assurance qualité – Prise de décisions – Processus de modélisation.

# Modelización de enfermedades infecciosas para fundamentar las políticas

G.C. Smith, R.R. Kao & M. Walker

## Resumen

Dada la creciente importancia que está cobrando la modelización como herramienta para ayudar a fundamentar las decisiones relativas a enfermedades animales, es esencial optimizar el proceso para que las instancias decisorias puedan aprovecharlo al máximo. Los autores exponen diez pasos que pueden mejorar el proceso para cuantos trabajan en este ámbito. En cuatro pasos se describe la inicialización, que sirve para definir debidamente la pregunta, la respuesta y la escala temporal de que se trate. En otros dos pasos se describe el proceso de modelización y de garantía de calidad, mientras que en los últimos cuatro pasos se describe la fase de producción de informes. Los autores consideran que el hecho de otorgar mayor peso a las fases iniciales y finales de un proyecto de modelización hará que el trabajo gane en pertinencia y que se entiendan mejor sus resultados, lo que a su vez contribuye a un proceso más eficaz de adopción de decisiones.

## Palabras clave

Adopción de decisiones – Garantía de calidad – Proceso de modelización.

## References

[1] Department for Environment, Food and Rural Affairs (Defra) (2013). – Animal Health Policy and Implementation Evidence Plan. Defra, London, United Kingdom, 15 pp. Available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/221083/pb13903-evidenceplan-animal-health.pdf (accessed on 27 July 2022).

[2] Department for Environment, Food and Rural Affairs (Defra) (2013). – Bovine Tuberculosis Evidence Plan. Defra, London, United Kingdom, 10 pp. Available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/221083/pb13903-evidenceplan-animal-health.pdf (accessed on 27 July 2022).

uk/government/uploads/system/uploads/attachment_data/file/221077/pb13909-evidenceplan-bovine-tuberculosis.pdf (accessed on 27 July 2022).

[3] Department for Environment, Food and Rural Affairs (Defra) (2013). – Public Health Protection Evidence Plan. Defra, London, United Kingdom, 10 pp. Available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/221063/pb13923-evidenceplan-public-health-protection.pdf (accessed on 27 July 2022).

[4] Department for Environment, Food and Rural Affairs (Defra) (2013). – Biodiversity and Ecosystems Evidence Plan. Defra, London, United Kingdom, 10 pp. Available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/221063/pb13923-evidenceplan-public-health-protection.pdf (accessed on 27 July 2022).
Kingdom, 18 pp. Available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/221078/pb13908-evidenceplan-biodiversity-ecosystems.pdf (accessed on 27 July 2022).

[5] Her Majesty's (HM) Treasury (2013). – Review of quality assurance of government analytical models: final report. HM Treasury, London, United Kingdom, 47 pp. Available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/206946/review_of_qa_of_govt_analytical_models_final_report_040313.pdf (accessed on 23 November 2022).

[6] Her Majesty's (HM) Treasury (2015). – The aqua book: guidance on producing quality analysis for government. HM Treasury, London, United Kingdom, 63 pp. Available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/416478/aqua_book_final_web.pdf (accessed on 23 November 2022).

[7] European Food Safety Authority (EFSA) Panel on Animal Health and Welfare (2009). – Guidance on good practice in conducting scientific assessments in animal health using modelling. *EFSA J.* **7** (12), 1419. https://doi.org/10.2903/j.efsa.2009.1419

[8] Kao R.R. (2002). – The role of mathematical modelling in the control of the 2001 FMD epidemic in the UK. *Trends Microbiol.*, **10** (6), 279–286. https://doi.org/10.1016/S0966-842X(02)02371-5

[9] Smith G.C., Cheeseman C.L., Clifton-Hadley R.S. & Wilkinson D. (2001). – A model of bovine tuberculosis in the badger *Meles meles*: an evaluation of control strategies. *J. Appl. Ecol.*, **38** (3), 509–519. https://doi.org/10.1046/j.1365-2664.2001.00609.x

[10] Anderson R.M. & Trewhella W. (1985). – Population dynamics of the badger (*Meles meles*) and the epidemiology of bovine tuberculosis (*Mycobacterium bovis*). *Philos. Trans. R. Soc. B Biol. Sci.*, **310** (1145), 327–381. https://doi.org/10.1098/rstb.1985.0123

[11] Smith G.C. & Budgey R. (2021). – Simulating the next steps in badger control for bovine tuberculosis in England. *PLoS One*, **16** (3), e0248426. https://doi.org/10.1371/journal.pone.0248426

[12] Birch C.P.D., Goddard A. & Tearne O. (2018). – A new bovine tuberculosis model for England and Wales (BoTMEW) to simulate epidemiology, surveillance and control. *BMC Vet. Res.*, **14** (1), 273. https://doi.org/10.1186/s12917-018-1595-9

[13] Cox D.R., Donnelly C.A., Bourne F.J., Gettinby G., McInerney J.P., Morrison W.I. & Woodroffe R. (2005). – Simple model for tuberculosis in cattle and badgers. *Proc. Natl Acad. Sci. USA*, **102** (49), 17588–17593. https://doi.org/10.1073/pnas.0509003102

[14] Caley P. (2006). – Bovine tuberculosis in brushtail possums: models, dogma and data. *N.Z. J. Ecol.*, **30** (1), 25–34. Available at: https://newzealandecology.org/nzje/2288.pdf (accessed on 23 November 2022).

[15] Gormley A.M., Holland E.P., Barron M.C., Anderson D.P. & Nugent G. (2016). – A modelling framework for predicting the optimal balance between control and surveillance effort in the local eradication of tuberculosis in New Zealand wildlife. *Prev. Vet. Med.*, **125**, 10–18. https://doi.org/10.1016/j.prevetmed.2016.01.007

[16] Ramsey D.S.L. & Efford M.G. (2010). – Management of bovine tuberculosis in brushtail possums in New Zealand: predictions from a spatially explicit, individual-based model. *J. Appl. Ecol.*, **47** (4), 911–919. https://doi.org/10.1111/j.1365-2664.2010.01839.x

[17] Lange M., Reichold A. & Thulke H.-H. (2021). – Modelling wild boar management for controlling the spread of ASF in the areas called white zones (zones blanche). *EFSA Support. Publ.*, **18** (5), 6573E. https://doi.org/10.2903/sp.efsa.2021.EN-6573

[18] O'Neill X., White A., Ruiz-Fons F. & Gortázar C. (2020). – Modelling the transmission and persistence of African swine fever in wild boar in contrasting European scenarios. *Sci. Rep.*, **10** (1), 5895. https://doi.org/10.1038/s41598-020-62736-y

[19] Reichold A., Lange M. & Thulke H.-H. (2022). – Modelling the effectiveness of measures applied in zones dedicated to stop the spread of African swine fever in wild boar when bordering with a region of limited control. *EFSA Support. Publ.*, **19** (5), 7320E. https://doi.org/10.2903/sp.efsa.2022.EN-7320

[20] Croft S., Massei G., Smith G.C., Fouracre D. & Aegerter J.N. (2020). – Modelling spatial and temporal patterns of African swine fever in an isolated wild boar population to support decision-making. *Front. Vet. Sci.*, **7**, 154. https://doi.org/10.3389/fvets.2020.00154

[21] Food and Agriculture Organization of the United Nations (FAO) & World Organisation for Animal Health (WOAH) (2015). – Global strategy for the control and eradication of PPR. In Proc. FAO and OIE International Conference for the control and eradication of peste des petits ruminants (PPR), 31 March to 2 April, Abidjan, Côte d'Ivoire, 255 pp. Available at: https://www.woah.org/app/uploads/2021/12/ ppr-global-strategy-avecannexes-2015-03-28.pdf (accessed on 27 July 2022).

[22] Allen L.J.S., Brown V.L., Jonsson C.B., Klein S.L., Laverty S.M., Magwedere K., Owen J.C. & van den Driessche P. (2012). – Mathematical modeling of viral zoonoses in wildlife. *Nat. Resour. Model.*, **25** (1), 5–51. https://doi.org/10.1111/j.1939-7445.2011.00104.x

[23] Barlow N.D. (1996). – The ecology of wildlife disease control: simple models revisited. *J. Appl. Ecol.*, **33** (2), 303–314. https://doi.org/10.2307/2404752

[24] Smith G.C. (2018). – The role of modelling in predicting rabies and understanding the impact of control measures. In Rabies (A.R. Fooks, ed.). *Rev Sci Tech*, **37** (2), 551–557. https://doi.org/10.20506/rst.37.2.2823

[25] Joseph M.B., Mihaljevic J.R., Arellano A.L., Kueneman J.G., Preston D.L., Cross P.C. & Johnson P.T.J. (2013). – Taming wildlife disease: bridging the gap between science and management. *J. Appl. Ecol.*, **50** (3), 702–712. https://doi.org/10.1111/1365-2664.12084

[26] Schmolke A., Thorbek P., DeAngelis D.L. & Grimm V. (2010). – Ecological models supporting environmental decision making: a strategy for the future. *Trends Ecol. Evol.*, **25** (8), 479–486. https://doi.org/10.1016/j.tree.2010.05.001

[27] Behrend M.R., Basáñez M.-G., Hamley J.I.D., Porco T.C., Stolk W.A., Walker M. & de Vlas S.J. for the NTD Modelling Consortium (2020). – Modelling for policy: the five principles of the Neglected Tropical Diseases Modelling Consortium. *PLoS Negl. Trop. Dis.*, **14** (4), e0008033. https://doi.org/10.1371/journal.pntd.0008033

[28] Grimm V., Railsback S.F. [...] & Ayllón D. (2020). – The ODD protocol for describing agent-based and other simulation models: a second update to improve clarity, replication, and structural realism. *J. Artif. Soc. Social Simul.*, **23** (2), 7. https://doi.org/10.18564/jasss.4259

[29] Crusoe M.R. & Brown C.T. (2016). – Walking the talk: adopting and adapting sustainable scientific software development processes in a small biology lab. *J. Open Res. Softw.*, **4** (1), e44. https://doi.org/10.5334/jors.35
[30] Oberkampf W.L., Trucano T.G. & Hirsch C. (2004). – Verification, validation, and predictive capability in computational engineering and physics. *Appl. Mech. Rev.*, **57** (5), 345–384. https://doi.org/10.1115/1.1767847

[31] Donnelly C.A., Boyd I., Campbell P., Craig C., Vallance P., Walport M., Whitty C.J.M., Woods E. & Wormald C. (2018). – Four principles to make evidence synthesis more useful for policy. *Nature*, **558** (7710), 361–364. https://doi.org/10.1038/d41586-018-05414-4

[32] Grimm V., Johnston A.S.A., Thulke H.-H., Forbes V.E. & Thorbek P. (2020). – Three questions to ask before using model outputs for decision support. *Nat. Commun.*, **11** (1), 4959. https://doi.org/10.1038/s41467-020-17785-2

[33] Probert W.J.M., Jewell C.P., Werkman M., Fonnesbeck C.J., Goto Y., Runge M.C., Sekiguchi S., Shea K., Keeling M.J., Ferrari M.J. & Tildesley M.J. (2018). – Real-time decision-making during emergency disease outbreaks. *PLoS Comput. Biol.*, **14** (7), e1006202. https://doi.org/10.1371/journal.pcbi.1006202

[34] Thompson R.N., Gilligan C.A. & Cunniffe N.J. (2018). – Control fast or control smart: when should invading pathogens be controlled? *PLoS Comput. Biol.*, **14** (2), e1006014. https://doi.org/10.1371/journal.pcbi.1006014

[35] Semenov M.A. & Stratonovitch P. (2010). – Use of multi-model ensembles from global climate models for assessment of climate change impacts. *Clim. Res.*, **41** (1), 1–14. https://doi.org/10.3354/cr00836

[36] Smith M.E., Singh B.K., Irvine M.A., Stolk W.A., Subramanian S., Hollingsworth T.D. & Michael E. (2017). – Predicting lymphatic filariasis transmission and elimination dynamics using a multi-model ensemble framework. *Epidemics*, **18**, 16–28. https://doi.org/10.1016/j.epidem.2017.02.006

[37] Ruiz D., Brun C., Connor S.J., Omumbo J.A., Lyon B. & Thomson M.C. (2014). – Testing a multi-malaria-model ensemble against 30 years of data in the Kenyan highlands. *Malar. J.*, **13**, 206. https://doi.org/10.1186/1475-2875-13-206

[38] Viboud C., Sun K., Gaffey R., Ajelli M., Fumanelli L., Merler S., Zhang Q., Chowell G., Simonsen L., Vespignani A. & the RAPIDD

Ebola Forecasting Challenge group (2018). – The RAPIDD ebola forecasting challenge: synthesis and lessons learnt. *Epidemics*, **22**, 13–21. https://doi.org/10.1016/j.epidem.2017.08.002

[39] Reich N.G., McGowan C.J. [...] & Shaman J. (2019). – Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S. *PLoS Comput. Biol.*, **15** (11), e1007486. https://doi.org/10.1371/journal.pcbi.1007486

[40] Webb C.T., Ferrari M., Lindström T., Carpenter T., Dürr S., Garner G., Jewell C., Stevenson M., Ward M.P., Werkman M., Backer J. & Tildesley M. (2017). – Ensemble modelling and structured decision-making to support emergency disease management. *Prev. Vet. Med.*, **138**, 124–133. https://doi.org/10.1016/j.prevetmed.2017.01.003

[41] Ray E.L. & Reich N.G. (2018). – Prediction of infectious disease epidemics via weighted density ensembles. *PLoS Comput. Biol.*, **14** (2), e1005910. https://doi.org/10.1371/journal.pcbi.1005910

[42] Grimm V. & Railsback S.F. (2012). – Pattern-oriented modelling: a ‘multi-scope’ for predictive systems ecology. *Philos. Trans. R. Soc. B Biol. Sci.*, **367** (1586), 298–310. https://doi.org/10.1098/rstb.2011.0180

[43] Pielke Jr R.A. (2007). – The honest broker: making sense of science in policy and politics. Cambridge University Press, Cambridge, United Kingdom, 200 pp. https://doi.org/10.1017/CBO9780511818110

© 2023 Smith G.C., Kao R.R. & Walker M.; licensee the World Organisation for Animal Health. This is an open access article distributed under the terms of the Creative Commons Attribution IGO Licence (https://creativecommons.org/licenses/by/3.0/igo/legalcode), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is properly cited. In any reproduction of this article there should not be any suggestion that WOAH or this article endorses any specific organisation, product or service. The use of the WOAH logo is not permitted. This notice should be preserved along with the article's original URL.