Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations
Creators: name: Sherratt, Katharine
name: Gruson, Hugo
name: Grah, Rok
name: Johnson, Helen
name: Niehus, Rene
name: Prasse, Bastian
name: Sandmann, Frank
name: Deuschel, Jannik
name: Wolffram, Daniel
name: Abbott, Sam
name: Ullrich, Alexander
name: Gibson, Graham
name: Ray, Evan L.
name: Reich, Nicholas G.
name: Sheldon, Daniel
name: Wang, Yijin
name: Wattanachit, Nutcha
name: Wang, Lijing
name: Trnka, Jan
name: Obozinski, Guillaume
name: Sun, Tao
name: Thanou, Dorina
name: Pottier, Loic
name: Krymova, Ekaterina
name: Meinke, Jan H.
name: Barbarossa, Maria Vittoria
name: Leithäuser, Neele
name: Mohring, Jan
name: Schneider, Johanna
name: Wlazło, Jarosław
name: Fuhrmann, Jan
name: Lange, Berit
name: Rodiah, Isti
name: Baccam, Prasith
name: Gurung, Heidi
name: Stage, Steven
name: Suchoski, Bradley
name: Budzinski, Jozef
name: Walraven, Robert
name: Villanueva, Inmaculada
name: Tucek, Vit
name: Smid, Martin
name: Zajíček, Milan
name: Pérez Álvarez, Cesar
name: Reina, Borja
name: Bosse, Nikos I.
name: Meakin, Sophie R.
name: Castro, Lauren
name: Fairchild, Geoffrey
name: Michaud, Isaac
name: Osthus, Dave
name: Alaimo Di Loro, Pierfrancesco
name: Maruotti, Antonello
name: Eclerová, Veronika
name: Kraus, Andrea
name: Kraus, David
name: Pribylova, Lenka
name: Dimitris, Bertsimas
name: Li, Michael Lingzhi
name: Saksham, Soni
name: Dehning, Jonas
name: Mohr, Sebastian
name: Priesemann, Viola
name: Redlarski, Grzegorz
name: Bejar Haro, Benjamin
name: Ardenghi, Giovanni
name: Parolini, Nicola
name: Ziarelli, Giovanni
name: Bock, Wolfgang
name: Heyder, Stefan
name: Hotz, Thomas
name: Singh, David E.
name: Guzman-Merino, Miguel
name: Aznarte, Jose L.
name: Moriña, David
name: Alonso, Sergio
name: Álvarez, Enric
name: López, Daniel
name: Prats, Clara
name: Burgard, Jan Pablo
name: Rodloff, Arne
name: Zimmermann, Tom
name: Kuhlmann, Alexander
name: Zibert, Janez
name: Pennoni, Fulvia
name: Divino, Fabio
name: Català, Marti
name: Lovison, Gianfranco
name: Giudici, Paolo
name: Tarantino, Barbara
name: Bartolucci, Francesco
name: Jona Lasinio, Giovanna
name: Mingione, Marco
name: Farcomeni, Alessio
name: Srivastava, Ajitesh
name: Montero-Manso, Pablo
name: Adiga, Aniruddha
name: Hurt, Benjamin
name: Lewis, Bryan
name: Marathe, Madhav
name: Porebski, Przemyslaw
name: Venkatramanan, Srinivasan
name: Bartczuk, Rafal P.
name: Dreger, Filip
name: Gambin, Anna
name: Gogolewski, Krzysztof
name: Gruziel-Słomka, Magdalena
name: Krupa, Bartosz
name: Moszyński, Antoni
name: Niedzielewski, Karol
name: Nowosielski, Jedrzej
name: Radwan, Maciej
name: Rakowski, Franciszek
name: Semeniuk, Marcin
name: Szczurek, Ewa
name: Zieliński, Jakub
name: Kisielewski, Jan
name: Pabjan, Barbara
name: Kirsten, Holger
name: Kheifetz, Yuri
name: Scholz, Markus
name: Biecek, Przemysław
name: Bodych, Marcin
name: Filinski, Maciej
name: Idzikowski, Radoslaw
name: Krueger, Tyll
name: Ozanski, Tomasz
name: Bracher, Johannes
name: Funk, Sebastian
Publisher: name: Universitäts- und Landesbibliothek Sachsen-Anhalt
Year: 2023
Subjects: subject: 610

description: Background: Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. Methods: We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1–4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance. Results: Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models’ forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models’ forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models’ forecasts of deaths (N=763 predictions from 20 models). Across a 1–4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. Conclusions: Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks.