Evaluation of individual and ensemble probabilistic forecasts of COVID-19
mortality in the US
Authors: Cramer Estee Y , Ray Evan L , Lopez Velma K , Bracher Johannes , Brennen Andrea , Castro Rivadeneira Alvaro J , Gerding Aaron , Gneiting Tilmann , House Katie H , Huang Yuxin , Jayawardena Dasuni , Kanji Abdul H , Khandelwal Ayush , Le Khoa , Mühlemann Anja , Niemi Jarad , Shah Apurv , Stark Ariane , Wang Yijin , Wattanachit Nutcha , Zorn Martha W , Gu Youyang , Jain Sansiddh , Bannur Nayana , Deva Ayush , Kulkarni Mihir , Merugu Srujana , Raval Alpan , Shingi Siddhant , Tiwari Avtansh , White Jerome , Abernethy Neil F , Woody Spencer , Dahan Maytal , Fox Spencer , Gaither Kelly , Lachmann Michael , Meyers Lauren Ancel , Scott James G , Tec Mauricio , Srivastava Ajitesh , George Glover E , Cegan Jeffrey C , Dettwiller Ian D , England William P , Farthing Matthew W , Hunter Robert H , Lafferty Brandon , Linkov Igor , Mayo Michael L , Parno Matthew D , Rowland Michael A , Trump Benjamin D , Zhang-James Yanli , Chen Samuel , Faraone Stephen V , Hess Jonathan , Morley Christopher P , Salekin Asif , Wang Dongliang , Corsetti Sabrina M , Baer Thomas M , Eisenberg Marisa C , Falb Karl , Huang Yitao , Martin Emily T , McCauley Ella , Myers Robert L , Schwarz Tom , Sheldon Daniel , Gibson Graham Casey , Yu Rose , Gao Liyao , Ma Yian , Wu Dongxia , Yan Xifeng , Jin Xiaoyong , Wang Yu-Xiang , Chen YangQuan , Guo Lihong , Zhao Yanting , Gu Quanquan , Chen Jinghui , Wang Lingxiao , Xu Pan , Zhang Weitong , Zou Difan , Biegel Hannah , Lega Joceline , McConnell Steve , Nagraj VP , Guertin Stephanie L , Hulme-Lowe Christopher , Turner Stephen D , Shi Yunfeng , Ban Xuegang , Walraven Robert , Hong Qi-Jun , Kong Stanley , van de Walle Axel , Turtle James A , Ben-Nun Michal , Riley Steven , Riley Pete , Koyluoglu Ugur , DesRoches David , Forli Pedro , Hamory Bruce , Kyriakides Christina , Leis Helen , Milliken John , Moloney Michael , Morgan James , Nirgudkar Ninad , Ozcan Gokce , Piwonka Noah , Ravi Matt , Schrader Chris , Shakhnovich Elizabeth , Siegel Daniel , Spatz Ryan , Stiefeling Chris , Wilkinson Barrie , Wong Alexander , Cavany Sean , España Guido , Moore Sean , Oidtman Rachel , Perkins Alex , Kraus David , Kraus Andrea , Gao Zhifeng , Bian Jiang , Cao Wei , Ferres Juan Lavista , Li Chaozhuo , Liu Tie-Yan , Xie Xing , Zhang Shun , Zheng Shun , Vespignani Alessandro , Chinazzi Matteo , Davis Jessica T , Mu Kunpeng , y Piontti Ana Pastore , Xiong Xinyue , Zheng Andrew , Baek Jackie , Farias Vivek , Georgescu Andreea , Levi Retsef , Sinha Deeksha , Wilde Joshua , Perakis Georgia , Bennouna Mohammed Amine , Nze-Ndong David , Singhvi Divya , Spantidakis Ioannis , Thayaparan Leann , Tsiourvas Asterios , Sarker Arnab , Jadbabaie Ali , Shah Devavrat , Penna Nicolas Della , Celi Leo A , Sundar Saketh , Wolfinger Russ , Osthus Dave , Castro Lauren , Fairchild Geoffrey , Michaud Isaac , Karlen Dean , Kinsey Matt , Mullany Luke C. , Rainwater-Lovett Kaitlin , Shin Lauren , Tallaksen Katharine , Wilson Shelby , Lee Elizabeth C , Dent Juan , Grantz Kyra H , Hill Alison L , Kaminsky Joshua , Kaminsky Kathryn , Keegan Lindsay T , Lauer Stephen A , Lemaitre Joseph C , Lessler Justin , Meredith Hannah R , Perez-Saez Javier , Shah Sam , Smith Claire P , Truelove Shaun A , Wills Josh , Marshall Maximilian , Gardner Lauren , Nixon Kristen , Burant John C. , Wang Lily , Gao Lei , Gu Zhiling , Kim Myungjin , Li Xinyi , Wang Guannan , Wang Yueying , Yu Shan , Reiner Robert C , Barber Ryan , Gakidou Emmanuela , Hay Simon I. , Lim Steve , Murray Chris J.L. , Pigott David , Gurung Heidi L , Baccam Prasith , Stage Steven A , Suchoski Bradley T , Prakash B. Aditya , Adhikari Bijaya , Cui Jiaming , Rodríguez Alexander , Tabassum Anika , Xie Jiajia , Keskinocak Pinar , Asplund John , Baxter Arden , Oruc Buse Eylul , Serban Nicoleta , Arik Sercan O , Dusenberry Mike , Epshteyn Arkady , Kanal Elli , Le Long T , Li Chun-Liang , Pfister Tomas , Sava Dario , Sinha Rajarishi , Tsai Thomas , Yoder Nate , Yoon Jinsung , Zhang Leyou , Abbott Sam , Bosse Nikos I , Funk Sebastian , Hellewell Joel , Meakin Sophie R , Sherratt Katharine , Zhou Mingyuan , Kalantari Rahi , Yamana Teresa K , Pei Sen , Shaman Jeffrey , Li Michael L , Bertsimas Dimitris , Lami Omar Skali , Soni Saksham , Bouardi Hamza Tazi , Ayer Turgay , Adee Madeline , Chhatwal Jagpreet , Dalgic Ozden O , Ladd Mary A , Linas Benjamin P , Mueller Peter , Xiao Jade , Wang Yuanjia , Wang Qinxia , Xie Shanghong , Zeng Donglin , Green Alden , Bien Jacob , Brooks Logan , Hu Addison J , Jahja Maria , McDonald Daniel , Narasimhan Balasubramanian , Politsch Collin , Rajanala Samyak , Rumack Aaron , Simon Noah , Tibshirani Ryan J , Tibshirani Rob , Ventura Valerie , Wasserman Larry , O’Dea Eamon B , Drake John M , Pagano Robert , Tran Quoc T , Tung Ho Lam Si , Huynh Huong , Walker Jo W , Slayton Rachel B , Johansson Michael A , Biggerstaff Matthew , Reich Nicholas G 
Published: February 2022

Abstract
Short-term probabilistic forecasts of the trajectory of the COVID-19
pandemic in the United States have served as a visible and important
communication channel between the scientific modeling community and both the
general public and decision-makers. Forecasting models provide specific,
quantitative, and evaluable predictions that inform short-term decisions such as
healthcare staffing needs, school closures, and allocation of medical supplies.
Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated,
and synthesized tens of millions of specific predictions from more than 90
different academic, industry, and independent research groups. A multi-model
ensemble forecast that combined predictions from dozens of different research
groups every week provided the most consistently accurate probabilistic
forecasts of incident deaths due to COVID-19 at the state and national level
from April 2020 through October 2021. The performance of 27 individual models
that submitted complete forecasts of COVID-19 deaths consistently throughout
this year showed high variability in forecast skill across time, geospatial
units, and forecast horizons. Two-thirds of the models evaluated showed better
accuracy than a naïve baseline model. Forecast accuracy degraded as models made
predictions further into the future, with probabilistic error at a 20-week
horizon 3-5 times larger than when predicting at a 1-week horizon. This project
underscores the role that collaboration and active coordination between
governmental public health agencies, academic modeling teams, and industry
partners can play in developing modern modeling capabilities to support local,
state, and federal response to outbreaks.

Significance Statement
This paper compares the probabilistic accuracy of short-term forecasts
of reported deaths due to COVID-19 during the first year and a half of the
pandemic in the US. Results show high variation in accuracy between and
within stand-alone models, and more consistent accuracy from an ensemble
model that combined forecasts from all eligible models. This demonstrates
that an ensemble model provided a reliable and comparatively accurate means
of forecasting deaths during the COVID-19 pandemic that exceeded the
performance of all of the models that contributed to it. This work
strengthens the evidence base for synthesizing multiple models to support
public health action.