Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States
Authors: Cramer Estee Y. , Ray Evan L. , Lopez Velma K. , Bracher Johannes , Brennen Andrea , Castro Rivadeneira Alvaro J. , Gerding Aaron , Gneiting Tilmann , House Katie H. , Huang Yuxin , Jayawardena Dasuni , Kanji Abdul H. , Khandelwal Ayush , Le Khoa , Mühlemann Anja , Niemi Jarad , Shah Apurv , Stark Ariane , Wang Yijin , Wattanachit Nutcha , Zorn Martha W. , Gu Youyang , Jain Sansiddh , Bannur Nayana , Deva Ayush , Kulkarni Mihir , Merugu Srujana , Raval Alpan , Shingi Siddhant , Tiwari Avtansh , White Jerome , Abernethy Neil F. , Woody Spencer , Dahan Maytal , Fox Spencer , Gaither Kelly , Lachmann Michael , Meyers Lauren Ancel , Scott James G. , Tec Mauricio , Srivastava Ajitesh , George Glover E. , Cegan Jeffrey C. , Dettwiller Ian D. , England William P. , Farthing Matthew W. , Hunter Robert H. , Lafferty Brandon , Linkov Igor , Mayo Michael L. , Parno Matthew D. , Rowland Michael A. , Trump Benjamin D. , Zhang-James Yanli , Chen Samuel , Faraone Stephen V. , Hess Jonathan , Morley Christopher P. , Salekin Asif , Wang Dongliang , Corsetti Sabrina M. , Baer Thomas M. , Eisenberg Marisa C. , Falb Karl , Huang Yitao , Martin Emily T. , McCauley Ella , Myers Robert L. , Schwarz Tom , Sheldon Daniel , Gibson Graham Casey , Yu Rose , Gao Liyao , Ma Yian , Wu Dongxia , Yan Xifeng , Jin Xiaoyong , Wang Yu-Xiang , Chen YangQuan , Guo Lihong , Zhao Yanting , Gu Quanquan , Chen Jinghui , Wang Lingxiao , Xu Pan , Zhang Weitong , Zou Difan , Biegel Hannah , Lega Joceline , McConnell Steve , Nagraj V. P. , Guertin Stephanie L. , Hulme-Lowe Christopher , Turner Stephen D. , Shi Yunfeng , Ban Xuegang , Walraven Robert , Hong Qi-Jun , Kong Stanley , van de Walle Axel , Turtle James A. , Ben-Nun Michal , Riley Steven , Riley Pete , Koyluoglu Ugur , DesRoches David , Forli Pedro , Hamory Bruce , Kyriakides Christina , Leis Helen , Milliken John , Moloney Michael , Morgan James , Nirgudkar Ninad , Ozcan Gokce , Piwonka Noah , Ravi Matt , Schrader Chris , Shakhnovich Elizabeth , Siegel Daniel , Spatz Ryan , Stiefeling Chris , Wilkinson Barrie , Wong Alexander , Cavany Sean , España Guido , Moore Sean , Oidtman Rachel , Perkins Alex , Kraus David , Kraus Andrea , Gao Zhifeng , Bian Jiang , Cao Wei , Lavista Ferres Juan , Li Chaozhuo , Liu Tie-Yan , Xie Xing , Zhang Shun , Zheng Shun , Vespignani Alessandro , Chinazzi Matteo , Davis Jessica T. , Mu Kunpeng , Pastore y Piontti Ana , Xiong Xinyue , Zheng Andrew , Baek Jackie , Farias Vivek , Georgescu Andreea , Levi Retsef , Sinha Deeksha , Wilde Joshua , Perakis Georgia , Bennouna Mohammed Amine , Nze-Ndong David , Singhvi Divya , Spantidakis Ioannis , Thayaparan Leann , Tsiourvas Asterios , Sarker Arnab , Jadbabaie Ali , Shah Devavrat , Della Penna Nicolas , Celi Leo A. , Sundar Saketh , Wolfinger Russ , Osthus Dave , Castro Lauren , Fairchild Geoffrey , Michaud Isaac , Karlen Dean , Kinsey Matt , Mullany Luke C. , Rainwater-Lovett Kaitlin , Shin Lauren , Tallaksen Katharine , Wilson Shelby , Lee Elizabeth C. , Dent Juan , Grantz Kyra H. , Hill Alison L. , Kaminsky Joshua , Kaminsky Kathryn , Keegan Lindsay T. , Lauer Stephen A. , Lemaitre Joseph C. , Lessler Justin , Meredith Hannah R. , Perez-Saez Javier , Shah Sam , Smith Claire P. , Truelove Shaun A. , Wills Josh , Marshall Maximilian , Gardner Lauren , Nixon Kristen , Burant John C. , Wang Lily , Gao Lei , Gu Zhiling , Kim Myungjin , Li Xinyi , Wang Guannan , Wang Yueying , Yu Shan , Reiner Robert C. , Barber Ryan , Gakidou Emmanuela , Hay Simon I. , Lim Steve , Murray Chris , Pigott David , Gurung Heidi L. , Baccam Prasith , Stage Steven A. , Suchoski Bradley T. , Prakash B. Aditya , Adhikari Bijaya , Cui Jiaming , Rodríguez Alexander , Tabassum Anika , Xie Jiajia , Keskinocak Pinar , Asplund John , Baxter Arden , Oruc Buse Eylul , Serban Nicoleta , Arik Sercan O. , Dusenberry Mike , Epshteyn Arkady , Kanal Elli , Le Long T. , Li Chun-Liang , Pfister Tomas , Sava Dario , Sinha Rajarishi , Tsai Thomas , Yoder Nate , Yoon Jinsung , Zhang Leyou , Abbott Sam , Bosse Nikos I. , Funk Sebastian , Hellewell Joel , Meakin Sophie R. , Sherratt Katharine , Zhou Mingyuan , Kalantari Rahi , Yamana Teresa K. , Pei Sen , Shaman Jeffrey , Li Michael L. , Bertsimas Dimitris , Skali Lami Omar , Soni Saksham , Tazi Bouardi Hamza , Ayer Turgay , Adee Madeline , Chhatwal Jagpreet , Dalgic Ozden O. , Ladd Mary A. , Linas Benjamin P. , Mueller Peter , Xiao Jade , Wang Yuanjia , Wang Qinxia , Xie Shanghong , Zeng Donglin , Green Alden , Bien Jacob , Brooks Logan , Hu Addison J. , Jahja Maria , McDonald Daniel , Narasimhan Balasubramanian , Politsch Collin , Rajanala Samyak , Rumack Aaron , Simon Noah , Tibshirani Ryan J. , Tibshirani Rob , Ventura Valerie , Wasserman Larry , O’Dea Eamon B. , Drake John M. , Pagano Robert , Tran Quoc T. , Ho Lam Si Tung , Huynh Huong , Walker Jo W. , Slayton Rachel B. , Johansson Michael A. , Biggerstaff Matthew , Reich Nicholas G. 
Published: April 2022

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (
https://covid19forecasthub.org/
) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.