Challenges of COVID-19 Case Forecasting in the US, 2020-2021
Authors: Lopez Velma K , Cramer Estee Y , Pagano Robert , Drake John M , O’Dea Eamon B , Adee Madeline , Ayer Turgay , Chhatwal Jagpreet , Dalgic Ozden O , Ladd Mary A , Linas Benjamin P , Mueller Peter P , Xiao Jade , Bracher Johannes , Rivadeneira Alvaro J Castro , Gerding Aaron , Gneiting Tilmann , Huang Yuxin , Jayawardena Dasuni , Kanji Abdul H , Le Khoa , Mühlemann Anja , Niemi Jarad , Ray Evan L , Stark Ariane , Wang Yijin , Wattanachit Nutcha , Zorn Martha W , Pei Sen , Shaman Jeffrey , Yamana Teresa K , Tarasewicz Samuel R , Wilson Daniel J , Baccam Sid , Gurung Heidi , Stage Steve , Suchoski Brad , Gao Lei , Gu Zhiling , Kim Myungjin , Li Xinyi , Wang Guannan , Wang Lily , Wang Yueying , Yu Shan , Gardner Lauren , Jindal Sonia , Marshall Maximilian , Nixon Kristen , Dent Juan , Hill Alison L , Kaminsky Joshua , Lee Elizabeth C , Lemaitre Joseph C , Lessler Justin , Smith Claire P , Truelove Shaun , Kinsey Matt , Mullany Luke C. , Rainwater-Lovett Kaitlin , Shin Lauren , Tallaksen Katharine , Wilson Shelby , Karlen Dean , Castro Lauren , Fairchild Geoffrey , Michaud Isaac , Osthus Dave , Bian Jiang , Cao Wei , Gao Zhifeng , Ferres Juan Lavista , Li Chaozhuo , Liu Tie-Yan , Xie Xing , Zhang Shun , Zheng Shun , Chinazzi Matteo , Davis Jessica T , Mu Kunpeng , Piontti Ana Pastore y , Vespignani Alessandro , Xiong Xinyue , Walraven Robert , Chen Jinghui , Gu Quanquan , Wang Lingxiao , Xu Pan , Zhang Weitong , Zou Difan , Gibson Graham Casey , Sheldon Daniel , Srivastava Ajitesh , Adiga Aniruddha , Hurt Benjamin , Kaur Gursharn , Lewis Bryan , Marathe Madhav , Peddireddy Akhil Sai , Porebski Przemyslaw , Venkatramanan Srinivasan , Wang Lijing , Prasad Pragati V , Walker Jo W , Webber Alexander E , Slayton Rachel B , Biggerstaff Matthew , Reich Nicholas G , Johansson Michael 
Published: June 2023

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.

Author Summary

As SARS-CoV-2 began to spread throughout the world in early 2020, modelers played a critical role in predicting how the epidemic could take shape. Short-term forecasts of epidemic outcomes (for example, infections, cases, hospitalizations, or deaths) provided useful information to support pandemic planning, resource allocation, and intervention. Yet, infectious disease forecasting is still a nascent science, and the reliability of different types of forecasts is unclear. We retrospectively evaluated COVID-19 case forecasts, which were often unreliable. For example, forecasts did not anticipate the speed of increase in cases in early winter 2020. This analysis provides insights on specific problems that could be addressed in future research to improve forecasts and their use. Identifying the strengths and weaknesses of forecasts is critical to improving forecasting for current and future public health responses.