Challenges of COVID-19 Case Forecasting in the US, 2020–2021
Creators: name: Bracher, Johannes
name: Xie, Xing
name: Zhang, Weitong
name: Hill, Alison L.
name: Suchoski, Brad
name: Gu, Quanquan
name: Chhatwal, Jagpreet
name: Johansson, Michael A.
name: Gao, Zhifeng
name: Tallaksen, Katharine
name: Biggerstaff, Matthew
name: Kim, Myungjin
name: Cramer, Estee Y.
name: Mullany, Luke C.
name: Dent, Juan
name: Hurt, Benjamin
name: Walraven, Robert
name: Gurung, Heidi
name: Gerding, Aaron
name: Kaur, Gursharn
name: Lee, Elizabeth C.
name: Li, Xinyi
name: Yu, Shan
name: Wilson, Daniel J.
name: Le, Khoa
name: Nixon, Kristen
name: Xiong, Xinyue
name: Stage, Steve
name: Osthus, Dave
name: Adee, Madeline
name: Karlen, Dean
name: Prasad, Pragati V.
name: Gardner, Lauren
name: Chinazzi, Matteo
name: Zorn, Martha W.
name: Stark, Ariane
name: Wilson, Shelby
name: Srivastava, Ajitesh
name: Reich, Nicholas G.
name: Shaman, Jeffrey
name: Rainwater-Lovett, Kaitlin
name: Wang, Yueying
name: Huang, Yuxin
name: Wang, Lijing
name: Dalgic, Ozden O.
name: Xiao, Jade
name: Peddireddy, Akhil Sai
name: Truelove, Shaun
name: Lewis, Bryan
name: Lopez, Velma K.
name: Wattanachit, Nutcha
name: Wang, Guannan
name: Baccam, Sid
name: Zheng, Shun
name: Lessler, Justin
name: Ladd, Mary A.
name: Kaminsky, Joshua
name: Webber, Alexander E.
name: Sheldon, Daniel
name: Pei, Sen
name: Pastore y Piontti, Ana
name: Michaud, Isaac
name: Gneiting, Tilmann
name: Venkatramanan, Srinivasan
name: Castro, Lauren
name: Linas, Benjamin P.
name: Porebski, Przemyslaw
name: Mu, Kunpeng
name: Chen, Jinghui
name: Wang, Yijin
name: Lemaitre, Joseph C.
name: Xu, Pan
name: Gibson, Graham Casey
name: Li, Chaozhuo
name: Mühlemann, Anja
name: Cao, Wei
name: Kanji, Abdul H.
name: Walker, Jo W.
name: Yamana, Teresa K.
name: Davis, Jessica T.
name: Fairchild, Geoffrey
name: Ayer, Turgay
name: Pagano, Robert
name: Mueller, Peter P.
name: Wang, Lily
name: Adiga, Aniruddha
name: Zhang, Shun
name: Jayawardena, Dasuni
name: Gao, Lei
name: Liu, Tie-Yan
name: Ferres, Juan Lavista
name: Shin, Lauren
name: Drake, John M.
name: Bian, Jiang
name: Slayton, Rachel B.
name: Gu, Zhiling
name: Marshall, Maximilian
name: Rivadeneira, Alvaro J. Castro
name: O’Dea, Eamon B.
name: Zou, Difan
name: Kinsey, Matt
name: Marathe, Madhav
name: Wang, Lingxiao
name: Ray, Evan L.
name: Niemi, Jarad
name: Tarasewicz, Samuel R.
name: Vespignani, Alessandro
name: Jindal, Sonia
name: Smith, Claire P.
Publisher: name: Public Library of Science (PLoS)
Year: 2024

description: During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.