Back2Future: Leveraging Backfill Dynamics for Improving Real-time Predictions in Future

1 INTRODUCTION . The current COVID-19 pandemic has challenged our response capabilities to large disruptive events , affecting the health and economy of millions of people . A major tool in our response has been forecasting epidemic trajectories , which has provided lead time to policymakers to optimize and plan interventions ( Holmdahl & Buckee , 2020 ) . Broadly two classes of approaches have been devised : traditional mechanistic epidemiological models ( Shaman & Karspeck , 2012 ; Zhang et al. , 2017 ) , and the fairly newer statistical approaches ( Brooks et al. , 2018 ; Adhikari et al. , 2019 ; Osthus et al. , 2019b ) including deep learning models ( Adhikari et al. , 2019 ; Deng et al. , 2020 ; Panagopoulos et al. , 2021 ; Rodríguez et al. , 2021a ) , which have become among the top-performing ones for multiple forecasting tasks ( Reich et al. , 2019 ) . These also leverage newer digital indicators like search queries ( Ginsberg et al. , 2009 ; Yang et al. , 2015 ) and social media ( Culotta , 2010 ; Lampos et al. , 2010 ) . As noted in multiple previous works ( Metcalf & Lessler , 2017 ; Biggerstaff et al. , 2018 ) , epidemic forecasting is a challenging enterprise because it is affected by weather , mobility , strains , and others . However , real-time forecasting also brings new challenges . As noted in multiple CDC real-time forecasting initiatives for diseases like flu ( Osthus et al. , 2019a ) and COVID-19 ( Cramer et al. , 2021 ) , as well as in macroeconomics ( Clements & Galvão , 2019 ; Aguiar , 2015 ) the initially released public health data is revised many times after and is known as the ’ backfill ’ phenomenon.The various factors that affect backfill are multiple and complex , ranging from surveillance resources to human factors like coordination between health institutes and government organizations within and across regions ( Chakraborty et al. , 2018 ; Reich et al. , 2019 ; Altieri et al. , 2021 ; Stierholz , 2017 ) . While previous works have addressed anomalies ( Liu et al. , 2017 ) , missing data ( Yin et al. , 2020 ) , and data delays ( Žliobaite , 2010 ) in general time-series problems , the backfill problem has not been addressed . In contrast , the topic of revisions has not received as much attention , with few exceptions . For example in epidemic forecasting , a few papers have either ( a ) mentioned about the ‘ backfill problem ’ and its effects on performance ( Chakraborty et al. , 2018 ; Rodríguez et al. , 2021b ; Altieri et al. , 2021 ; Rangarajan et al. , 2019 ) and evaluation ( Reich et al. , 2019 ) ; or ( b ) proposed to address the problem via simple models like linear regression ( Chakraborty et al. , 2014 ) or ’ backcasting ’ ( Brooks et al. , 2018 ) the observed targets c ) used data assimilation and sensor fusion from a readily available stable set of features to refine unrevised features ( Farrow , 2016 ; Osthus et al. , 2019a ) . However , they focus only on revisions in the target and typically study in the context of influenza forecasting , which is substantially less noisy and more regular than the novel COVID-19 pandemic or assume access to stable values for some features which is not the case for COVID-19 . In economics , Clements & Galvão ( 2019 ) surveys several domain-specific ( Carriero et al. , 2015 ) or essentially linear techniques for data revision/correction behavior of several macroeconomic indicators ( Croushore , 2011 ) . Motivated from above , we study the more challenging problem of multi-variate backfill for both features and targets . We go further beyond prior work and also show how to leverage our insights for a more general neural framework to improve both predictions ( i.e . refinement of the model ’ s predictions ) and performance evaluation ( i.e . rectification from the evaluator ’ s perspective ) . Our specific contributions are the following : •Multi-variate backfill problem : We introduce the multi-variate backfill problem using real-time epidemiological forecasting as the primary motivating example . In this challenging setting , which generalizes ( the limited ) prior work , the forecast targets , as well as exogenous features , are subject to retrospective revision . Using a carefully collected diverse dataset for COVID-19 forecasting for the past year , we discover several patterns in backfill dynamics , show that there is a significant difference in real-time and revised feature measurements , and highlight the negative effects of using unrevised features for incidence forecasting in different models both for model performance and evaluation . Building on our empirical observations , we formulate the problem BFRP , which aims to ‘ correct ’ given model predictions to achieve better performance on eventual fully revised data . • Spatial and Feature level backfill modeling to refine model predictions : Motivated by the patterns in revision and observations from our empirical study , we propose a deep-learning model Back2Future ( B2F ) to model backfill revision patterns and derive latent encodings for features . B2F combines Graph Convolutional Networks that capture sparse , cross-feature , and cross-regional backfill dynamics similarity and deep sequential models that capture temporal dynamics of each features ’ backfill dynamics across time . The latent representation of all features is used along with the history of the model ’ s predictions to improve diverse classes of models trained on real-time targets , to predict targets closer to revised ground truth values . Our technique can be used as a ‘ wrapper ’ to improve model performance of any forecasting model ( mechanistic/statistical ) . • Refined top models ’ predictions and improved model evaluation : We perform an extensive empirical evaluation to show that incorporating backfill dynamics through B2F consistently improves the performance of diverse classes of top-performing COVID-19 forecasting models ( from the CDC COVID-19 Forecast Hub , including the top-performing official ensemble ) significantly . We also utilize B2F to help forecast evaluators and policy-makers better evaluate the ‘ eventual ’ true accuracy of participating models ( against revised ground truth , which may not be available until weeks later ) . This allows the model evaluators to quickly estimate models that perform better w.r.t revised stable targets instead of potentially misleading current targets . Our methodology can be adapted for other time-series forecasting problems in general . We also show the generalizability of our framework and model B2F to other domains by significantly improving predictions of non-trivial baselines for US National GDP forecasting ( Marcellino , 2008 ; Tkacz & Hu , 1999 ) . 2 NATURE OF BACKFILL DYNAMICS . In this section , we study important properties of the revision dynamics of our signals . We introduce some concepts and definitions to aid in the understanding of our empirical observations and method . Real-time forecasting . We are given a set of signals F = Reg × Feat , where Reg is the set of all regions ( where we want to forecast ) and set Feat contains our features and forecasting target ( s ) for each region . At prediction week t , x ( t ) i,1 : t is a time series from 1 to t for feature i , and the set of all signals results in the multi-variate time series X ( t ) 1 : t 1 . Similarly , Y ( t ) 1 : t is the forecasting target ( s ) time series . Further , let ’ s call all data available at time t , D ( t ) 1 : t = { X ( t ) 1 : t , Y ( t ) 1 : t } as real-time sequence . For clarity we refer to ‘ signal ’ i ∈ F as a sequence of either a feature or a target , and denote it as d ( t ) i,1 : t. 1In practice , delays are possible too , i.e , at week t , we have data for some feature i only until t− δi . All our results incorporate these situations . We defer the minor needed notational extensions to Appendix for clarity . Thus , at prediction week t , the real-time forecasting problem is : Given D ( t ) 1 : t , predict next k values of forecasting target ( s ) , i.e . ŷt+1 : t+k . Typically for CDC settings and this paper , our time unit is week , k = 4 ( up to 4 weeks ahead ) and our target is COVID-19 mortality incidence ( Deaths ) . Revisions . Data revisions ( ‘ backfill ’ ) are common . At prediction week t+ 1 , the real-time sequence D ( t+1 ) 1 : t+1 is available . In addition to the length of the sequences increasing by one ( new data point ) , values of D ( t+1 ) 1 : t+1 already in D ( t ) 1 : t may be revised i.e. , D ( t ) 1 : t 6= D ( t+1 ) 1 : t . Note that previous work has studied backfill limited to Y ( t ) , while we address it in both X ( t ) and Y ( t ) . Also , note that the data in the backfill is the same used for real-time forecasting , but just seen from a different perspective . Backfill sequences : Another useful way we propose to look at backfill is by focusing on revisions of a single value . Let ’ s focus on value of signal i at an observation week t′ . For this observation week , the value of the signal can be revised at any t > t′ , which induces a sequence of revisions . We refer to revision week r ≥ 0 as the relative amount of time that has passed since the observation week t′ . Defn . 1 . ( Backfill Sequence BSEQ ) For signal i and observation week t′ , its backfill sequence is BSEQ ( i , t′ ) = 〈d ( t ′ ) i , t′ , d ( t′+1 ) i , t′ , . . . , d ( ∞ ) i , t′ 〉 , where d ( t′ ) i , t′ is the initial value of the signal and d ( ∞ ) i , t′ is the final/stable value of the signal . Defn . 2 . ( Backfill Error BERR ) For revision week r of a backfill sequence , the backfill error is BERR ( r , i , t′ ) = |d ( t ′+r ) i , t′ − d ( ∞ ) i , t′ | / |d ( ∞ ) i , t′ | . Defn . 3 . ( Stability time STIME ) of a backfill sequence BSEQ is the revision week r∗ that is the minimum r for which the backfill error BERR < for all r > r∗ , i.e. , the time when BSEQ stabilizes . Note : We ensured that BSEQ length is at least 7 , and found that in our dataset most signals stabilize before r = 20 . For d ( ∞ ) i , t′ , we use d ( tf ) i , t′ , at the final week tf in our revisions dataset . In case we do not find BERR < in any BSEQ , we set STIME to the length of that BSEQ . We use = 0.05 . Example : For BSEQ { 223 , 236 , 236 , 404 , . . . , 404 } , BERR for third week is |236−404|404 = 0.41 and STIME is 4 . 2.1 DATASET DESCRIPTION Table 1 : List of features in our CoVDS Type Features Patient Line-List ERVisits , HospRate , +veInc , HospInc , Recovered , onVentilator , inICU Testing TestResultsInc , -veInc , Facilities Mobility RetailRec , Grocery , Parks , Transit , WorkSpace , Resident , AppleMob Exposure DexA Social Survey FbCLI , FbWiLi We collected important publicly available signals from a variety of trusted sources that are relevant to COVID-19 forecasting to form the COVID-19 Surveillance Dataset ( CoVDS ) . See Table 1 for the list of 20 features ( |Feat| = 21 , including Deaths ) . Our revisions dataset contains signals that we collected every week since April 2020 and ends on July 2021 . Our analysis covers 30 observation weeks from June 2020 to December 2020 ( to ensure all our backfill sequences are of length at least 7 ) for all |Reg| = 50 US states . The rest of the unseen data from Jan 2021 to July 2021 is used strictly for evaluation . Patient line-list : traditional surveillance signals used in epidemiological models ( Chakraborty et al. , 2014 ; Brooks et al. , 2018 ) derived from line-list records e.g . hospitalizations from CDC ( CDC , 2020 ) , positive cases , ICU admissions from COVID Tracking ( COVID-Tracking , 2020 ) . Testing : measure changes in testing from CDC and COVIDTracking e.g . tested population , negative tests , used by Rodríguez et al . ( 2021b ) . Mobility : track people movement to several point of interests ( POIs ) , from LLC ( 2020 ) and Inc. ( 2020 ) , and serve as digital proxy for social distancing ( Arik et al. , 2020 ) . Exposure : digital signal measuring closeness between people at POIs , collected from mobile phones ( Chevalier et al. , 2021 ) Social Survey : previously used by ( Wang et al. , 2020 ; Rodríguez et al. , 2021b ) CMU/Facebook Symptom Survey Data , which contains self-reported responses about COVID-19 symptoms .

Summary: The authors consider the effects of backfill dynamics--the correction of historical data--on time series prediction. They use COVID-19 forecasting as the motivating application. Both the features (ER visits, hospitalization rate, etc.) and the target (deaths) are subject to revision, and the authors collect a data set showing that the backfill error due to revisions can be extremely large. The authors make several other interesting observations regarding backfill error and sequences in their data. A key finding is that real-time forecasts tend to overestimate their accuracy when compared to the stable accuracy after errors are corrected. The authors then propose a Back2Future (B2F) pipeline that can be used to refine predictions from a model when backfill dynamics are present. They demonstrate consistent improvement on stable accuracy for multiple COVID-19 forecasts.