Back2Future: Leveraging Backfill Dynamics for Improving Real-time Predictions in Future

1 INTRODUCTION . The current COVID-19 pandemic has challenged our response capabilities to large disruptive events , affecting the health and economy of millions of people . A major tool in our response has been forecasting epidemic trajectories , which has provided lead time to policymakers to optimize and plan interventions ( Holmdahl & Buckee , 2020 ) . Broadly two classes of approaches have been devised : traditional mechanistic epidemiological models ( Shaman & Karspeck , 2012 ; Zhang et al. , 2017 ) , and the fairly newer statistical approaches ( Brooks et al. , 2018 ; Adhikari et al. , 2019 ; Osthus et al. , 2019b ) including deep learning models ( Adhikari et al. , 2019 ; Deng et al. , 2020 ; Panagopoulos et al. , 2021 ; Rodríguez et al. , 2021a ) , which have become among the top-performing ones for multiple forecasting tasks ( Reich et al. , 2019 ) . These also leverage newer digital indicators like search queries ( Ginsberg et al. , 2009 ; Yang et al. , 2015 ) and social media ( Culotta , 2010 ; Lampos et al. , 2010 ) . As noted in multiple previous works ( Metcalf & Lessler , 2017 ; Biggerstaff et al. , 2018 ) , epidemic forecasting is a challenging enterprise because it is affected by weather , mobility , strains , and others . However , real-time forecasting also brings new challenges . As noted in multiple CDC real-time forecasting initiatives for diseases like flu ( Osthus et al. , 2019a ) and COVID-19 ( Cramer et al. , 2021 ) , as well as in macroeconomics ( Clements & Galvão , 2019 ; Aguiar , 2015 ) the initially released public health data is revised many times after and is known as the ’ backfill ’ phenomenon.The various factors that affect backfill are multiple and complex , ranging from surveillance resources to human factors like coordination between health institutes and government organizations within and across regions ( Chakraborty et al. , 2018 ; Reich et al. , 2019 ; Altieri et al. , 2021 ; Stierholz , 2017 ) . While previous works have addressed anomalies ( Liu et al. , 2017 ) , missing data ( Yin et al. , 2020 ) , and data delays ( Žliobaite , 2010 ) in general time-series problems , the backfill problem has not been addressed . In contrast , the topic of revisions has not received as much attention , with few exceptions . For example in epidemic forecasting , a few papers have either ( a ) mentioned about the ‘ backfill problem ’ and its effects on performance ( Chakraborty et al. , 2018 ; Rodríguez et al. , 2021b ; Altieri et al. , 2021 ; Rangarajan et al. , 2019 ) and evaluation ( Reich et al. , 2019 ) ; or ( b ) proposed to address the problem via simple models like linear regression ( Chakraborty et al. , 2014 ) or ’ backcasting ’ ( Brooks et al. , 2018 ) the observed targets c ) used data assimilation and sensor fusion from a readily available stable set of features to refine unrevised features ( Farrow , 2016 ; Osthus et al. , 2019a ) . However , they focus only on revisions in the target and typically study in the context of influenza forecasting , which is substantially less noisy and more regular than the novel COVID-19 pandemic or assume access to stable values for some features which is not the case for COVID-19 . In economics , Clements & Galvão ( 2019 ) surveys several domain-specific ( Carriero et al. , 2015 ) or essentially linear techniques for data revision/correction behavior of several macroeconomic indicators ( Croushore , 2011 ) . Motivated from above , we study the more challenging problem of multi-variate backfill for both features and targets . We go further beyond prior work and also show how to leverage our insights for a more general neural framework to improve both predictions ( i.e . refinement of the model ’ s predictions ) and performance evaluation ( i.e . rectification from the evaluator ’ s perspective ) . Our specific contributions are the following : •Multi-variate backfill problem : We introduce the multi-variate backfill problem using real-time epidemiological forecasting as the primary motivating example . In this challenging setting , which generalizes ( the limited ) prior work , the forecast targets , as well as exogenous features , are subject to retrospective revision . Using a carefully collected diverse dataset for COVID-19 forecasting for the past year , we discover several patterns in backfill dynamics , show that there is a significant difference in real-time and revised feature measurements , and highlight the negative effects of using unrevised features for incidence forecasting in different models both for model performance and evaluation . Building on our empirical observations , we formulate the problem BFRP , which aims to ‘ correct ’ given model predictions to achieve better performance on eventual fully revised data . • Spatial and Feature level backfill modeling to refine model predictions : Motivated by the patterns in revision and observations from our empirical study , we propose a deep-learning model Back2Future ( B2F ) to model backfill revision patterns and derive latent encodings for features . B2F combines Graph Convolutional Networks that capture sparse , cross-feature , and cross-regional backfill dynamics similarity and deep sequential models that capture temporal dynamics of each features ’ backfill dynamics across time . The latent representation of all features is used along with the history of the model ’ s predictions to improve diverse classes of models trained on real-time targets , to predict targets closer to revised ground truth values . Our technique can be used as a ‘ wrapper ’ to improve model performance of any forecasting model ( mechanistic/statistical ) . • Refined top models ’ predictions and improved model evaluation : We perform an extensive empirical evaluation to show that incorporating backfill dynamics through B2F consistently improves the performance of diverse classes of top-performing COVID-19 forecasting models ( from the CDC COVID-19 Forecast Hub , including the top-performing official ensemble ) significantly . We also utilize B2F to help forecast evaluators and policy-makers better evaluate the ‘ eventual ’ true accuracy of participating models ( against revised ground truth , which may not be available until weeks later ) . This allows the model evaluators to quickly estimate models that perform better w.r.t revised stable targets instead of potentially misleading current targets . Our methodology can be adapted for other time-series forecasting problems in general . We also show the generalizability of our framework and model B2F to other domains by significantly improving predictions of non-trivial baselines for US National GDP forecasting ( Marcellino , 2008 ; Tkacz & Hu , 1999 ) . 2 NATURE OF BACKFILL DYNAMICS . In this section , we study important properties of the revision dynamics of our signals . We introduce some concepts and definitions to aid in the understanding of our empirical observations and method . Real-time forecasting . We are given a set of signals F = Reg × Feat , where Reg is the set of all regions ( where we want to forecast ) and set Feat contains our features and forecasting target ( s ) for each region . At prediction week t , x ( t ) i,1 : t is a time series from 1 to t for feature i , and the set of all signals results in the multi-variate time series X ( t ) 1 : t 1 . Similarly , Y ( t ) 1 : t is the forecasting target ( s ) time series . Further , let ’ s call all data available at time t , D ( t ) 1 : t = { X ( t ) 1 : t , Y ( t ) 1 : t } as real-time sequence . For clarity we refer to ‘ signal ’ i ∈ F as a sequence of either a feature or a target , and denote it as d ( t ) i,1 : t. 1In practice , delays are possible too , i.e , at week t , we have data for some feature i only until t− δi . All our results incorporate these situations . We defer the minor needed notational extensions to Appendix for clarity . Thus , at prediction week t , the real-time forecasting problem is : Given D ( t ) 1 : t , predict next k values of forecasting target ( s ) , i.e . ŷt+1 : t+k . Typically for CDC settings and this paper , our time unit is week , k = 4 ( up to 4 weeks ahead ) and our target is COVID-19 mortality incidence ( Deaths ) . Revisions . Data revisions ( ‘ backfill ’ ) are common . At prediction week t+ 1 , the real-time sequence D ( t+1 ) 1 : t+1 is available . In addition to the length of the sequences increasing by one ( new data point ) , values of D ( t+1 ) 1 : t+1 already in D ( t ) 1 : t may be revised i.e. , D ( t ) 1 : t 6= D ( t+1 ) 1 : t . Note that previous work has studied backfill limited to Y ( t ) , while we address it in both X ( t ) and Y ( t ) . Also , note that the data in the backfill is the same used for real-time forecasting , but just seen from a different perspective . Backfill sequences : Another useful way we propose to look at backfill is by focusing on revisions of a single value . Let ’ s focus on value of signal i at an observation week t′ . For this observation week , the value of the signal can be revised at any t > t′ , which induces a sequence of revisions . We refer to revision week r ≥ 0 as the relative amount of time that has passed since the observation week t′ . Defn . 1 . ( Backfill Sequence BSEQ ) For signal i and observation week t′ , its backfill sequence is BSEQ ( i , t′ ) = 〈d ( t ′ ) i , t′ , d ( t′+1 ) i , t′ , . . . , d ( ∞ ) i , t′ 〉 , where d ( t′ ) i , t′ is the initial value of the signal and d ( ∞ ) i , t′ is the final/stable value of the signal . Defn . 2 . ( Backfill Error BERR ) For revision week r of a backfill sequence , the backfill error is BERR ( r , i , t′ ) = |d ( t ′+r ) i , t′ − d ( ∞ ) i , t′ | / |d ( ∞ ) i , t′ | . Defn . 3 . ( Stability time STIME ) of a backfill sequence BSEQ is the revision week r∗ that is the minimum r for which the backfill error BERR < for all r > r∗ , i.e. , the time when BSEQ stabilizes . Note : We ensured that BSEQ length is at least 7 , and found that in our dataset most signals stabilize before r = 20 . For d ( ∞ ) i , t′ , we use d ( tf ) i , t′ , at the final week tf in our revisions dataset . In case we do not find BERR < in any BSEQ , we set STIME to the length of that BSEQ . We use = 0.05 . Example : For BSEQ { 223 , 236 , 236 , 404 , . . . , 404 } , BERR for third week is |236−404|404 = 0.41 and STIME is 4 . 2.1 DATASET DESCRIPTION Table 1 : List of features in our CoVDS Type Features Patient Line-List ERVisits , HospRate , +veInc , HospInc , Recovered , onVentilator , inICU Testing TestResultsInc , -veInc , Facilities Mobility RetailRec , Grocery , Parks , Transit , WorkSpace , Resident , AppleMob Exposure DexA Social Survey FbCLI , FbWiLi We collected important publicly available signals from a variety of trusted sources that are relevant to COVID-19 forecasting to form the COVID-19 Surveillance Dataset ( CoVDS ) . See Table 1 for the list of 20 features ( |Feat| = 21 , including Deaths ) . Our revisions dataset contains signals that we collected every week since April 2020 and ends on July 2021 . Our analysis covers 30 observation weeks from June 2020 to December 2020 ( to ensure all our backfill sequences are of length at least 7 ) for all |Reg| = 50 US states . The rest of the unseen data from Jan 2021 to July 2021 is used strictly for evaluation . Patient line-list : traditional surveillance signals used in epidemiological models ( Chakraborty et al. , 2014 ; Brooks et al. , 2018 ) derived from line-list records e.g . hospitalizations from CDC ( CDC , 2020 ) , positive cases , ICU admissions from COVID Tracking ( COVID-Tracking , 2020 ) . Testing : measure changes in testing from CDC and COVIDTracking e.g . tested population , negative tests , used by Rodríguez et al . ( 2021b ) . Mobility : track people movement to several point of interests ( POIs ) , from LLC ( 2020 ) and Inc. ( 2020 ) , and serve as digital proxy for social distancing ( Arik et al. , 2020 ) . Exposure : digital signal measuring closeness between people at POIs , collected from mobile phones ( Chevalier et al. , 2021 ) Social Survey : previously used by ( Wang et al. , 2020 ; Rodríguez et al. , 2021b ) CMU/Facebook Symptom Survey Data , which contains self-reported responses about COVID-19 symptoms .

Summary: The authors deal with the problem of revising previous recorded data and its effect on timeseries predictions. They showcase how revisions in past data, quantified as the backfill error, can introduce a considerable error in predictions. Towards that, they propose a novel deep learning approach, the Back2Future, that refines the model predictions using backfill dynamics. They demonstrate its efficiency on a real COVID dataset.