Back2Future: Leveraging Backfill Dynamics for Improving Real-time Predictions in Future
Authors: Harshavardhan Kamarthi
Alexander Rodríguez
B. Aditya Prakash
Venue: ICLR 2022 Poster (2022)
Keywords: Epidemic Forecasting
Data revisions
Graph Representation learning
Time Series Forecasting

Abstract: For real-time forecasting in domains like public health and macroeconomics, data collection is a non-trivial and demanding task. Often after being initially released, it undergoes several revisions later (maybe due to human or technical constraints) - as a result, it may take weeks until the data reaches a stable value. This so-called ‘backfill’ phenomenon and its effect on model performance have been barely addressed in the prior literature. In this paper, we introduce the multi-variate backfill problem using COVID-19 as the motivating example. 
We construct a detailed dataset composed of relevant signals over the past year of the pandemic. 
We then systematically characterize several patterns in backfill dynamics and leverage our observations for formulating a novel problem and neural framework, Back2Future, that aims to refines a given model's predictions in real-time. Our extensive experiments demonstrate that our method refines the performance of the diverse set of top models for COVID-19 forecasting and GDP growth forecasting. Specifically, we show that Back2Future refined top COVID-19 models by 6.65% to 11.24% and yield an 18% improvement over non-trivial baselines. In addition, we show that our model improves model evaluation too; hence policy-makers can better understand the true accuracy of forecasting models in real-time.

Published as a conference paper at ICLR 2022
BACK2FUTURE: LEVERAGING BACKFILL DYNAMICS FOR IMPROVING REAL-TIME PREDICTIONS IN FUTURE
Harshavardhan Kamarthi, Alexander Rodriguez, B. Aditya Prakash College of Computing
Georgia Institute of Technology
(harsha .k, a arodr guezc, badi @ggatcch.dei
ABSTRACT
For real-time forecasting in domains like public health and macroeconomics, data collection a non-trivial and demanding task. Often after being initially released, it undergoes several revisions later (mayhe due to human technical constraints) as result, may take weeks until the data reaches a stable value. This so- called 'backfill" phenomenon and its effect on model performance have been barely addressed in the prior literature. In this paper, we introduce the multi-variate backfill problem using COVID-19 as the motivating example. We construct a detailed dataset composed of relevant signals over the past year of the pandemic. We then systematically characterize several patterns in backfill dynamics and leverage our observations for formulating a novel problem and neural framework, Back2Future, that aims to refines given model's predictions in real-time. Our extensive experiments demonstrate that our method refines the performance of diverse set of top models for COVID-19 forecasting and GDP growth forecasting. Specifically, we show that Back2Future refined top COVID-19 models by 6.65% 11. 11.24% and yield 18% improvement over non-trivial baselines. In addition, we show that our model improves model evaluation too; hence policy-makers can better understand the true accuracy of forecasting models real-time.
INTRODUCTION
The current COVID-19 pandemic has challenged our response capabilities to large disruptive events, affecting the health and economy millions of people. A major tool our response has been forecasting epidemic trajectories which enabled policymakers
plan interventions (Holmdahl & Buckee 2020). Broadly two classes of approaches have been devised: traditional mechanistic epidemiological models (Shaman & Karspeck [0012] Zhang et al.) 2017). and the fairly newer statistical approaches (Brooks et al. [2018] Adhikari et al. 22019 Osthus et al. (201bb including deep learning models (Adhikari et al.) 22099] Panagopoulos
al. [02 Rodriguez et al. (021a), which have become among the top-pen pop-pertorming ones for multpple forecasting tasks (Rich et al. 20119) These models also use newer digital indicators like search queries (Ginsberg et al. 2009)] Yang al. (0155)
and social media (Culottal 2010). Epidemic forecasting is still challenging enterprise (Metcalf & Lessler 2017 Biggerstaff al. 2018) because it is affected by weather, mobility, strains, and others.
However, real-time forecasting also brings new challenges. As noted in multiple CDC real-time forecasting initiatives for diseases like flu (Osthus et al. 2019a) and COVID-19 (Cramer et al., [021], } as well in macroeconomies (Clements & Galvão 2019 Aguiar 2015) the initially released public various factors that affect backfill multiple and complex, ranging from surveillance resources to human factors like coordination between health institutes and government organizations within and across regions (Chakraborty al. 2018 Reich et al. 2019 Altieri et al. 2021
health data is revised many times after and is known as the backfilll phenomenon.
Stierholz 2017).
While previous works have addressed anomalies Liu et al. 2017). missing data (Yin et al. 2020). and data delays (Zliobaite 2010) in general time-series problems, the backfill problem has not been addressed. In contrast, the topic of revisions has received as much attention, with few exceptions. For example in epidemic forecasting, a few papers have either (a) mentioned about the 'backfill
problem' and its effects performance (Chakraborty al. 2018 Rodríguez al. 2021b Altieri

---

Published as a conference paper at ICLR 2022
et al. 2021 Rangarajan et al. 2019) and evaluation (Reich et al. 2019): (b) proposed to address the problem em via simple models like linear regression (Chakraborty et 20144 or "backcasting' (Brooks et al. 2018) the observed targets. The related problem of nowcasting involves prediction of revised stable value of current week's 's target from sequence of right-truncated past values [Aditya: cite one paper J. Prior works have used data assimilation and sensor fusion from readily available stable set of features to refine unrevised features for accurate nowcasting (Farrow) 2016 Osthus et al. 2019a). However, most methods focus only on revisions in the target and typically study in the context of influenza forecasting, which is substantially less noisy and more regular than the novel COVID-19 pandemic or assume access to stable values for some features which not the case for COVID-19. In economics, [Clmmets
[Clnmmnss,,llmenttss. & Galvão (2019) surveys several domain-speciinn (Carriero et al., 2015) or essentially linear techniques for data revision/correction behavior of several macroeconomic indicators (Croushore 2011
Motivated from above, we study the challenging problem of multi- backfill for both features and targets. We go further beyond prior work and also show how to leverage our insights towards general neural framework to improve model predictions and performance evaluation (i.e. rectification of current target from the evaluator's perspective). Our specific contributions are the following:
We introduce the multi-variate backfill problem using real-time epidemiological forecasting as the primary motivating example. In this challenging setting, which generalizes (the limited) prior work, the forecast targets, as well as exogenous features, are subject to retrospective revision. Using carefully collected diverse dataset for COVID-19 forecasting for the past year, we discover several patterns in backfill dynamics, show that there is a significant difference in real-time and revised feature measurements, and highlight the negative effects of using unrevised features for incidence forecasting in different models both for model performance and evaluation. Building on our empirical observations, we formulate the problem BFRP, which aims 'correct' given model predictions to achieve better performance on eventual fully revised data.
varti-v backfill problem:
Spatial and Feature level backfill modeling refine model predictions: Motivated by the patterns in revision and observations from our empirical study, we propose a deep-learning model Back2Future (B2F) to model backfill revision patterns and derive latent encodings for features. B2F combines Graph Convolutional Networks that capture sparse, cross-feature, and cross-regional backfill dynamics similarity and deep sequential models that capture temporal dynamics of each features' backfill dynamics across time. The latent representation of all features is used along with the history of the model's 's predictions to improve diverse classes of models trained on real-time targets, to predict targets closer to revised ground truth values. Our technique can be used as 'wrapper' to improve model performance of any forecasting model (mechhanistic/statistccalll
Refined top models' predictions and improved model evaluation: We perform extensive empirical evaluation to show that incorporating backfill dynamics through B2F consistently improves the performance of diverse classes of top-performing COVID-19 forecasting models (from the CDC COVID-19 Forecast Hub, including the top-performing official ensemble) significantly. B2F also enables forecast evaluators and
better evaluate the 'eventual' true accuracy of participating models (against revised ground truth). This allows the model evaluators to quickly estimate models that perform better w.r.t revised stable targets instead of potentially misleading current targets. Our methodology can also be further adapted for nowcasting and other general time-series forecasting problems. We also show the generalizability of our framework and model B2F to other domains by significantly improving predictions of non-trivial baselines for US National GDP forecasting (Marcellino) 2008).
2 NATURE OF BACKFILL DYNAMICS
In this section, we study important properties of the revision dynamics of our signals. We introduce some concepts and definitions to aid in the understanding of our empirical observations and method.
Real-time forecasting. We are given a set of signals F = Reg x Feat, where Reg is the set of all regions (where want to forecast) and set Feat contains features and forecasting target(s) for (t) each region. At prediction week i, 22, is a time series from I to t for feature i, and the set of all signals results in the multi-variate time series x Similarly, (2(() is the forecasting target(s) time
In practice, delays are possible too, i.e, at week f, we have data for some feature only until t 8, All our results incorporate these situations. We defer the minor needed notational extensions Appendix for clarity.
2

---

Published as a conference paper at ICLR 2022
211.) real-time sequence. For series. Further, let's call all data available at time 6, D/. { clarity refer "signal' i E F as a sequence of either feature or a target, and denote it di,1:t- Thus, prediction week I, the real-time forecasting problem is: Given Di- predict next l values of forecasting larget(s), i.e. Ie++- Typically for CDC settings and this paper, our time unit is week, 4 (up to 4 weeks ahead) and our target COVID-19 mortality incidence (Deaths).
Revisions. Data revisions ('backfill"" are common. At prediction week t + 1, the real-time sequence D(++1] is available. In addition the length of the sequences increasing by one (new data point), values of D Di++1 already in Di. may be revised i.e., D + D Note that previous work has studied backfill limited to y(()), while we address it in both x²(+) and y/tt) Also, note that the data in the backfill is the same used for real-time forecasting, but just seen from a different perspective.
Backfill sequences: Another useful way we propose look at backfill is by focusing on revisions of a single value. Let's focus on value of signal at an observation week t'. For this observation week, the value of the signal can be revised at any t t', which induces sequence of revisions. We refer to revision week r 2 0 as the relative amount of time that has passed since the observation week t'.
Defn. 1. (Backfill Sequence BSEQ) For signal i and observation week t', its backfill sequence is BSEQ(i, (*) = (d d((+") ....,, (.)))) where di()) is the initial value of the signal and dl is the final/stable value of the signal.
Defn. 2. (Backfill Error BERR) For revision week r of a backfill sequence, the backfill error is =d d |
BERR i, i,t')
Defn. 3. (Stability time STIME) of a backfill sequence BSEQ is the revision week 1 that is the minimum for which the backfill error BERR eforr all r > r", i.e. the time when BSEQ stabilizes.
Note: We ensured that BSEQ length is at least 7, and found that in our dataset most signals stabilize before r = 20. For de we use d((()) at the final week ty in our revisions dataset. In case we do not find BERR < e in any BSEQ, we set STIME to the length that BSEQ We use = 0.05. Example: 226-404| = 0.41 and STIME is 4.
For BSEQ {223, 236, 236, 404,... 4043, BERR for third week is
404
2.1 DATASET DESCRIPTION
Table 1: List of features in our CoVDS
We collected and pre- preprocessed important publicly available signals from a variety of trusted sources that are relevant to COVID-19 forecasting to form the COVID-19 Surveillance Datasel (CoVDS).
Type Patient Line-List
Features
See Table 1 for the list of 20 features (Feat] = 21, including Deaths). We collected revised features every week from April 2020 to July 2021. Our analysis covers 30 observation weeks from June 2020 to December 2020 (to ensure all our backfill sequences are of length at least 7) for all |Reg| = 50 US states. The rest of the wnseen data from Jan 2021 to July 2021 is used strictly for evaluation.
ERVisits.
HospRate,
vveIne, IlospInc Recovered, enVentilaton InICU
Testing
TestResultsine,
-veInc. Facilities
Patient line- traditional surveillance signals used in epidemiologi- cal models (Chkrraborty al. 2014 Brooks al., 2018) derived from line-list records e.g. dospitalization from CDC (CDC) 20200, posi- tive cases, ICU admissions from COVID Tracking (COvID-Tracking 2020). Testing: measure changes in testing from CDC and COVID- Tracking used by RRodriuuez et al. 20211). Mobility: quantify change in people's movement to several point of interests (POIs); derived from mobility reports released by Google (2020); Apple (220)). Exposure: digital signal measuring closeness between people at POIs.
Mobility
RetailRec, Grocery,
Parka. Transit. WorkSpace,
Resident,
AppleMob
Expusure
DexA
Social Sur-
DCCLI,PbRIL.
vey
al., 2021) Social Survey: used by (Wang et al. 2020 [Rodriguez et al., 2021b) CMU//acehook
Symptom Survey Data contains self-reported responses about COVID-19 symptoms.
2.2 OBSERVATIONS
We first study different facets of the significance of backfill in CoVDS. Using our definitions, generate a backfill sequence for every combination of signal, observation week, and region (not all signals are available for all regions). In total, we generate more than 30, 000 backfill sequences.
3

---

Published as a conference paper at ICLR 2022
Backfill error BERR is significant. We computed BERR for the initial values, i... BERR(T (,,,t),, for all signals i and observation weeks t'.
Obs. 1. (BERR across signals and regions) Compute the average BERR for each signal; the median of all these averages is 32%, i.e. at least half of all signals are corrected by 32% of their initial value. Similarty in least half of the regions the signal corrections are 280% of their initial value.
We also found large variation of BERR. For features (Figure [1) compare avg. BERR 1743% of five most corrected features with 1.6% of the five least corrected features. Also, in contrast to related work that focuses on traditional surveillance data Yang et al.) (2015), perhaps unexpectedly, that digital indicators also have a significant BERR (average of 108%). For regions (see Figure compare 1594% of the five most corrected regions with 38% of the five least corrected regions.
found
Backdhi Enea
type
Festure
(a) BERR per feat. type
(b) BERR per region
(c) STIME per feat. type (d) STIME per region
Figure I: BERR and STIME across feature type and regions, heat maps are log scaled.
Stability time STIME is significant. similar analysis for STIME found significant variation across signals (from I weeks for to 21 weeks for COVID-19, 19, see Figure D for STIME across feature types) and regions (from 1.55 weeks for GA to 3.83 weeks for TX, see Figure Tt). This also impacts our target, thus, actual accuracy not readily available which undermines real-time evaluation and decision making.
Obs. 2. (STIME features and target) Compute the average STIME for each signal; the average of all these averages for features is around 4 weeks and for our target Deaths is around 3 weeks, i.e. on average, it takes over 3 weeks to reach the stable values of features. Backfill sequence BSEQ patterns. There is significant similarity among BSEQs. We cluster BSEQS via K-means using Dynamic Time Warping (DTW) pair-wise distance (as DTW can handle sequences of varying magnitude and length). We found five canonical categories of behaviors (see each of size roughly 11.58% of all BSEQs. Also, each cluster is not defined only by signal
Figure 2
Early Decline
Early Increase
Steady/Spike
Late rise
Mid Decrease
(0:1 scaled)
BSeg BSeqvalle value
Revision week
Figure 2: Centroid BSEQ of each cluster, scaled between [0, 1]. showing canonical backfill behaviors
nor region. Hence there is non-trivial similarity across both signals and regions.
Obs. 3. (BSEQ similarity and variety) Five canonical behaviors were observed in backfill sequences (Figure []. No cluster has over 2 21% of BSEQs from the same region, and no cluster has over 14% of BSEQS from the same signal. Model performance vs BERR. To study the rela- tionship between model performance (via Mean Abso- lute Error MAE of prediction) and B BERR, we use the difference between MAE com- puted against real-time target value and one against the stable target value. We analyze the top-performing real- time forecasting models per the comprehensive evalu- ation of all models in COVID-19 19 Forecast Hub (Cramer al.) 2021). YYG and UMASS-MB are mechanistic while CMU-TS and GT-DC are statistical models. The top performing ENSEMBLE is composed of all contributing models to the hub. We expect well-trained real-time model will have higher REVDIFAAE with larger BERR in its target (Reich) et al. 2019). However, we found that higher BERR does not necessarily mean worse performance. See Figure YYG has even better performance with more revisions. This may be due to the more complex backfill accivity/dependencies in COVID in comparison to the more regular seasonal flu.
REVDIEMMAE:
Jaden Ent
Sacofa Civer
(b) YYG Figure 3: BERR vs model REVDIFFMAE.
(a) GT-DC
Obs. 4. (Model performance and backfill) Relation between BERR and REVDIFFMAE can be non-monotonouy and posifively or negatively correlated depending on model and signal.
4

---

Published as a conference paper at ICLR 2022
Real-time target values to measure model performance: Since targets undergo revisions (5% BERR on average), we study how this BERR affects the real-time evaluation of models. From Figure that the scores are not similar with real-time scores over-estimating model accuracy. The average difference in scores is positive which implies that evaluators would overestimate models' forecasting ability.
Obs. 5. MAE evaluated at real-time overestimates model performance by 9.6 on average, with the maximum for TX at 22.63.
REFINING THE FUTURE VIA B2F
Our observations naturally motivate improving training and evaluation aspects of real-time forecasting by leveraging revision information. Thus, we propose the following two problems. Let predictions of model M for week t + k be (M, k)t- Since models are trained on real-time targets, ((M,)) k) is the model's estimate of target Vi+k (t+R)
Reak-mmeeE
w-week ahead Backfill Refinement Problem, BFRP At predic- (Di our target (0(0 For a model M trained on real-time targets, given history of model's predictions till last week (y, k)1,... yM, -)) and prediction for current week (M, k)t, our goal is to refine
Figure 4: Real-time vs sla- ble MAE.
tion week i, we are given a revision dataset 1:P' Jese, which includes
(ty)
y(M, k)), to better estimate the stable target Vitk- i.e. the 'future' of our target value at / + k:
Leaderboad Refinement problem, LBRP: At each week t, evaluators are given current estimate of our target Vi and forecasts of models submitted on week - k. Our goal is to refine ( to Dt. better estimate of (ty) so that using Út surrogate for y/- to evaluate predictions of models provides a better indicator of their actual performance (i.e., we obtain a refined leaderboard of models). Since LBRP aims refine the current estimate of the target, is closely related to the nowcasting problem. LBRP is also a special case of BFRP: Assume a hypothetical model Meval
(ty)
whose predictions are real-time ground truth, i.e. y(Menat, 0). Vi (t) VL. Then, refining Merat is equivalent to refining Vi
(t) better estimate (ty) which leads solving LBRP.
We leverage observations from Section to derive Back2Future (B2F), a deep-learning model that uses revision information from BSEQ to refine predictions. Obs. and show that real- time values of signals are poor estimates of stable values. Therefore, we leverage patterns in BSEQ of past signals and exploit cross-signal similarities (Obs. 3} to extract information from BSEQS. We also consider that the relation of models' forecasts to BERR of targets is complex (Obs. 4 and 5) to refine their predictions. B2F combines these ideas through its four modules: GRAPHGEN: Generates signal graph (where each node maps to a signal in Reg x Feat) whose edges are based on BSBQ similarities. BSEQENC: Leverages the signal graph as well as temporal dynamics of BSEQs to learn latent representation of BSEQS using a Recurrent Graph Neural Network. MODELPREDERS Encodes the history of the model's predictions, the real-time value of the target, and past revisions of the target through recurrent neural network. REFINER: Combines encodings from BSEQENC and MODEL.PREDES to predict the correction to model's 's real-time prediction.
Overview:
In contrast to previous works that studies target BERR (Reich al.) 2019). simultaneously model all BSEQ available till current week / using spatial and signal similarities in the temporal dynamics of BSQQ Recent works that attempt to model spatial relations for COVID19 forecasting need explicitly structural data (like cross-region mobility) (Panagopoulos et al. 2020) to generate graph or use attention over temporal patterns of regions' death trends. B2F, contrast, directly models the structural information of signal graph (containing features from each region) using BSEQ similarities. Thus, we first generate useful latent representations for each signal based on BSEQ revision information of that feature as well as features that have shown similar revision patterns in the past. Due to the large number of signals that cover all regions, we cannot model the relations between every pair using fully connected modules or attention similar to (Jin et al.) 2020). Therefore, we first construct a sparse graph between signals based on past BSEQ simi- larities. Then we inject this similarity information using Graph Convolutional Networks (GCNs) and combine it with deep sequential models to model temporal dynamics of BSEQ of each signal while combining information from BSEQ of signals in the neighborhood of the graph. Further,
5

---

Published as a conference paper at ICLR 2022
these latent representations and leverage the history of model M's predictions to re- fine its prediction. Thus, B2F solves BFRP, (M) assuming M a black box, accessing only its past forecasts. Our training process, that involves pre-training on model-agnostic auxiliary task, greatly improves training time for refining any given model M. The full pipeline of B2F is also shown in Figure 5 Next, describe each of the components of B2F in detail. For the rest of this section, we will assume that we are forecasting k weeks ahead given data till current week t.
GRAPHGEN generates an undirected signal graph E.) whose edges represent similar- ity in BSEQS between signals, where vertices == Reg x Feat. We measure similarity using DTW distance due to rea- sons described in Section 2 GRAPHGEN leverages the similarities across BSEQ patterns irrespective of the exact nature of canonical behaviors which may vary across domains. We compute the sum of DTW distances of BSEQs for each pair of nodes summed over L' E (1, t -- 5 to make the BSEQS to be reasonable length (at least 5) to capture temporal similarity without discounting too many BSEQS. Top node pairs with lowest total DTW distance are assigned an edge.
G
RNN
GComy
GraphGen
Bseq Similarity
RNNN
G.
RNNN
Gpiinnl Graph
BSeqs
Backfill
BSeqEnc
Dynamics
Embedding
Backfill
Refined Prediction
Past Model Predictions
Model EEc
Model Blas
Refiner
Embudding
Current Week
Prediction
Figure 5: B2F pipeline with all components
5). We threshold L' till
BSEQENC. While we can model backfill sequences for each signal independently using a recurrent neural network, this doesn't capture the behavioral similarity of BSEQ across signals. Using fully-connected recurrent neural network that considers all possible interactions between signals also may not learn from the similarity information due to the sheer number of signals (50 21 1050) while greatly increasing the parameters of the model. Thus, we utilize the struetural prior of graph G, generated by GRAPHGEN and train an autoregressive model BSEQENC which consists of graph recurrent neural network to encode a latent representation for each of backfill sequence in B, = (BSQQii, t) E FJ. At week f, BSEQENC is first pre-trained and then fine-tuned for specific model M (more details later in this section).
Our encoding process is in Figure 5 Let BSEQr be first + values of BSEQ(i, t') (till week t' +)). For past week t' and revision week r, we denote (t,)
h,, E R. to be the latent encoding of BSEQ,, (i, t') where t = t' +r and t' S t I t. We initialize [. for any observation week t' to be learnable parameter n!!)) E RL specific to signal i. For each week E.. we combine latent encoding hi (1,-1) and signal value ((()) using GRU (Gated Recurrent Unit) (Cho et al. 2014) cell to
get the intermediate embedding u,,t"
(t') Then, we leverage the signal graph G2 and pass the embeddings (1.)
(4.) fure
E F} through Graph Convolutional layer (Kipf & Welling 2016 get h
GRUBE (d Ri
(11-1),
(6) (ier (h Jier GConv(G,, (u.p)) her).
Thus, N contains information from BSEQ (i, (') and structural priors from G.. Using (C))
BSEQENC predicts the value d(f(+1) by passing through a 2-layer feed-forward network FFN,; ((++1) = (M)))) During inference, we only have access to real-time values of signals for the current week. We autoregressively predict (1+1) for each signal by initially passing {d( her through BSEQENC and using the output (d((++1) JieF as input for BSEQENC. Iterating this 7 times we get [
(1+1) heF along with ((++)) HeFR where is hyperparameter.
MODeLPred ENC. To learn from history of a model's predictions and its relation to target revisions, MODEL encodes the history of model's predictions, previous real-time targets, and revised (up to current week) targets using a Recurrent Neural Network. Given a model M, for each observa- tion week / 2,, ... 1-1-k), k). concatenate the model's predictions y( (M, k)e, real-time target

---

Published as a conference paper at ICLR 2022
Bi as seen on observation week t' and revised target #/ as C = MMM, k)v @ Bi 2 (t)
yerk
where @ is the concatenation operator. A GRU is used encode the sequence {c
c(u) --1-
{2 9 (1) A GRUMB([C1
C -1)1-)))
REFINER. It leverages the information from above three modules of B2F to refine model M's predic- tion for current week y(M, k),, Specifically, it receives the latent encodings of signals { hee from BSEQENC, - from MODELPREDENC, and the model's prediction y(M, k)r for week t.
(1+4)
BSEQ encoding from different signals may have variable impact on refining the signal since a few signals may not very useful for current week's forecast (e.g.. small revisions mobility signals may not be important in some weeks). Moreover, because different models use signals from COVDS differently, we may need to focus some signals over others to refine its prediction. Therefore,
first take attention over BSEQ encodings from all signals (m(++)) heer w.r.t yMM, k)). We use multiplicative attention mechanism with parameter w E R. based Vaswani et al. (2017):
ai softmax, k)ewh ki,"",
70(00
Σ SEF
(3)
- through a two layer feed-forward layer FNNRF which outputs a 1-dim value followed by tanh activation to get the correction % E [-,,]] i.e., 7t tanh(FFNRr(hh
Finally we combine (,(4) and (1)
@ 21th =)) Finally, the refined prediction is (M, K)₂ = (> + (),(M, K)e- Note that we limit the correction by B2F by at most the magnitude of model's prediction because the average BERR of targets is 4.9% and less than 0.6% of them have BERR over 1. Therefore, we limit the refinement of prediction this range.
Training: There are two steps of training involved for B2F: 1) model agnostic autoregressive BSEQ prediction task pre-train BSEQENC; 2) model-specific training for BFRP.
Autoregressive BSEQ prediction: Pre-training on auxiliary tasks to improve the quality of latent
embedding a well-known technique for deep learning methods (Devlin et al. 2019 Radford et al..
2018 We pre-train BSEEENC to predict the next values of backfill sequences {
(f,
her. Note
that we only use BSEQ sequences {BSEQ, (t', (e) available till current week t for training BSEoENC. The training procedure in itself is similar to Seq2Seq prediction problems (Sutskever
et al. 2014) where for initial epochs we use the ground truth inputs each step (teacher foreing) and then transition to using output predictions of previous time step by the recurrent module as input to next time step. Once we pre-train BSEQENC, we can it for BFRP well as LBRP for current week for any model M. Fine-tuning usually takes less than half the epochs required for pre-training enabling quick refinement of multiple models in parallel.
Model specific end-to-end training: Given the pre-trained BSEQENC, we train, end-to-end, the parameters of all modules of B2F. The training set consists of past model predictions (y(M, k) U(M, k)2.... ... -) and backfill sequences (BSEQ, i)eer For datapoint (M, k)) of week t' < k. we input backfill sequences of signals whose observation week is
(t) (t) (her.. We also derive 2/ from MODELPREDENC and finally REFINER ingests 28 (1) (hy FiiE and v(M, k)) to get //. Overall, we optimize the loss (0) 2
t' into BSEQENC to get latent encodings
function: _()) = Σ'= (HH)(M, k)r ve++ Following real-time forecasting, we train B2F each week from scratch (including pre-training). Throughout training and forecasting for week 4, we use Ge as input to BSEQENC since it captures average similarities in BSEQS till current week t.
4 BACK2FUTURE EXPERIMENTAL RESULTS
In this section, we describe detailed empirical study to evaluate the effectiveness of our framework B2F. All experiments were run in an Intel i7 4.8 GHz CPU with Nvidia Tesla A4 GPU. The model typically takes around 1 hour to train for all regions. The appendix contains additional details (all hyperparameters and results for June-Dec 2020 and i = 3 and GDP forecasting). We also release the code and datasets at www. github. com/ Adi tyaLab/ / BBcck2uturre
Setup: We perform real- time forecasting of COVID-19 related mortality hDeaths) for 50 US states. We leveraged observations (Section 2) from BSEQ for period June 2020- Dec. 2020 to design B2F.
7

---

Published as a conference paper at ICLR 2022
We tuned the model hyperparameters using data from June 2020 to Aug. 2020 and tested it on the rest of dataset including completely unseen data from Jan. 2021 to June 2021. For each week t, we train the model using the CoVDS dataset available till the current week t (including BSEQs for all signals revised till 1) for training. As described in Section 3 for each week, we first pre-train BSEQENC on BSEQ data and then train all components of B2F for each model we aim to refine. Then, we predict the forecasts Deaths ha y (M, k), for each model M. Similarly we also evaluated B2F for real-time GDP forecasting task with detailed results in Appendix. We observed that setting hy hyperparameter cl where E {2. 3, 4, 5 provided best results. Note that influences the sparsity of the graph as well as the efficiency of the model since sparser graphs lead fast inference across GConv layers. We also found setting = 5 provided the best performance.
Refined prediction y/ (M, K)+ are evaluated against the most revised version of the target -g where (f is second week of Feb 2021. For evaluation, we use standard metrics in this do-
Evaluation:
main Reich et al. 2019 Adhikari et al. 2019). Let absolute error of prediction e(M, k): = -iite
(2y)
y" (M, K) for a week / and model M. We use (a) Mean Absolute Error MAE(M) 77 E
e(M, k)). / Wi-+h I- e(M,k) and (b) Mean Absolute Percentage Error MAPE + E
(3g)
Candidate models: We focus on refining/rectifying the top models from the COVID-19 Forecast Hub described in Section these represent different variety of statistical and mechanistic models.
Table B2F consistently refines all models. % improvements in MAE and the novel problem. there are no standard baselines. Therefore, 2.85 11.3 our baselines are used to study the effect of using BSEQs and model bias(MB) refinement. (a) FFN: train a simple feed-forward neural
Baselines: Due to
MAPE scorey averaged over all regions from Jan 2021 to June 2021
k=
k=4
Cand. Model
Refining Model
MAE
MAPE
MAE
MAPE
FFN
20.3500.1
0.12
0.22
0.7 0.64
0.77
B2F-MB
2.23 4 U.2
1.37 4 0.65
0.33
ENSEMBLE
B2F-NOGRAPH
-143 2 0.14
2.7 1 0.35
3.722.0.11
-6.72 0.7220082
B2F-BSEQ
1,42
0,37 = 0.75
0.74
0.44
Back2Fuutue
5,25 1 0.13
4.39 + 0.62
4.41 + 0.73
3.15 + 0.57
FFN
2.22:0022
-1.51
+ 0.90
-1.54
1 0.57
-0.48 =: 0.43
B2F-MB
0.440
-3.41 1 0.16
-2.91 0.229
-3.22 -
GT-DC
NOI-NOGRAPH
2,24 0.33
3,51
1 0.21
1,93 0.39
0.78
B2F BSEQ
2.13 +.142..12 0.12
3.84
+ 0.78
1.08 0.23
2.33 0.99
Back2Future
10.33
11.4 0.18
9.92 0.98
11.27 : 0.38
FFN
-208 0.39
0.12
-2.64
0.13
-3.36
network for regression
B2F-B
3384 4 0.88
6.99 4 0.50
-8.84 4 0.96
-5.61 4.61
task that takes as inputs model's pre-
YYG
B2F-NAG
-123 0.30
-0.7
1 0.90
-6.13
-5,31 0.06
B2F-
-1.78 0.74
-2.26 0.83
0.22
-0.62
Backue
8.93 17340.20 0.26
6.32 4 0.44
7.32 4: 0.42
5.73 0.66
diction and real-time
FFN
-5.74 0.75
-1.01
0.18
-5.28 : 0.07
target to predict the
B2F-MB
-.61
-7.54 1 0.26
-6.49 +
-7.56 0.30
stable target. (b)
UMASS-MB
BOF-NOGRAPH
2.22
-1.88 + 0.67
0.215
2.34
BSEQ
1.58 0.49
0.86 + 0.17
0.36 0.06
B2F-MB: exploits 0.96 0.82
Back2Future
1.51
4.66 1 0.63
3.32
3.21
only model bias and
FFN
+.54
4.9 + 0.39
0.71
-0.65
uses the MODELPRE-
B2F.MB
8.174
8.21 + 0.24
4..24...3
-6.11 0.34
CMU-TS
NOGRAPH
40.77 0.69
-0.57 0.32
-0.46
1.4600000
-1.77
DENC architecture and
B2F-BSEQ
1,46
1.05
- 0.16
2,38 4 0.43
2.26 append linear layer
BackrFuture
7.5 + 0.60
8.04
+ 0.58
5.73 +: 0.19
6.22 + 0.58 that takes encodings
from MODELPREDENC and model's prediction (c) B2F-BSEQ: exploits only BSEQ without model bias and only uses BSEQENC architecture and append a linear layer that takes encodings from BSEQENC and model's prediction (d) B2F-NOGRAPH: does not use BSEQ similarity from GRAPHGEN by removing graph convolutional layers and retain only RNNs.
Refining real-time model-predietions We compare the mean percentage improvement (decrease) in scores of B2F refined predictions of diverse set of top models w.r.t stable targets over 50 US states. We observe that B2F is the only method, compared baselines, that improves scores for all candidate models consistently [T. The poor scores of baselines also shows the necessity of incorporating both backfill information (unlike FFN and B2F-MB) and model prediction history (unlike 22F-BSFQ and B2F-NOGRAPPH
We achieve substantial avg. improvements of 6.93% and 6.79% in MAE and MAPE respectively with low standard deviation across 29 test weeks which shows that the improvements were consistent across time and not just over few weeks that experienced large revision anomalies. Candidate models refined by B2F show improvement of over 10% in over 25 states and over 15% in 5 states (NJ, LA, GA, CT, MD). The improved predictions of CMU-TS and GT-DC (ranked 3rd and 4th in COVID-19
8

---

Published as a conference paper at ICLR 2022
Forecast Hub) due to B2F, outperform all the models in the hub (except for ENSEMBLE) with 7.17% and 4.1%% improvements in MAE respectively. UMASS-MB, ranked 2nd, is improved by 11.24%. B2F also improves ENSEMBLE. the current best-performing model of the hub, by 3.6% 5.18% with over 5% improvement in 38 states, and with II and TX experiencing over 15% improvement.
Rectifying real-time model-evluuaiin: We evaluate the efficacy of B2F in rectifying the real-time evaluation scores for the LBRP prob- lem. We noted in Obs X that real-time MAE was lower than stable MAE by 9.6 on average. The difference between B2F rectified estimates MAE stable MAE was reduced 4.4, a $1.117 decrease (Figure 7 This results in increased MAE scores across most regions towards stable estimates. Eg: We reduce the MAE difference in large highly populated states such as GA by 26.1% % (from 22.52 to 16.64) and TX by 90% (from 10.8 1.04) causing increase in MAE scores from real-time estimates by 5.88 and 9.4 respectively.
(a) Average % decrease of MAE Figure 6: (a) B2F refines ENSEMBLE predictions significantly for most states. (b) efficacy of B2F ramps up within 6 weeks of revision data.
(b) % improve.
MAE for each week
Refinement function of data availability: During the initial weeks of the pandemic, have access to very little revision data both in terms of length and number of B BSQQQ. So we evaluate the mean performance improvement for each week across all regions (Figure (p). B2F's performance ramps and quickly stabilize in just 6 weeks. Since signals need around 4 weeks (Obs 2 to stabilize, this ramp-up time using small amount of revision data is impressive.
B2F adapts to anomalies During real-time forecasting, models need to also robust rare events of large anomalous data revisions, like during the initial stages of the pandemic when data collection was not fully streamlined. Consider observation week 5 where there was an abnormally large revision to deaths nationwide (Figure Ba) when the ERR was 48%. B2F still provided significant improvements of up to 74.2% for most model predictions (Figure Sb).
250 1WO 190000
Figure 7: B2F rectified MAE are closer to sluble MAE
B2F refines GDP forecasts To evaluate the extensibility of B2F to other domains that encounter the problem of backfill, we tested on the task of forecasting US National GDP using 25 macrocconomic indicators from past and their revision history for years 2000-2021. We found that B2F improves predictions of candidate models by 6%-15% and significantly outperforms baselines. The details of the dataset and results are found in Appendix Sections B
E.2
5 CONCLUSION
%6 improvemennt MAE
We introduced the important and chal- lenging multi- variate backfill problem using COVID-19 19 and GDP forecast- ing as examples. We compiled and released the comprehensive Co CDV dataset to study revision patterns in features and targets as well as aid in COVID-19 forecasting. We presented Back2Future (B2F), the novel deep-learning method to model this phenomenon, which exploits our observations of cross- signal similarities using Graph Recurrent Neural Networks refine predictions and rectify evaluations for wide range of models. Our extensive experiments showed that leveraging similarity among backfill patterns as well as model bias via our proposed method leads to significant 6- 11 % improvements in all the top models.
X
No. Weoks
Emannoe 0T-06
VYG
CHU-TT
(a) Revision of US deaths
(b) % decrease in MAE due to B2F refinement on week targets Figure 8: B2F adapts to abnormally high revisions
week 5
As future work, our work can potentially help improve data collection and alleviate systematic differences in reporting capabilities across regions. For example, B2F provides significant gains consistently across time including when there are large anomalies in data. Therefore, our revision modelling approach can be helpful for anomaly detection (Homayouni al. 2021). We can also study how backfill can affect uncertainty calibration in time-series analysis (Yonn et al. 2020). Adapting to situations where data revisions can occur at different frequencies is another research direction.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
The features used in the CoVDS dataset and for GDP forecasting are publicly available and anonymized without any sensitive information. Our backfill refinement framework and B2F is generalizable to any domain that deals with real-time prediction tasks with feature revisions. Due to the relevance of our dataset to public health and macroeconomics, prospects for misuse should be discounted. The disparities in data collection across features and regions can also have implications on equity of prediction performance and is an interesting direction of research.
REPRODUCIBLITY STATEMENT
As described in Section we evaluated our model over runs with different random seeds to show the statistical significance of our method. We also provide more extensive description of hyperparameters and data pre-processing in the Appendix. The code for B2F and the CoVDS dataset publicly available at
8 ACKNOWLEDGEMENTS
This work was supported in part by the NSF (Expeditions CCF-1918770, CAREER IIS-2028586. RAPID 2IS-202786,, Medium IIS-1955883, Medium IIS-2106961, CCF-2115126), CDC MInD program, ORNL, and faculty research awards from Facebook, funds/computing resources from Georgia Tech.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Bijaya Adhikari, Xinfeng Xu, Naren Ramakrishnan, and B Aditya Prakash. Epidcep: Exploiting embeddings for epidemic forecasting. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 577-586, 2019.
Angel Aguiar. Macroeconomic data. 2015.
Nick Altieri, Rebecca L Barter, James Duncan, Raaz Dwivedi, Karl Kumbier, Xiao Li, Robert Netzorg, Briton Park, Chandan Singh, Yan Shuo Tan, Tiffany Tang, Yu Wang, Chao Zhang, and Bin Yu. Curating a covid-19 data repository and forecasting county-level death counts in the united states. Harvard Data Science Review, 2 2021. doi: 10. 10.1162/99608892.166688 Ideedda. URLpbttps:/ codsr.mippress mit httptt/e///pbbpppeeyeet// http:///sssmmipppss.....cct///pp
Apple. Apple mobility trends reports., 2020. URL www.
Alberto Baffigi, Roberto Golinelli, and Giuseppe Parigi. Bridge models forecast the euro area gdp. International Journal of forecasting, 20(3):447-460, 2004.
Matthew Biggerstaff, Michael Johansson, David Alper, Logan C Brooks, Prithwish Chakraborty, David C Farrow, Sangwon Hyun, Sasikiran Kandula, Craig McGowan, Naren Ramakrishnan, et al. Results from the second year of collaborative effort forecast influenza seasons in the united states. Epidemics, 24:26-33 33, 2018.
Logan C. Brooks, David C. Farrow, Sangwon Hyun, Ryan J. Tibshirani, and Roni Rosenfeld. Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions. PLOS Computational Biology, 14(6):e org/ 1006134, BtLLhttp::/// wttp://ppp................. 1t.p///...111.1/..1 13710 pcbi. ptt.:e..p.....11....1 1006134
June 2018. ISSN 1553-7358. doi: 10.
plos.
Andrea Carriero, Michael P Clements, and Ana Beatriz Galvão. Forecasting with bayesian multivari- ate vintage-based vars. International Journal of Forecasting, 31(3):757-768, 2015.
CDC. Coronavirus Disease 2019 (COVID-19), 2020. URL https: / www. www.edec.... gov
Eoronavirus/2019 Sanaavirss/2119-noo//raeaee www.occaeeeeoeeo ncor/casss-updttess/
Prithwish Chakraborty, Pejman Khadivi, Bryan Lewis, Aravindan Mahendiran, Jiangzhuo Chen, Patrick Butler, Elaine O Nsoesie, Sumiko R Mekaru, John S Brownstein, Madhav V Marathe, et al. Forecasting a moving target: Ensemble models for ili case count predictions. In Proceedings of the 2014 SIAM international conference on data mining, pp. 262-270. SIAM, 2014.
Prithwish Chakraborty, Bryan Lewis, Stephen Eubank, John S. Brownstein, Madhav Marathe, and Naren Ramakrishnan. What to know before forecasting the flu. PLOS Computational Biology, 14(10), October 2018. ISSN 1553-734X. doi: 10. 12r1331/jjoo 1w..137/jjoumm..ehee pebi. 1005664. |
7
nlm. nih wtt.://wwww....//o/////////ca http://////rrrtttseseeeeeeeeee
Judith A Chevalier, Jason L Schwartz, Yihua Su, Kevin R Williams, et al. Measuring movement and social contact with smartphone data: A real-time application to covid-19. Technical report, Cowles Foundation for Research in Economics, Yale University, 2021.
Kyunghyun Cho, B. V. Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. ArXiv, abs/ 12ss19999.15,, 2014.
Michael P Clements and Ana Beatriz Galvão. Data revisions and real-time forecasting, In Oxford Research Encyclopedia of Economics and Finance. 2019.
COVID--raccknn. The covid tracking project., 2020. URL https:://co..dtrrckkt/
Estee Y Cramer, Velma K Lopez, Jarad Niemi, Glover E George, Jeffrey C Cegan, Ian D Dettwiller, William P England, Matthew W Farthing, Robert H Hunter, Brandon Lafferty, et al. Evaluation of individual and ensemble probabilistic forecasts of covid- 19 mortality the us. medRxiv. 2021.
Dean Croushore. Forecasting with real-time data vintages. The Oxford handbook of economic forecasting, pp. 247-267, 2011.
11

---

Published as a conference paper at ICLR 2022
Aron Culotta. Towards detecting influenza epidemics by analyzing twitter messages. In Proceedings of the first workshop on social media analytics, pp. 115-122, 2010.
Songgaojun Deng, Shusen Wang, Huzefa Rangwala, Lijing Wang, and Yue Ning. Cola-gnn: Cross- location attention based graph neural networks for long-term ili prediction. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 245-254, 2020.
Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT,, 2019.
David Farrow. Modeling the past, present, and future of influenza. Private Communication, Farrow's work will be part of the published Phd thesis at Carnegie Mellon University, 2016.
Jeremy Ginsberg, Matthew H Mohebbi, Rajan S Patel, Lynnette Brammer, Mark S Smolinski, and Larry Brilliant. Detecting influenza epidemics using search engine query data, Nature, 457(723:): 1012-1014, 2009.
Google. Google covid-19 19 community mobility reports., 2020. URL https: httpp://www.googz 2
Iwona Hawryluk, Henrique Hoeltgebaum, Swapnil Mishra, Xenia Miscouridou, Ricardo P Schneken- berg, Charles Whittaker, Michaela Vollmer, Seth Flaxman, Samir Bhatt, and Thomas A Mel- lan. Gaussian process nowcasting: Application to covid-19 mortality reporting. arXiv preprint 122. 2021.
Inga Holmdahl and Caroline Buckee. Wrong but Useful What Covid-19 Epidemiologic Models Can and Cannot Tell Us. NEJM, 383(4):303-335, 2020.
Hajar Homayouni, Indrakshi Ray, Sudipto Ghosh, Shlok Gondalia, and Michael G Kahn. Anomaly detection in covid-19 19 time-series data. SN Computer Science, 2(4):1-17, 2021.
Xiaoyong Jin, Yu-Xi Wang, and Xifeng Yan. Inter-series attention model for covid-19 19 forecasting. ArXiv, abs/2010. 13006, 2020.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 0w..//699909000 2016.
Vasileios Lampos, Tijl De Bie, and Nello Cristianini. Flu detector-tracking epidemies on twitter. In Joint European conference on machine learning and knowledge discovery in databases, pp. 599-602. Springer, 2010.
Vasileios Lampos, Andrew C Miller, Steve Crossan, and Christian Stefansen. Advances in nowcasting influenza-like illness rates using search query logs. Scientific reports, 5(1):1-10, 2015.
JF Lawless. Adjustments for reporting delays and the prediction of occurred but not reported events. Canadian Journal of Statistics, 22(1):15-31, 1994.
Shenghua Liu, Bryan Hooi, and Christos Faloutsos. Holoscope: andology-nnd-spike aware fraud detection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1539- 1548, 2017.
Massimiliano Marcellino. A linear benchmark for forecasting gdp growth and inflation? Journal of Forecasting, 27(3) -3000 2008
Jessica E Metcalf and Justin Lessler. Opportunities and challenges in modeling emerging infectious discases. Science, (5776667):149--552 2017.
Baltazar Nunes, Isabel Natário, and M Lucília Carvalho. Nowcasting influenza epidemics using non-homogeneous hidden markov models. Statistics in Medicine, 32(15):2443-2660 2013.
Dave Osthus, Ashlynn R Daughton, and Reid Priedhorsky. Even good influenza forecasting model can benefit from internet-basee nowcasts, but those benefits are limited. PLoS computational biology, 15(2)c 1065599, 2019a.
12

---

Published as a conference paper at ICLR 2022
Dave Osthus, James Gattiker, Reid Priedhorsky. Sara Y Del Valle, et al. Dynamic bayesian influenza forecasting in the united states with hierarchical discrepancy (with discussion). Bayesian Analysis, 14(1):261-31,, 2019b.
G. Panagopoulos, Giannis Nikolentzos, and M. Vazirgiannis. United we stand: Transfer graph neural networks for pandemic forecasting. ArXiv, abs/2009.08388. 2020.
George Panagopoulos, Giannis Nikolentzos, and Michalis Vazirgiannis. Transfer graph neural net- works for pandemic forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. 2018.
Prashant Rangarajan, Sandeep K Mody, and Madhav Marathe. Forecasting dengue and influenza incidences using a sparse representation of google trends, electronic health records, and time series data. PLoS computational biology, 15(11): 15000)18 2019.
Nicholas G. Reich, Logan C. Brooks, Spencer J. Fox, Sasikiran Kandula, Craig J. McGowan, Evan Moore, Dave Osthus, Evan L. Ray, Abhinav Tushar, Teresa K. Yamana, Matthew Biggerstaff, Michael A. Johansson, Roni Rosenfeld, and Jeffrey Shaman. A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States. Proceedings of the National Academy of Sciences of the United States of America, 1 116(8)3114-3154 2019. ISSN 1091-6490. doi: 10. 107paasss 1812594116 16.
John Robertson and Ellis W Tallman. Vector autoregressions: forecasting and reality. Economic Review-Federal Reserve Bank of Atlanta, 84(1):4, 1999.
Alexander Rodríguez, Nikhil Muralidhar, Bijaya Adhikari, Anika Tabassum, Naren Ramakrishnan, and B Aditya Prakash. Steering a historical disease forecasting model under a pandemic: Case of flu and COVID-19, In Proceedings of the AAAI Conference on Artificial Intelligence, 202 la.
Alexander Rodríguez, Anika Tabassum, Jiaming Cui, Jiajia Xie, Javen Ho, Pulak Agarwal, Bijaya Adhikari, and B. Aditya Prakash. DeepCOVID: An Operational Deep Learning- Drivvn Framework for Explainable Time COVID-19 Forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence. 2021b.
Gerhard Rünstler and Franck Sédillot. Short-term estimates of euro area real gdp by means of monthly data. Technical report, ECB working paper, 2003.
Jeffrey Shaman and Alicia Karspeck. Forecasting seasonal outbreaks of influenza. Proceedings of the National Academy of Sciences, 109(50):20425 20430, 2012.
Katrina Stierholz. Economic data revisions: What they are and where to find them. DmP, 45:4, 2017.
Oliver Stoner and Theo Economou. Multivariate hierarchical frameworks for modeling delayed reporting in count data. Biometrics, 76(3):789-79, 2020.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In NIPS, 2014.
Greg Tkacz and Sarah Hu. Forecasting gdp growth using artificial neural networks. Technical report, Bank of Canada, 1999.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/ 0bs//700.005 700603372, 2017.
Peng-Wei Wang, Wei-Hsin Lu, Nai- Nai-Ying Ko, Yi-Lung Chen, Dian-Jeng Li, Yu-Ping Chang, and Cheng-Fang Yen. Covid-19-relant 19-related information sources and the relationship with confidence in people coping with covid-19: Facebook survey study in taiwan. Journal of medical Internet research, 2266):220 2020.
Shihao Yang, Mauricio Santillana, and Samuel C Kou. Accurate estimation of influenza epidemics using google search data via argo. Proceedings of the National Academy of Sciences, 112(47): 14473-14478, 2015.
13

---

Published as a conference paper at ICLR 2022
Changchang Yin, Ruoqi Liu, Dongdong Zhang, and Ping Zhang. Identifying sepsis subphenotypes via time-aware multi modal auto-encoder. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 862-872 2020.
Jaesik Yoon, Gautam Singh, and Sungjin Ahn. Robustifying sequential neural processes. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 10861-10870. PMLR, 13-18 Jul 2020.
ptep:rcceeeoo............eee///e http://........eeeo/o/oooo/oo/.. html
mir
Qian Zhang, Nicola Perra, Daniela Perrotta, Michele Tizzoni, Daniela Paolotti, and Alessandro Vespignani. Forecasting seasonal influenza fusing digital indicators and mechanistic disease model. In Proceedings of the 26th International Conference World Wide Web, pp. 311-319. International World Wide Web Conferences Steering Committee, 2017.
Indre Zliobaite. Change with delayed labeling: When is it detectable? In 2010 IEEE International Conference on Data Mining Workshops, pp. 843-850. IEEE, 2010.
14


Reviews:
[{"review_id": "j-jB-owjyC3", "reviewer": "Reviewer_9pAe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- Characteristics of the backfill dynamics problem is very well motivated with observations from COVID-19 data that the authors have collected. While people have long known that corrections to data over time will have effects on forecasts, this paper is the first (to my knowledge) that carefully examines the prevalence and effects of backfill dynamics.\n- Very general B2F pipeline that can be applied to \"post-process\" the predictions of any forecasting model. The wide applicability could lead to high impact.\n- Strong experimental results on real data, showing consistent improvement of COVID-19 death forecasts for 5 different candidate models.\n- Paper is very well organized and written, with only some minor issues that I list below.\n\nWeaknesses:\n\n- Fit for ICLR may not be the best. There's not really a contribution towards learning representations, as the B2F pipeline integrates existing neural network architectures. I think this is a better fit for a data mining venue such as KDD.\n- Text in many figures, including Figure 1, 3, and 4, is too small to be readable. I recommend removing some of the results, for example, converting Table 2 to a figure that shows only the ensemble model. A more complete version of Table 2 is available in Table 4 of the appendix anyways.\n\nMinor presentation issues:\n\n- Figure 3 caption: Usually an x-y plot is described as y vs. x, whereas this caption is x vs. y.\n- Obs. 4: non-monotonous $\\rightarrow$ non-monotonic\n- Page 9, 2nd paragraph: \"B2F rectified estimates MAE stable MAE\": Insert \"in\" between \"MAE\" and \"stable\".", "summary_of_the_paper": "The authors consider the effects of backfill dynamics--the correction of historical data--on time series prediction. They use COVID-19 forecasting as the motivating application. Both the features (ER visits, hospitalization rate, etc.) and the target (deaths) are subject to revision, and the authors collect a data set showing that the backfill error due to revisions can be extremely large. The authors make several other interesting observations regarding backfill error and sequences in their data. A key finding is that real-time forecasts tend to overestimate their accuracy when compared to the stable accuracy after errors are corrected. The authors then propose a Back2Future (B2F) pipeline that can be used to refine predictions from a model when backfill dynamics are present. They demonstrate consistent improvement on stable accuracy for multiple COVID-19 forecasts.", "summary_of_the_review": "The paper contributes both new data and a new problem setting, motivated by forecasting COVID-19 deaths over time. Due to the high novelty and very general proposed B2F pipeline for refining model predictions, this paper could have high impact, although it may not be the best fit for the ICLR audience.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "QOXchl1lGDg", "reviewer": "Reviewer_4Gyj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose an pipeline to alleviate the effects of revising historical data \n(backfill problem) in the context of timeseries prediction.\n\nStrong points:\n- Well motivated. The authors present clearly the backfill problem and its effects on the predictions thus enabling the reader to understand the importance of the suggested solution.  \n- Well written. The use of the COVID dataset as the driving \napplication has successfully contributed to the understanding of the pipeline. \n- Generalizability. The pipeline can be deployed to refine the predictions and improve the performance of any model. \n- Thorough experimental analysis and strong results. \n- Sound technical backbone.\n\nWeak points:\n- At parts the writing feels aesthetically dense. I would suggest that the authors try and cut some of \nthe text so that tables/figures such as Table 2 (which is one of the main results of the paper) \nis not so tightly surrounded by text. \n\n-- Other:\n\nAbstract\n\"Back2Future, that aims to refines a given model’s predictions ...\" -> \"Back2Future, that aims to refine a given model’s predictions ...\"\n\nIntroduction\nWorth unfolding the CDC acronym to centers for disease control and prevention. Currently I suspect that the majority of the readers will be well aware of this acronym but this might not be the case as some point later in the future. \n\n\" many times after and is known as the ’backfill’ phenomenon.The various factors ...\" --> please mind the right direction in the single quotes and for aesthetics, it is worth leaving a space after a period and before the beginning of the next sentence. \n\n’backcasting’ --> please correct the direction of the single quotes.\n\n\"Building on our empirical observations, we formulate the problem BFRP, ...\"--> Please unfold he acronym as this is the first time introduced. \n\nNature of backfill dynamics\n\"Also, note that the data in the backfill is the same used for real-time forecasting, but just seen from a different perspective.\" --> This needs clarification. What is the different perspective. \n", "summary_of_the_paper": "The authors deal with the problem of revising previous recorded data and its effect on timeseries predictions. They showcase how revisions in past data, quantified as the backfill error, can introduce a considerable error in predictions. Towards that, they propose a novel deep learning approach, the Back2Future, that refines the model predictions using backfill dynamics. They demonstrate its efficiency on a real COVID dataset. \n", "summary_of_the_review": "The proposed work is well motivated, well written, sound and contributes a model that solves a very interesting problem in timeseries prediction with strong results. Its capability of use along with any model adds to its value. I would be happy to see this work accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "PI0k2hh-nYB", "reviewer": "Reviewer_apgM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an approach to overcoming the so-called backfill problem when time-series data are revised to correct previous recorded values. The authors propose a recurrent neural network to carry out the task. The technical part of the paper is very well written and the intuition behind various modeling decisions is explained in a clear and simple way. I also appreciated the careful preliminary data analysis of the covid data that motivate and back the model proposal. The authors carry out a comparative evaluation against several CDC-approved forecasting models showing convincing better results.   \n\nStrengths:\n- The paper is very well written. The problem is well presented, notation is defined carefully, and the model is described with enough detail.\n- A careful COVID data analysis serves to motivate and back the model proposal. \n- The work is very relevant to the current situation and ongoing research\n\nMinor comments:\n- is it possible to give some reasons for the revisions in the case of covid?\n- In section 2.1, clarify what is the spatial analysis unit adopted.\n- What is the model performance when a finer spatial unit is adopted such as counties.\n \n", "summary_of_the_paper": "Summary: The paper proposes a real-time forecasting model for situations where the predictive features and the response variable are both subject to recording delays. The authors propose a recurrent neural network model and evaluate it against several baseline models using a COVID dataset.", "summary_of_the_review": "This is a very well-written paper with good explanations, proper motivation, clear objectives, and results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces Back2Future, a deep learning approach for refining predictions when 
backfill dynamics are present.  

All reviewers agree on that the authors successfully motivate their work and 
introduce a topic of great interest, i.e. that of dealing with the effect of revising previously recorded data and its effect 
timeseries predictions. The reviewers also underline the strong and thorough experimental section.
Among the reviews is also underlined the potential impact of the work for the research domain. 

Many thanks to the authors for replying to the minor concerns raised.

I concur with the reviews and find this submission very interesting, convincing and thus 
recommend for accept. 

Thank you for submitting the paper to ICLR.
Decision: Accept (Poster)