Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors deal with the problem of revising previous recorded data and its effect on timeseries predictions. They showcase how revisions in past data, quantified as the backfill error, can introduce a considerable error in predictions. Towards that, they propose a novel deep learning approach, the Back2Future, that refines the model predictions using backfill dynamics. They demonstrate its efficiency on a real COVID dataset. 

main_review: The authors propose an pipeline to alleviate the effects of revising historical data 
(backfill problem) in the context of timeseries prediction.

Strong points:
- Well motivated. The authors present clearly the backfill problem and its effects on the predictions thus enabling the reader to understand the importance of the suggested solution.  
- Well written. The use of the COVID dataset as the driving 
application has successfully contributed to the understanding of the pipeline. 
- Generalizability. The pipeline can be deployed to refine the predictions and improve the performance of any model. 
- Thorough experimental analysis and strong results. 
- Sound technical backbone.

Weak points:
- At parts the writing feels aesthetically dense. I would suggest that the authors try and cut some of 
the text so that tables/figures such as Table 2 (which is one of the main results of the paper) 
is not so tightly surrounded by text. 

-- Other:

Abstract
"Back2Future, that aims to refines a given model’s predictions ..." -> "Back2Future, that aims to refine a given model’s predictions ..."

Introduction
Worth unfolding the CDC acronym to centers for disease control and prevention. Currently I suspect that the majority of the readers will be well aware of this acronym but this might not be the case as some point later in the future. 

" many times after and is known as the ’backfill’ phenomenon.The various factors ..." --> please mind the right direction in the single quotes and for aesthetics, it is worth leaving a space after a period and before the beginning of the next sentence. 

’backcasting’ --> please correct the direction of the single quotes.

"Building on our empirical observations, we formulate the problem BFRP, ..."--> Please unfold he acronym as this is the first time introduced. 

Nature of backfill dynamics
"Also, note that the data in the backfill is the same used for real-time forecasting, but just seen from a different perspective." --> This needs clarification. What is the different perspective. 

summary_of_the_review: The proposed work is well motivated, well written, sound and contributes a model that solves a very interesting problem in timeseries prediction with strong results. Its capability of use along with any model adds to its value. I would be happy to see this work accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors consider the effects of backfill dynamics--the correction of historical data--on time series prediction. They use COVID-19 forecasting as the motivating application. Both the features (ER visits, hospitalization rate, etc.) and the target (deaths) are subject to revision, and the authors collect a data set showing that the backfill error due to revisions can be extremely large. The authors make several other interesting observations regarding backfill error and sequences in their data. A key finding is that real-time forecasts tend to overestimate their accuracy when compared to the stable accuracy after errors are corrected. The authors then propose a Back2Future (B2F) pipeline that can be used to refine predictions from a model when backfill dynamics are present. They demonstrate consistent improvement on stable accuracy for multiple COVID-19 forecasts.
main_review: Strengths:

- Characteristics of the backfill dynamics problem is very well motivated with observations from COVID-19 data that the authors have collected. While people have long known that corrections to data over time will have effects on forecasts, this paper is the first (to my knowledge) that carefully examines the prevalence and effects of backfill dynamics.
- Very general B2F pipeline that can be applied to "post-process" the predictions of any forecasting model. The wide applicability could lead to high impact.
- Strong experimental results on real data, showing consistent improvement of COVID-19 death forecasts for 5 different candidate models.
- Paper is very well organized and written, with only some minor issues that I list below.

Weaknesses:

- Fit for ICLR may not be the best. There's not really a contribution towards learning representations, as the B2F pipeline integrates existing neural network architectures. I think this is a better fit for a data mining venue such as KDD.
- Text in many figures, including Figure 1, 3, and 4, is too small to be readable. I recommend removing some of the results, for example, converting Table 2 to a figure that shows only the ensemble model. A more complete version of Table 2 is available in Table 4 of the appendix anyways.

Minor presentation issues:

- Figure 3 caption: Usually an x-y plot is described as y vs. x, whereas this caption is x vs. y.
- Obs. 4: non-monotonous $\rightarrow$ non-monotonic
- Page 9, 2nd paragraph: "B2F rectified estimates MAE stable MAE": Insert "in" between "MAE" and "stable".
summary_of_the_review: The paper contributes both new data and a new problem setting, motivated by forecasting COVID-19 deaths over time. Due to the high novelty and very general proposed B2F pipeline for refining model predictions, this paper could have high impact, although it may not be the best fit for the ICLR audience.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary: The paper proposes a real-time forecasting model for situations where the predictive features and the response variable are both subject to recording delays. The authors propose a recurrent neural network model and evaluate it against several baseline models using a COVID dataset.
main_review: The paper proposes an approach to overcoming the so-called backfill problem when time-series data are revised to correct previous recorded values. The authors propose a recurrent neural network to carry out the task. The technical part of the paper is very well written and the intuition behind various modeling decisions is explained in a clear and simple way. I also appreciated the careful preliminary data analysis of the covid data that motivate and back the model proposal. The authors carry out a comparative evaluation against several CDC-approved forecasting models showing convincing better results.   

Strengths:
- The paper is very well written. The problem is well presented, notation is defined carefully, and the model is described with enough detail.
- A careful COVID data analysis serves to motivate and back the model proposal. 
- The work is very relevant to the current situation and ongoing research

Minor comments:
- is it possible to give some reasons for the revisions in the case of covid?
- In section 2.1, clarify what is the spatial analysis unit adopted.
- What is the model performance when a finer spatial unit is adopted such as counties.
 

summary_of_the_review: This is a very well-written paper with good explanations, proper motivation, clear objectives, and results.
review_title: null

Meta-review: This paper introduces Back2Future, a deep learning approach for refining predictions when 
backfill dynamics are present.  

All reviewers agree on that the authors successfully motivate their work and 
introduce a topic of great interest, i.e. that of dealing with the effect of revising previously recorded data and its effect 
timeseries predictions. The reviewers also underline the strong and thorough experimental section.
Among the reviews is also underlined the potential impact of the work for the research domain. 

Many thanks to the authors for replying to the minor concerns raised.

I concur with the reviews and find this submission very interesting, convincing and thus 
recommend for accept. 

Thank you for submitting the paper to ICLR.
Decision: Accept (Poster)