Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method to automatically evaluate the contribution of a data point from some historical distribution to a new task without knowing the exact distributions of both old data and the new task. Experiental results show that the proposed wighting method works consitently better than several baseline methods on some simulated data sets.
strength_and_weaknesses: Strength: 1. it will be very convenient if we are able to evaluate the contribution of a data point to a new task without knowing exactly which distribution it were sampled from.
 2. the idea of using a DL method for importance weight estimating is interesting.

Weaknesses:
1. the theoretical properties are lacking - The issue of how the variation of the estimator is influenced by the data and how to controll the variation is not discussed in the paper.

2. there are many other methods to do the same thing (importance evaluating), e.g., using influence functions - some clarifying about the difference to those methods is needed.

3.  the supervised learning experiments are mostly based on simulated data without real data shifting -  something like recommendation applications may be considered where people's interests are changing constantly.

4. for the RL part - since the data is depended on what the underlying policy is, I don't think that learning a weighting network without considering the policy that generate the data is very appropriate - those policies are not necessarily changing graduately.
clarity,_quality,_novelty_and_reproducibility: good
summary_of_the_review: This paper proposes a neural network to estimate the importance sampling weight. But large variance is one critical issue of any importance sampling weight estimators, so it's better to further consider the effect of variance when using the proposed methods. Another point is that more realistic illustration of the effect of the proposed method is lacking.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This draft studies the problem of continual learning with streaming data in the presence of distribution change. The data distribution is changing over time. The proposed method uses a time-varying importance weight estimator to correct the distribution change. Experiments on online supervised learning and reinforcement learning show the effectiveness of their method.
strength_and_weaknesses: Strengths:
- This draft studies a well-motivated problem that is very common in real-world applications.

Weakness:
- The description of the proposed method is not very clear. Section 3 is difficult to follow. 
- The proposed method appears to need to first store all the data, then adaptively estimate the weights for all the past data, then optimize. This is not exactly a streaming data learning approach. How does the proposed method differ from offline methods like domain generalization?
- The authors propose two models for estimating $p_t(x)$, the exponential family and deviations from the marginal. Assumptions on the data evolution process are strong. These generational models are not well-discussed. How are these scenarios used, for instance, in what applications? Moreover, these two data generation assumptions do not match the experiments in which the authors use a label shift dataset.
- In the experimental part, the authors use a label shift dataset. It is suggested to compare with the baseline method in Wu et al. (2021).
clarity,_quality,_novelty_and_reproducibility: This draft is clear except for the method part. The novelty of the proposed method is limited while the experiments seem good. 
summary_of_the_review: This draft studies an interesting but important problem in real-world machine learning tasks, especially with the explosion of big data, which are collected over time and the distribution can also change over time. The proposed method is simple, but it lacks novelty. Assumptions on how data evolve are strong and do not accurately reflect reality. Moreover, the proposed method requires the storage of past data, which limits its application to real-world tasks. While the experiments seem good, the datasets used do not match the assumptions made in the method section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper deals with the problem that the behavior of the system to be modeled may change during the time of data acquisition. For the case that the behavior changes only gradually, a method is proposed, theoretically justified and empirically tested, which is intended to achieve good model quality for the time shortly after the end of the data acquisition.
strength_and_weaknesses: **Strengths**
* The topic is relevant in practice
* The method is theoretically justified and embedded in a broad context of previous work
* Systematic studies on this topic are still rare, in my opinion.
* The results are convincing

**Weaknesses**
* As far as I understand it, each experiment is repeated only three times, so the statistical significance of the results is somewhat unclear.

**Further suggestions for improvement**
* In some places boldmath should be used so that formula expressions are also written in bold, e.g. Remark 2, 3; Method 1; Figure 2 
* Please check if the sentence "The goal in these experiments is to build model that" is really meant that way, or if it should be "a model" or "models".
* Figure 2 partially covers the words "sample cpmplexity" above.
* In the bibliography, the first names of the authors are not written out in some entries, as is the case in the vast majority. E.g. J. Baxter, D.R. Cox.
* Please check the author list for "Alex Krizhevsky, , and Geoffrey Hinton."

**Further comments**
* It would be interesting to apply the method in an example where there is NO shift, i.e., where all data stems from the same distribution and thus the baseline method "Everything" is optimal. Can the proposed method achieve the same performance in this setting?
* The practice of exponentially downweighting older data relative to newer data is not new. Perhaps the authors can clarify this or provide a reference.
clarity,_quality,_novelty_and_reproducibility: **Clarity** Clearly written, although it's a bit of a pity Algorithm 1 didn't get a place in the main text.

**Quality** High

**Novelty** Good

**Reproducibility** Presumably sufficient.

summary_of_the_review: A well-written paper on an important topic that proposes, theoretically justifies, and empirically tests a new method. An increase in significance through a larger number of experiments and an investigation of how the method performs in the case of stationary data is desirable.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the setting of training under a gradual distribution shift over time. To mitigate the effect of distribution shift on the performance on future data, they propose applying importance weighting to simulate training under the most recent data distribution. They motivate this by arguing that the latest distribution will be similar to the next future time step since the change in distribution happens gradually. The main contributions of this paper are 
- A formalisation of training under distribution shift
- A method to estimate importance weights that depend on time
- Experiments in a supervised and RL setting to showcase the benefits of the method
strength_and_weaknesses: This paper focuses on the important setting of learning under continual distribution shifts. Indeed data naturally comes as a sequence and waiting until enough data is gathered to marginalize over time can be impractical in many applications. On top of that the authors make a strong case for the weakness of the marginal approach if one is interested in predicting the immediate future. I've found most of the paper to be clearly written; The transition from train/test propensity scores to time dependent ones was easy to follow, even when one isn't familiar with importance weighting and propensity scores. And the formalization of "gradual" shift seems to be coherent and valid in many cases. Therefore using p_T as a proxy for p_T+dt seems reasonable.

However, the experimental setting seems a little limited since 1) it requires access to the whole histore of the data 2) starts training from scratch at each time step. I can hardly imagine a setting where this is feasible in practice or for bigger models / larger datasets. The benefits on the datasets chosen except Gaussian (which might be too toyish) might not justify the overhead of training the discriminative model to estimate importance weights. 
The paper focuses on data that evolves through time but also assumes access to multiple samples per time step that are IID. The paper seems to try to be separate from the setting of continual learning but ends up using a sequence of datasets with known boundaries (the timestesp). I'm not 100% sure where the difference lies. 
clarity,_quality,_novelty_and_reproducibility: The paper is quite clear except for the section 3.2 which I did not fully grasp. The baselines are chosen fairly and well explained and motivated.
summary_of_the_review: The setting proposed is promising and an important one to study. The idea presented is simple (in a good way) and well motivated. However, the practical implementation of the algorithm seems to have too much overhead and the chosen benchmark datasets aren't different enough from already well established continual learning settings.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a novel time-varying importance weight estimator that can detect gradual shifts in the distribution of data. They evaluate their methods on synthetic image classification and reinforcement learning tasks that exhibit distribution shifts and show that the proposed method largely outperforms the standard methods (training on all data, "recent" data only, and fine-tuning on the recent data).
strength_and_weaknesses: The method is very well described, and the results on synthetic data are very compelling. 

Weaknesses: 

I find that the "CONTINUOUS LABEL SHIFT SETTINGS" paragraph in the appendix is not detailed enough. It is hard to get a sense of the distribution shift that is induced in the data. Maybe adding some visualization here would be great. 
Also, there is a public "corrupted" CIFAR10 dataset that allows to get CIFAR10 data, with different levels of corruptions. It might be useful (and more standard) to use this dataset for further analysis. 
 
An important component of supervised learning models is their uncertainty representation. if the proposed method correctly approximates the distribution shift, it should be reflected in the uncertainty representation and in the calibration of the model. How well does the model perform in terms of ECE, for example, and how does the uncertainty evolves as distribution shifts? I think such analysis would strengthen the paper. 

Finally, I would have liked to see real data experiments. There are some "common" settings and datasets for distributional shifts. For example, in neuroscience, decoding neural data during learning would be one (the distribution of the input changes as the animal learns the task) or in climate science (the basic models do not accurately forecast weather or temperature as the CO2 emissions increase, for example)


clarity,_quality,_novelty_and_reproducibility: The paper is of good quality, and very clear. 
summary_of_the_review: The derivations are sound and the proposed model is very interesting, but the evaluation lacks a bit of depth and real-data experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper deals with handling changes in non-stationary environments. The authors assume that data change gradually over time: this idea is formally made explicit using a constant to upper bound the total variation divergence of the distributions of data in a small amount of time. The assumption thus allows predicting samples at time T+dT (in the future) using a time-dependent propensity score built on samples at time T. The authors try to reuse past samples (t<=T) in order to make predictions on samples at time T by using a mechanism based on importance weighting. The weights are computed using the output of a probabilistic binary classifier that maps each sample to the time it was seen, that coincides with a specific distribution of data. In the end, the authors test their approach in both continuous supervised and reinforcement learning settings comparing it with different baselines.
strength_and_weaknesses: **Strengths**
- The approach is simple to understand
- The approach is applicable to different settings from supervised to reinforcement learning

**Weaknesses**
- The approach seems computational (and possibly sample) demanding due to the need for retraining the binary classifier (for the weights) and of the prediction model at each time step.
- The experimental evaluation does not succeed in showing a clear advantage of the proposed method
clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written, but there are some points that may deserve further attention. 

- Analyzing the algorithm details reported in the appendix, I found some aspects that are not clear to me. It seems there is a mistake in the definition of Algorithm2 "Generate Data" as the generation of the tuple in the case of z=-1 leads to triplets of the form (x_i, t', t) where t_i does not correspond to t, as instead stated in the definition of Method 1. It seems that the provided formulation of the function "Generate Data" is someway more appropriate to Method2. Also, in the section "Algorithm details" of the Appendix, the formulation (19) seems wrong as it never uses the value t_1 contained in the tuple. Could the authors try to clarify this point?
- Concerning the experimental section, it presents some critical aspects: the results reported for the Gaussian experiment do not show any advantage in using the proposed correction compared to the Recent and Finetune baselines. It is also not clear how, in the experiments with CIFAR-10, the Finetune approach shows worse performances than the Everything approach. Do you have an intuitive explanation for this? Furthermore, the authors only perform 3 runs for this set of experiments. I would suggest to report results averaged over more runs, as done for the Reinforcement Learning's experiments.
- Regarding the experiments related to the Reinforcement Learning, would it have been more appropriate to use as baselines other algorithms suited for non-stationary settings, besides only using vanilla SAC as done in ROBEL D'Claw environment?
- Another aspect is that, in the main paper, the authors claim that the experiments will show the performance of both method 1 and 2, however, in the appendix they state that for the experiments they only used method 2, thus not providing experiments using method 1. I would suggest the authors to also show the results of method 1.

The approach of reweighting past samples in gradually drifting environments seems also novel in the literature. As for the reproducibility, the authors report in the appendix all the hyperparameters needed to run the different experiments, but they do not provide the source code.
summary_of_the_review: The paper tackles the important problem of non-stationarity of data using the standard technique of importance weighting enriched with a time component in order to help the mapping of each sample to its associated time of arrival. The main ideas and assumptions presented in the paper are sound. The use of an importance-weighted estimator provides an unbiased estimate of the samples arriving at time T, which is a good proxy for the samples to come in the near future. Nevertheless, the concerns raised make me opt for a borderline negative score at the moment.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper poses an approach to continuous learning, a topic that deserves ample attention since it's one where ML systems are notoriously poor in. It addresses this experimentally by varying importance weights.

The authors are applauded for their extensive rebuttal efforts.  Yet, despite its strong experimental section, with experiments both with supervised learning (with data distributions evolving in time) and RL, the paper lacks a thorough theoretical background.  A paper in this form is not sufficiently consequential.
justification_for_why_not_higher_score: As described in the meta-review.
justification_for_why_not_lower_score: none.
Decision: Reject