Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a mechanism of re-learning (ITER; iterative re-learning) to handle issue from non-stationarity in RL. Some gain of using the proposed method is experimentally presented. My major concern is the limited understanding of the non-stationarity issue and ITER. 

Pros)
- The authors propose a new approach to resolve the non-stationarity issue in RL. The intuition itself makes a sense as human often performs such relearning processes to escape from the local minimum. 

- Some gain of ITER is empirically demonstrated in various setting.

Cons)
- Although the underlying intuition sounds delightful, the understanding on the non-stationarity is insufficient. In particular, the authors design a setup of "supervised" learning to show importance of non-stationarity. However, it is a weak evidence on the importance of non-stationarity in "reinforcement" learning. Indeed, the gain of using ITER is not significant.

- The authors do not clearly define the generalization which the relearning aims to improve. It may depend on the neural network architecture and environment and so on. However, presuming that the paper reports the best gain possible, it is hard to accept that ITER always improves generalization.

- I have a concern on the little gain, which is shown by the comparison between PPO vs. PPO+ITER with the same hyperparameters. A fair comparison may use the best hyerparameters for each of PPO and PPO+ITER. Or, at least, there need comparisons with different hyperparameters to claim consistent improvement.

review_title: Limited understanding on the non-stationarity in RL and benefit from ITER

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper investigates an interesting problem that transient non-stationarity can affect the generalization of the neural network. This paper first conducts experiments on a supervised learning task to illustrate that transient non-stationarity can lead to degenerated performance on testing set. Then, the paper proposes an RL algorithm called ITER to avoid the negative impact of such non-stationarity.

Strengths: 
+ This paper observes a novel problem that may appear in RL and designs a new algorithm to prevent such a problem.
+ This paper investigates this problem through experiments on supervised learning tasks.

Weaknesses:
- The new algorithm ITER doubles the computational costs.
- Experiments on Page 7 are trying to further illustrate the mechanism for such a phenomenon. However, this subsection is not very clear.
- Experiment results on Procgen is fair but not significant.

Minor comments:
- In Section 2, the advantage function is defined as A^\pi(s,a,s') but later used as A^\pi(s,a) without additional explaination.
- f in $\mathcal{D}_{f,m}$ is not explained when it first appears. I guess it is the ratio of the modification.
review_title: Official Blind Review #3

final_score: rating: 7: Good paper, accept
review_content: The paper deals with a relevant issue. The simplified supervised learning setting is a good way of looking at the issue of non-stationarity in isolation and it makes a compelling case that neural networks optimized by SGD can have generalization issues in settings where the data distribution changes over time, even after the data distribution converges. The solution proposed by the paper is simple and can be applied to most off-the-shelf RL algorithms. My main criticism would be that the hybrid objective feels rather ad-hoc and that the proposed method could use a bit more theoretical justification. Also, since the issue being tackled here is quite general to RL, a wider set of benchmarks and base-algorithms (in addition to PPO) would be necessary to get a better picture.
Regardless, I believe the community would benefit from the inclusion of this paper. 

### Pros

* The paper is well-written. The story is easy to follow and well-motivated.
* The central problem that is being tackled is highly relevant.
* I like that the problem is illustrated in a supervised-learning setting, which allows to investigate without the noise of a typical RL setup.
* The proposed modification is simple and can be applied to a wide-range of algorithms.
* The "legacy feature" effect is an interesting phenomenon and the additional experiments inspecting it are useful.

### Cons

* In Figure 5, the middle panel is a bit hard to parse. I don't know how to improve this but in its current state, too many variables are being presented at once.
* While the method being presented is quite simple, it also seems a bit ad-hoc. For instance, it would be nice if the teacher-student distillation could be supported by some convergence guarantees, e.g. by showing that (under some circumstances) replacing the teacher by the student does not increase the loss.
* The set of benchmarks considered in the paper is a good proof-of-concept, but additional experiments with different environments and base-algorithms is necessary to better judge the merits of the approach.
review_title: This paper tackles the problem of distribution shift in the context of reinforcement learning, e.g. resulting from gradual exploration of the state-space or optimization of the critic. The main idea is that this kind of non-stationarity can have a lasting negative effect on generalization, even after the data distribution has converged. The paper proposes a solution which is based on iteratively distilling the current model into a new model that is optimized from scratch.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents empirical evidence that non-stationarity data typical in deepRL settings can affect the intermediate representation of deep neural network and affect testing performance. The paper is easy to read and the authors provide experiments to support the their observations and claims. Overall I think this is a good paper and in the following I suggest some good to have additions.

(1) The examples for the supervised learning setting clearly demonstrates the impact of non-stationary data. However, given that this is inspired by the problems under DRL setting, it will be interesting to do more analysis of this effect on some DRL tasks. For example, an analysis for offline RL might be a good setting to study this effect.

(2)Imitation learning algorithm like Dagger might be another good example to demonstrate the effect of nonstationarity. The data under the Dagger setting is also changing overtime and it will be interesting to see how it affects the student policy.

(3) The RL experiment is mainly done in the on policy (PPO) settings. Some experiments with off policy RL setting might be useful, and the effect of the non-stationarity might be more pronounced as well.
review_title: interesting paper analyzing how non-stationarity affects generaliztion

Meta-review: There is a substantial contribution in identifying novel questions/issues, as this paper certainly does. Neither I nor the reviewers have seen this issue of transient non-stationary before, and the authors make a compelling case for it, especially in the supervised setting with the CIFAR experiments. It is less compelling through the RL experiments. As such, this paper is likely to inspire new work within the field. To me, Figure 1 is the most interesting aspect of the whole paper.

The initial approach by the authors is questionable in its effectiveness, and is likely to be improved by others in the future. Some of the results in Figure 3 are questionable, especially when you look at the individual curves in Figure 8. So overall, this means that the authors have identified a truly novel issue, and proposed an initial method that is just okay.  They've done a nice job investigating this in a supervised setting, and need to push further in the RL setting.

The question is whether the novel contribution of the problem outweighs that the algorithm and its evaluation could use improvement.  The reviewers debated this in the discussion, with points on both sides, but the novelty of the question/issue (even if the investigation could use work) is likely to inspire further work in this direction.

Other notes:
The authors could have evaluated the (impractical) version of their algorithm proposed in the first paragraph of Section 4.2. This would inform 1) whether their parallel training approximation is close to the optimal algorithm, and 2) whether the optimal (impractical) algorithm is capable of improving generalization significantly. If the latter is true, it would leave open a huge avenue of investigation to find better approximate solutions.
Decision: Accept (Poster)