Transient Non-stationarity and Generalisation in Deep Reinforcement Learning
Authors: Maximilian Igl
Gregory Farquhar
Jelena Luketina
Wendelin Boehmer
Shimon Whiteson
Venue: ICLR 2021 Poster (2021)
Keywords: Reinforcement Learning
Generalization

Abstract: Non-stationarity can arise in Reinforcement Learning (RL) even in stationary environments. For example, most RL algorithms collect new data throughout training, using a non-stationary behaviour policy. Due to the transience of this non-stationarity, it is often not explicitly addressed in deep RL and a single neural network is continually updated. However, we find evidence that neural networks exhibit a memory effect, where these transient non-stationarities can permanently impact the latent representation and adversely affect generalisation performance. Consequently, to improve generalisation of deep RL agents, we propose Iterated Relearning (ITER). ITER augments standard RL training by repeated knowledge transfer of the current policy into a freshly initialised network, which thereby experiences less non-stationarity during training. Experimentally, we show that ITER improves performance on the challenging generalisation benchmarks ProcGen and Multiroom.

Published as a conference paper at ICLR 2021
TRANSIENT NON-STATIONARITTY AND GENERALISA- TION IN DEEP REINFORCEMENT LEARNING
Maximilian Igl
Gregory Farquhar" +-f
Jelena Luketina
Wendelin Böhmer'
Shimon Whiteson
ABSTRACT
can arise in Reinforcement Learning (RL) even in stationary environments. For example, most RI. algorithms collect new data throughout training, using a non-stationary behaviour policy. Due to the transience of this non-stationarity, it is often not explicitly addressed in deep RL and a single neural network continually updated. However, we find evidence that neural networks exhibit memory effect where these transient non-stationarities can permanently impact the latent representation and adversely affect generalisation performance. Consequently, to improve generalisation of deep RL agents, we propose Iterated Relearning (ITER). ITER augments standard RL training by repeated knowledge transfer of the current policy into a freshly initialised network, which thereby experiences less non-stationarity during training. Experimentally, we show that ITER improves performance on the challenging generalisation benchmarks ProcGen and Multiroom.
INTRODUCTION
In RL, as an agent explores more of its environment and updates its policy and value function, the data distribution it uses for training changes. In deep RL, this non-stationarity is often not addressed explicitly. Typically, single neural network model is initialised and continually updated during training. Conventional wisdom about catastrophic forgetting (Kemker et al.) 2018) implies that old updates from different data-distribution will simply forgotten. However, we provide evidence for an alternative hypothesis: networks exhibit a memory effect in their learned representations which can harm generalisation permanently if the dittrdistribution changed over the course of training.
To build intuition, first study this phenomenon in supervised setting on the CIFAR-10 10 dataset. We artificially introduce transient non-stationarity into the training data and investigate how this affects the asymptotic performance under the final, stationary data in the later epochs of training. Interestingly, we find that while asymptotic training performance nearly unaffected, test performance degrades considerably, even after the ditt--iitribution has converged. In other words, we find that latent representations in deep networks learned under certain types of non-stationary data can be inadequate for good generalisation and might not be improved by later training stationary data.
Such transient non-stationarity is typical in RI.. Consequently, we argue that this observed degradation of generalisation might contribute the inferior generalisation properties recently attributed to many RL agents evaluated on held out test environments (Zhang et al. 201 b Zhao et al., 2019). Furthermore, in contrast to Supervised Learning (SL), simply re- t-trrinnigg the agent from scratch once the data data-distribution has changed is infeasible in RL as current state of the art algorithms require data close to the on-policy distribution, even for off-policy algorithms like Q-learning (Fedus et al. 2020).
To improve generalisation of RI. agents despite this restriction, we propose Iterated Relearning (ITER). In this paradigm for deep RI training, the agent's policy and value are periodically distilled into a freshly initialised student, which subsequently replaces the teacher for further optimisation. While this occasional distillation step simply aims to re-learn and replace the current policy and
University of Oxford. Corresponding author: Maximilian Igl (maximilian.ig gmail.com
I Now at Deed London
Delft University of Technology

---

Published as a conference paper at ICLR 2021
value outputs for the training data, it allows the student to learn a better latent representation with improved performance for unseen inputs because it eliminates con-staiinnarity during distillation. We propose practical implementation of ITER which performs the distillation in parallel to the training process without requiring additional training data. While this introduces a small amount of non-stationarity into the distillation step, it greatly improves sample efficiency without noticeably impacting performance.
Experimentally, we evaluate ITER on the Multiroom environment, well several environments from the recently proposed ProcGen benchmark and find that it improves generalisation. This provides further support to our hypothesis and indicates that the non statiintton inherent to many RL algorithms,, even when training on stationary environments, should not be ignored when aiming to learn robust agents. Lastly, to further support this claim and provide more insight into possible causes of the discovered effect, we perform additional ablation studies on the CIFAR-10 dataset.
BACKGROUND
We describe an RL problem as a Markov decision process (MDP) r.. (Puterman 2014
with actions a A.. states S,, initial state 8p ~up transition dynamics s'~T((s, a). reward function r(s, a) E IR and discount factor The unnormalised discounted state distribution induced by a policy π is defined as d, (s) Er PrPr (S,=s|So ~ Po, Ar~ ( (S,,,, At)). In ITER, we learn sequence of policies and value functions, which we denote with x (als) and (s))s) at the kth iteration (k E {0, 1,,1.2.... }), parameterized by 9g.
We briefly discuss some forms of non-stationarity which can arise in RL, even when the envi- ronment is stationary. For simplicity, we focus the exposition on actor-critic methods which use samples from interaction with the environment to estimate the policy gradient given by = E[Va log Te (Ta (s, a, s,)s') mated as (s,s, a,
a).. The advantage is often esti- d V*ss. Typically, we also use neural networks to approximate the baseline V (s) and for bootstrapping from the future value V (s'). Φ can be learned by minimising (s, a, s') by stochastic semi-gradien descent, treating VT(ss)
r(s, a) + -
constant.
There are at least three main types of non-stationarity in deep RL. First, we update the policy To, which leads to changes in the state distribution dne (s). Early on in training, random policy To only explores states close to initial states sg- As Ta improves, new states further from So are encountered. Second, changes the policy also change the true value function (*) which V/7 (s) is approximating. Lastly, due to the use of bootstrap targets in temporal difference learning, the learned value V (s) is not regressed directly towards V (s()). Instead V fits gradually evolving target sequence even under a fixed policy ग, thereby also changing the policy gradient estimator g.
THE IMPACT OF NON-STATIONARITY ON GENERALISATION
In this section we investigate how asymptotic performance is affected by changes to the data- distribution during training. In particular, we assume an initial, transient phase of non-stationarity, followed by an extended phase of training on a stationary data-distribution. This is similar to the situation in RL where the dindri-istibbiionnnnnn is affected by a policy which converges over time. We show that this transient non non-stationarity has a permanent effect on the Iearned representation and negatively impacts generalisation. As interventions RL training can lead to confounding factors due to off-policy data or changed exploration behaviour, we utilise Supervised Learning (SL) here to provide initial evidence in a more controlled setup. We use the CIFAR-10 10 dataset for image classification (Krizhevsky et al. 2009) and artificially inject non-stationarity.
Our goal is to provide qualitative results on the impact of non-stationarity, not to obtain optimal performance. We use a ResNet18 (He et al., 2016) architecture, similar to those used by Espeholt ct al. (2018) and Cobbe et al. (2019a). Parameters are updated using Stochastic Gradient Descent (SGD) with momentum and, following standard practice in RL, we use constant learning rate and do not use batch normalisation. Weight decay is used for regularisation. details can be found in appendix|B]
parpee-prraeters and more
We train for a total of 2500 epochs. While the last 1500 epochs are trained on the full, unaltered dataset, we modify the training data in three different ways during the first 1000 epochs. Test data is
2

---

Published as a conference paper at ICLR 2021
left unmodified throughout training. While each modification is loosely motivated by the RL setting, our goal is not to mimic it exactly (which would be infeasible), nor to disentangle the contributions of different types of non-stationarity. Instead, aim to show that these effects reliably oceur in the presence to various types of non-stationarity, and provide intuitions that can be brought into the RI. setting in Section
For the first modification, called Dataset Size, we initially train only on a small fraction of the full dataset and gradually add more data points after each epoch, constant rate, until the full dataset is available after epoch 1000. During the non-stationary phase, data points are reused multiple times ensure the same number of network updates are made each epoch. For the modification Wrong Labels s we replace all training labels by randomly drawn incorrect ones. After each epoch, a constant number of these labels are replaced by their correct values. Lastly, No i sy Labe similar to Wrong Label lbels, s, but the incorrect labels are sampled uniformly at the start of each epoch. For both, all training labels are correct after epoch 1000. While Dataset Size zi inspired by the changing state distribution seen by an evolving policy, Wrong Labels and Noisy Labels are motivated by the consistent bias or fluctuating errors a learned critic can introduce the policy gradient estimate.
The results are shown in fig. While the final training accu- racy (left) is almost unaffected (see table [ in the appendix for exact results), all three non- stationary modifications signif- icantly reduce the test accu- racy (right). The plateau in accuracy shows that this ef- fect persists even after the mod- els are further trained using the full dataset with correct la- bels: non-stationarity early in training has a permanent effect on the learned representations and quality of generalisation. These results indicate that the non-stationarity introduced by the gradual convergence the policy in RL might similarly de- teriorate the generalisation of the agent. To overcome this, we propose ITER the next section. The key insight enabling ITER is that the observed negative effect is restricted to the test data, whereas the predictions on the training data are unaffected and of high quality.
100
98
84
96
82
Training acDvESicy
accuracy
92
78
90
Ulassdiliedd
76
88
Noiry Labels
Wring Labels
86
Size
72
1.0
2.0
0.0
0.5
1.0
1.5
2.0
2.5
Epochs
163
Epochs
1c3
Figure 1: Accuracy on CIFAR- 10 when training data non- stationary over the first 1000 epochs (dashed line). The remaining epochs are trained on the full, unaltered training data. Testing is performed on unaltered data throughout. While final training performance (left) is almost unaffected, test accuracy (right) is significantly reduced hy initial, transient non-stationarity.
4 ITER
In section 3 we have seen evidence that the non-stationarity which is present in many deep RL algorithms might lead impaired generalisation on held-out test environments. To mitigate this and improve generalisation to previously unseen states, we propose Iterated Relearning (ITER): instead of updating single agent model throughout the entire training process, ITER learns sequence of models, each of which is exposed to less non-stationarity during its training, As we will show in section this improves generalisation. ITER can be applied on top of wide range of base RL training methods. For simplicity, we focus in the following exposition on actor-critic methods and use Proximal Policy Optimization (PPO) (Schulman et al., 2017) for the experimental evaluation.
The underlying insight behind ITER is that at any point during RI. training the latent representation of our current agent network might be significantly damaged by non-stationarity, but its outputs on the training data are comparatively unaffected (see fig. D]. Consequently, ITER aims to periodically replace the current agent network, the 'teacher', by a 'student" network which was freshly initialised and trained to mimic the teacher on the current training data. Because this re-leannigg and replacement step can be performed on stationary data, it allows us to re-learn a policy that matches performance on the training data but generalises better novel test environments.
3

---

Published as a conference paper at ICLR 2021
ITER begins with initial policy (r and value function V and then repeats the following steps, starting with iteration k 0.
1. Use the base RL algorithm to train (kk) and (()),
2. Initialise new student networks for x (((++) and V(++1), We refer to the current policy 7(k) and value function V the teacher.
3. Distill the teacher into the student. This phase is discussed in more detail in section 4.1
4. The student replaces the teacher: ((K) and ((k) can be discarded.
5. Increment & and return step 1. Repeat as many times as needed.
This results in alternating training with distillation into freshly initialised student. The RL training phases continue to introduce non-stationarity until the models converge, so we want to iterate the process, repeating steps 1-4. How often do depends as hyper-parameter. In practise we found the results to be quite robust to this choice and recommend, as general rule, to iterate often as possible within the limits outlined in section 4.2 There, we introduce a practical implementation of ITER which re-uses data between steps and 3 in order to not require additional samples from the environment.
the environment and can be chosen
4.1 DISTILLATION LOSS
Our goal during the distillation phase (step 3) is learn a new student policy ((++1) and value function ((++) that imitate the current teacher (x),,, (())). If the student and teacher share the same network architecture, the student could of course imitate the teacher exactly by copying its parameters. However, since the teacher trained under non-stationarity, its generalisation performance likely degraded (see section 3). Consequently, we want to instead train a freshly initialised student network mimic the teacher's 's outputs for the available data, but learn better internal representation by training a stationary data distribution collected by the teacher xt (8), i... da (s) (alss.
The student, parameterised by 0g+1, is trained using a linear combination of four loss terms:
[(0++1) az + Ly + Lrs + XIDCID
where ATD a fixed hyper-parameter and we linearly anneal Or and av from some fixed initial value to zero over the course of each distillation phase.
C, and Lv are supervised losses minimising the disagreement between outputs the student and the teacher:
L. (0x+1) = Exnd (8)
(T) (++b)]
(2)
(1)(0())-V(++)()))) v(k+t) (s)
[v(0x+1)= Eandy(R))
The additional terms LPG and LTD are off-policy RL objectives for updating the actor and critic:
(ro(0x+1) = -Bardd (1) ama s'
log log (k+1)
2(11
( (s,t)(s,,
(())())))
2 (s. (s, (s),, (ID(Ok+1) Egnd (1) (k) s' s'
((+))(()) where (1++1((8, a') r(s,a) - - ( choosing action and 1 is a stop- gradient operator, its operand treated as a constant when taking derivatives of the objective. In practice the losses in eq. (2) remain nonzero during distillation, potentially causing a drop in performance once the student replaces the teacher. The off-policy RL losses in eq. (3) allow the student already take performance the RL task into account, reducing or eliminating this performance drop. We use PPO losses to optimise eq. (3) in our experiments.
denotes the estimated advantage of
4.2 COMBINING TRAINING AND DISTILLATION
To fully eliminate non-stationarity during the distillation step we need to collect additional data from the environment using fixed teacher policy. However, this would slow down training by increasing the total number of samples required. Here, to improve sample efficiency, we propose two practical implementations of ITER which reuse data between teacher and student:
4

---

Published as a conference paper at ICLR 2021
0,70
10
PPO+IEER
PPO-UER
0.65
I20
PPO
F20
0.60
0.8
0.55
5000085 Fale
0.6
0.50
-5
0.45
-10
0.40
0.2
-15
0.55
0.30
-20
1.0
1.5
3.0
0.9
0.2
0.4
0.5
0.8
1.0
Steps
1e8
r0om
rooms
3 rooms
rooms
1c9
(a) Multiroom
(b) Multiroom, by layout
(c) Boxoban
Figure 2: Evaluation Multinoom and Boxoban. Shown are mean and standard error over twelve seeds. Left: Retur for with and without ITER on Multiruom. Dotted lines indicate when the network was replaced by a new student. Middle: Evaluation on layouts with fixed number of rooms; training is still with random number of rooms. ITER's 's advantage is more pronounced for harder levels. Right: Return on Boxoban.
Sequential training: Store the last N transitions that were used to update the teacher in dataset D. During the distillation phase, draw batches from DD instead of collecting new data from the environment. While this does not introduce non-stationarity, it leads to evaluating the teacher on old, off-policy data, for which the quality of its outputs may be degraded. Furthermore, some of the data might be obsolete under the current tiatediistrbbttin and we require additional memory to store D.
Parallel training: Whenever the teacher is updated on batch of data B, also update the the student on the same batch. This approach introduces some non-stationarity as distillation performed multiple batches B while the teacher is changing. However, the teacher evolves much less over the course of the distillation phase than does a policy trained from scratch to achieve the same performance. In practise no additional memory is required, the teacher is only evaluated on data on which it is currently trained and updates the student and teacher can be performed parallel.
found this be worthy trade-off. Advantages of this method are that
Both approaches perform similarly in our experimental validation. We use parallel training for the main experiments due to the smaller memory requirements, the ability efficiently perform the student distillation in parallel and because tuning the hyper-parameter was significantly easier: While tuning the size of for sequential ITER involves trading off overfitting (for small D) against off- off-policy data (for large D). in parallel ITER the results were robust to the choice of how many batches were used the distillation phase as long as some minimum number was surpassed. Consequently, hyper-paramett tuning for parallel ITER only involved increasing the length of the distillation phase until no drop in student performance was observed when replacing the teacher. We set a and av = 0.5 initial values without further tuning as preliminary experiments showed no impact within reasonable ranges.
5 EXPERIMENTS
In the following, we evaluate ITER on Multiroom (Chevalier-Bootss several environments from the ProcGen (Cobbe et al.) (019a) benchmark which was specifically designed to measure generalisation by introducing separate test- and training levels. We also provide ablation studies showing that parallel and sequential training perform comparably, and that the loss find that ITER improves generalisation, which also supports our hypothesis about the negative impact of transient non-stationarity in RL. Lastly, we re-visit the SL setting from section [ and perform additional experiments leading us to the formulation the legacy feature hypothesis to explain the observed effects.
Willems, 2018) and on
terms eq. 33 in eq. ( are beneficial.
5.1 EXPERIMENTAL RESULTS ON MULTIROOM
First, evaluate ITER the Multiroom environment. The agent's task is to traverse sequence of rooms to reach the goal (see fig. for example layout) as quickly as possible. can take discrete actions to rotate 90" in either direction, move forward, and open or close the doors between rooms.
5

---

Published as a conference paper at ICLR 2021
1.10
25
1.05
11.5
1.00
11.0
Tet Pefformace
Teol Tert Performnnne
3.95
10.5
0.90
10.0
9.5
PPGHTER
0.80
9.0
Evalustion on Tod leve Training levels
10
0.75
IRACHTTER
8.5
ABAC
0.70
8.0
2
Steps
168
Steps
168
Steps 168
Figure 3: Evaluation on ProcGen. Dashed lines indicate replacing the teacher. Left: Test performance averaged over six environments (SturPilot, Dodgeball. Climber, Ninja, Fruitbot and BigFish). Shown are mean and standard
all 30 runs (five per enviromment). Results normalised by the final test- sest-prffrmmncc of the PPO baseline each respective environment make them comparable. We also compare against the previous state of the art method IBAC-SNI [Igl et al. 2019). Middle: Evaluation on Climber. ITER improves test performance without improving training, supporting our claim that ITER improves the latent representation of the agent. Right: Evaluation on BigFish. On some environments, ITER improves both train- and test- performance.
The observation contains the full grid, one pixel per square. Object type, including empty space and walls, as well as any object status, like direction, are encoded in the three 'colour' channels. For each episode, new layout is generated by randomly placing between one and four connected rooms on the grid. The agent is always initialised in the room furthest from the goal. This randomness favours agents that are better at generalising between layouts memorisation is impossible due to the high number of feasible layouts. The results are shown in fig. 2 Using ITER on top of PPO increases performance. The performance difference is pronounced for layouts with because such layouts are harder and likely only solved later in training, at which point negative effects due to prior non-stationarity in training are more pronounced.
rooms, possibly
5.2 EXPERIMENTAL RESULTS ON BOXOBAN
We also evaluate ITER on Boxoban (uuez al. 2018 Schrader 2018
See fig. for an example. Similarly to Multiroom the observation contains the full grid, one pixel per square. Object types are encoded by colour. Again, new layout is generated at the beginning of each new episode, favouring agents that can generalise well between states. Actions allow to push, pull move in all four cardinal directions, or do nothing. The goal of the agent is to position the four boxes, which can be pushed or pulled, the four available targets. Walls prevent movement for both the agent and the boxes. Positive rewards are provided for positioning a box on a target (r, 1) and successfully solving each level (rz 10). A small negative reward per time-step (r, = 0.1) encourages fast solutions. As shown in fig. 2 ITER learns much faster. We provide additional results and examples for wrongly chosen distillation lengths in fig. 8 Note that both for Multiroom and Boxoban we train and test on the same (very large) set of possible layout configurations, so the main expected advantage of ITER is more sample efficient training due better generalisation. In the next section, we will evaluate the agent on previously unseen environments, directly measuring its generalisation performance.
5.3 EXPERIMENTAL RESULTS ON PROCGEN
Next, evaluate ITER several environments from the ProcGen benchmark. We follow the evaluation protocol proposed in (Cobbe et al. 2019a): for each environment, we train on 500 randomly generated level layouts and test additional, previously unseen levels. Due to computational constraints, we consider a subset of five environments. We chose StarPilot, Dodgeball, Climber, Ninja, Fruitbot and BigFish based the results presented in (Cobbe al. 2019a) as they showed baseline generalisation performance better than random policy, but with large enough generalisation gap. ITER improves performance for both PPO and Information Bottleneck Actor Critic (IBAC) with selective noise injection (Igl et al.) 2019). Results are presented in fig. 3 and more individual plots, including performance on training levels, can be found in the appendix. fig. we show in ablations
OOur simplified Hoxoban-Iiral n-v0 also allows pulling boxes reduce computational costs

---

Published as a conference paper at ICLR 2021
1.2
1.1
Train encoders
Phase
(x)((x)
Phase
1 (x)
Tex: Performaice
9
1.0
Train last layer
Wolm bfT
Wfm bfo 2
On S
0.9
0.8
PPO ITER
Evaluate
How relevant is
How well does //th generalize?
ITER Sequencal
aoRL
On D test
gf for D test
0.7
0
2
3
6
Steps
1c8
Figure 4: Left: Ablation studies with sequential ITER and ITER without terms Lxs LTD (eq. (3).. Right: Schematic depiction of training setup for fig. [ (middle and right). More details are given in section 5.4 D is the unmodified CIFAR-10 training data while for DÍ. modification m E (Noi sy Labels, Wrong Labels Dataset. S1 ze} } is applied to the fraction (1 - f) of all data-points. In this two phase training setup, we first train on D during phase and continue on 2> during phase 2. A linear predictor parameterised by (W..m b((")) trained D after each phase i, while holding the encoder (.""(x) (x) fixed. Evaluation of the resulting classifiers is performed the original test data. Classifier measures the relevance the legacy features while classifier measure the final generalisation performance.
that both the parallel and sequential implementations of perform comparably, while using the off-policy RL terms LPG and LyD in eq. (I) decreases performance.
Similarly to previous literature (Cobbe al., 2019b Igl et al. 2019). we found that weight decay improves performance and apply it to all algorithms evaluated on ProcGen. Our results show that the negative effects of non-stationarity cannot easily be avoided through standard network regularisation: we can improve test returns through ITER despite regularisation with weight decay and IBAC, both shown to among the most effective regularisation methods on this benchmark (igl et al. 2019).
In the previous two sections we have shown the effectiveness of ITER in improving generalisation of RT. agents. Because the main mechanism of ITER reducing the non-stationarity the training data which is used to train the agent, this result further supports that such transient non-stationarity is detrimental to generalisation in RL.
5.4
SUPERVISED LEARNING ON CIFAR- 10
86
88
0.0
84
86
-0,5
-0.3
10'
Unmodifeed Dutsei
12
-0,9
Nomalize Eigenalus
Nuty Labds
Wrong
60
oiniaal
-1.
Dataed Size
#2
-1.5
-1.8
Vanadiid Deloce
80
-2.1
74
Nairy Label
Wring
78
Nekey Labda
Waing
-24
72
Dursst Size
Dannel Size
-.7
76
-30
25
100
Epochs
Accumay Intermetiate Repe.
Figure 5: Left: Test accuracy of students (solid lines) that only lear to mimic the behaviour of poorly generalising teachers in fig. (dashed lines). Middle: Final test accuracy of networks trained consecutively on two different datasets. only the last layer on the second: nearly useless earlier representations impact future learning much less than slightly sub-ope sub-optimal ones. Markers indicate modifications to first dataset: colours indicate the fraction of unmodified data points f. Dashed line shows accuracy for normalised by the largest SV. Solid lines show intermediate values of with low accuracy, dashed lines small values of with higher accuracy. More plots can be found in the appendix.
z-axis shows accuracy of using encoders trained on the dataset, retraining
Right: Singular values of feature matrix &
In this section, we aim to further understand the mechanism by which non-stationarity impacts generalisation by revisiting the easily controlled SI. setting presented in section
7

---

Published as a conference paper at ICLR 2021
First, we confirm that, like with ITER for RL, we can improve generalisation while only learning to mimic the outputs of a poorly generalising teacher for the training data. We train the teacher using the same setup in section In a second step, train freshly initialised student for 1000 epochs to fit the argmax predictions of this teacher on the training data, i.e., the true labels in the training data are unused. Test accuracy is still measured using the true labels. The results of this distillation phase are shown in fig. 5 (left). The student (solid lines) recovers the test accuracy achieved by stationary training, compared to the poor asymptotic teacher performance (dashed lines) from fig. This confirms that the teacher's 's outputs on the training data are suitable targets for distillation.
Second, we aim to better understand why non- non-stationarity affects the generalisation performance. To do so, we investigate the latent representation of the network. We view all but the last layer as the encoder, producing the latent representation <(x) E Rp for input x, on which the classification is linear: y = softmax ((x) + b) with W E R denoting a weight matrix and b E R/C| denoting a bias vector. By features we refer not to the representation ((x) as a whole, but to aspects of the input to which the encoder learned to pay attention and which therefore can impact the latent representation (Kriegeskorte & Kievit 2013). More quantitatively, we can define the representation matrix 1 E _" consisting of the latent representations of all N = 10000 test data points. Performing Singular Value Decomposition (SVD) on @ yields mutually orthogonal directions (the sight-singular vectors) in which the latent representations of the various inputs are different from one another. We can see each such direction as corresponding to one feature, with the corresponding singular value expressing its strength, i.e., how strongly it impacts the latent representation.
Our hypothesis is that under a non-stationary data distribution, the encoder is more likely to reuse previously learned features (as these are already available) instead of learning new features from scratch. If these old (or "legacy") features generalise worse on the new data, for example because they are overfit to smaller or less diverse dataset, this in turn deteriorates generalisation permanently if they are not replaced. This leads to two predictions: First, the observed drop in generalisation should only occur if the previously learned features are relevant for the new task, but suboptimal. If they are irrelevant, they will not be reused. If they are optimal, they do not negatively impact generalisation. Second, we expect the final network to rely on more features in its latent representation if it is reusing suboptimal features: because these are not as general, more features are required to discriminate between all inputs.
To experimentally evaluate both predictions, we simplify the experimental setting to two phases (see fig. 4 for schematic depiction of the training setup). The first training phase uses a stationary, but modified, dataset Dí.m, and the second phase uses the full, unmodi- fied, training dataset DD To generate DJ.m. we use the same modifications as before, m (Noisy Labels, Wrong Labels, Dataset Size), but instead of annealing the fraction of correct data points f from 0 I allows us to tune the relevance of the features learned on Df.m (also see fig. 6). In this setup the only -oatttiiionaitty is the change in data from phase 1 to phase 2. We first train the network for 700 epochs on Dm. which yields an intermediate representation (g))))())) followed by another 800 epochs on D yielding the final representation ofmal (x) (see fig. for training curves).
in section 3 it is fixed at certain value. Changing this value
To test our first hypothesis, we want to measure how relevant the representation offter (x) is for the final data distribution and how well f. ff generalies. We train a linear predictor for each fixed
representation on the full dataset D The test accuracy of the classifier based on &inter (x) measures how well we can perform on D with features learned on Df.m, i..., their relevance. The test accuracy of the classifier based on final measures how well the final network was able to recover from the initial bias and learn to generalise well despite non-stationarity.
f.m
plot the accuracy of the intermediate predictor (i.e., the relevance) the a-sxis and of the final generalisation performance on the y-axis. Each point corresponds to one value of f E (0, 1J. shown colour, and one modification type m as indicated by the marker shape. By changing ff from 0 to (i.c., from blue to dark red) we can increase the relevance of the intermediate features from nearly irrelevant to optimal on D. Interestingly, an almost useless intermediate representation (30% on the z-axis) does not impede the final performance much while relevant but suboptimal intermediate features (around 60% on the z-axis) lead to a marked drop in
In fig. 5 (middle),
Thhe linear predictor y = (W f. (2) + b), where a the softmax function and x the input image.
8

---

Published as a conference paper at ICLR 2021
performance. This supports our first hypothesis. The strong final performance for - 0 (i.e., for low relevance) also rules out decreased network flexibility, for example due to dead neurons (for RsLUs) or saturation (for tanh), the main driver of reduced generalisation.
Our second prediction is that relevant, but suboptimal features in Pinter should lead to the usage of more features in of compared to irrelevant or optimal features. To test this prediction, we plot in Figure (right) the singular values (SVs) of the representation matrix & E RR- as defined above. We plot the values for the smallest values of f (dashed lines, "irrelevant features") and for intermediate values of f (solid lines, 'relevant but sub-optimal features"), ). The blue line corresponds to optimal features. The tails of the singular values are heavier for intermediate values of f. indicating that the network is relying more features in those cases, supporting second prediction.
RELATED WORK
Knowledge distillation (Hinton et al. 2015) with identical teacher and student architectures has been shown to improve test accuracy (Furlanello et al. 2018), even in the absence of non-stationarities in the data. This improvement has been attributed to the case of predicting the output distribution of the
teacher compared to the original 'hard' labels (Mobahi et al. 2020: Gotmare et al. 2019). While apply such 'soft' distillation for ITER on RI, we use 'hard' labels in our SI. experiments.
Policy distillation has been applied RL (Czarnecki et al., 2019), for example for multi-task learning and compression (Teh et al. 2017: Rusu et al. [015]] Parisotto et al. 2016), imitation learning (Ross al. 10, or faster training (Schmitt et al. 2018 Ghosh al., 2018). Closer to ITER, Czarnecki et al. (2018) use policy distillation to learn sequence of agents. However, their Mix & Match algorithm solves tasks of growing complexity, for example, grow the action space of the agent, not to tackle generalisation or non non-stationarity.
While the topic of non-stationarity is central the area of continual learning (see (Parisi al. 2019) for a recent review), the field is primarily concerned with preventing catastrophic forgetting (French 1999) when the environment or task changes during training (Li & Hoiem 2017: Schwarz et al. 2018). For non-stationary environments during agent deployment, the approach is typically to detect such changes and respond accordingly (Choi et al.) 2000; Da Silva et al. 2006 Doya et al., 2002). By contrast, we assume a stationary environment and investigate the impact of transient non-stationn non-stationarity, for example induced by an improving policy. We also show that intentionally forgetting the representation, but not the learned outputs, improve generalisation in this case.
Neural networks are used in deep RL to allow gencralisation across similar states (Sutton & Barto 2018). Several possibilities have been proposed to further improve generalisation, including to provide more diverse training environments (Tobin et al.] 2017). inject noise into the environment (Stulp al. 2011 Lee et al. 2020), incorporate inductive biases in the architecture (Kansky et al.) 2017). or regularise the network (Cobbe et al. 2019bb] [gl et al.) 2019: Liu et al.) 2019). While regularisation reduces overfitting, show in our experiments that this is insufficient to counter the negative effects
of non-stationarity, and that ITER can be complementary to other types of regularisation.
7 CONCLUSION
In this work, we investigate the impact of non-stationarity the generalisation performance of trained RI. agents. First, in several SI. experiments on the CIFAR-10 dataset, we confirm that non-stationarity can considerably degrade test performance while leaving training performance nearly unchanged. To explain this effect, we propose and experimentally support the legacy feature hypothesis that networks exhibit a tendency adopt, rather then forget, features learned earlier during training they are sufficiently relevant, though not necessarily optimal, for the new data. We also show that self-distillation, even without using the true training labels, improves performance on the test-data.
Many deep RI. algorithms induce similar transient con-stationaaity, for example due to gradually converging policy. Consequently, to improve generalisation in deep RI., we propose ITER which reduces the non-stationarity the agent networks experience during training. Our experimental results on the Multiroom and ProcGen benchmarks empirically support the benefits of ITER, indicating that transient non-stationarity has negative impact in deep RL.

---

Published as a conference paper at ICLR 2021
REFERENCES
Maxime Chovalier-Boisert and Lucas Willems, Minimalistic gridworld environment for openai gym.
https://github: www./o/wo///.a..eecaeccooooooeoe http://///gtttbb...ommmmmmeeee gym minigrid 2018.
Samuel PM Choi, Dit-Yan Yeung, and Nevin Lianwen Zhang. An environment model for nonstation- ary reinforcement learning. In Advances in neural information processing systems, pp. 987-993,
2000.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint arXiv: 1912.0091,,,
2019a. 20199.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, 2019b.
Wojeiech Czarnecki, Siddhant Jayakumar, Max Jaderberg. Leonard Hasenclever, Yee Whye Teh, Nicolas Heess, Simon Osindero, and Razvan Pascanu. Mix & match agent curricula for reinforce- ment learning. International Conference on Machine Learning, pp. 1087-1095, 2018.
Wojciech Marian Czamecki, Razvan Pascanu, Simon Osindero, Siddhant M Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. arXiv preprint arXiv:1902.02168. 2019.
Bruno C Da Silva, Eduardo W Basso, Ana LC Bazzan, and Paulo M Engel. Dealing with non- stationary environments using context detection. Proceedings of the 23rd international confer- ence on Machine learning, pp. 217-224, 2006.
Kenji Doya, Kazuyuki Samejima, Ken-ichi Katagiri, and Mitsuo Kawato. Multiple model-based reinforcement learning. Neural computation, 14(6):1347-1369 2002.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint
2018.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. arXiv preprint arXiv:2007.06700. 2020.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3 1335 1999.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pp. 1607-1616, 2018.
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning. International Conference on Learning Representations, 2018.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristies: Learning rate restarts, warmup and distillation. In International Conference Learning Representationss, 2019.
Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sebastien Racaniere, Theophane We- ber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, Timothy Lillicrap, and Victor Valdes. An investigation of model-free planning: boxoban levels. http://gghhbb..ommeeemmmmbobbbb/
2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint /503.0500..22 2015.
10

---

Published as a conference paper at ICLR 2021
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. In Advances in Neural Information Processing Systems, pp. 13956-13968, 2019.
Ken Kansky, Tom Silver, David A Mély, Mohamed Eldawy, Miguel Lázaro-Gredilla, Xinghua Lou, Nimrod Dorfman, Szymon Sidor, Scott Phoenix, and Dileep George. Schema networks: Zero-shot transfer with a generative causal model of intuitive physics. In Proceedings of the 34th International Conference on Machine Srarning-Volumee 70, pp. 1809-1818. JMLR. org, 2017.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Mea- suring catastrophic forgetting in neural networks. In Thirty-second AAAI conference on artificial intelligence, 2018.
Nikolaus Kriegeskorte and Rogier A Kievit. Representational geometry: integrating cognition, computation, and the brain. Trends in cognitive sciences, 4718)441-412, 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. International Conference on Learning Representations, 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-290) 2017.
Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell. Regularization matters in policy optimiza- tion. arXiv preprint 19X0099900001999 2019.
Hossein Mobahi, Mehrdad Farajtabar, and Peter L Bartleu. Self-distillation amplifies regularization in hilbert space. arXiv preprint 2000:2000..0715, 2020.
German Parisi, Ronald Kemker, Jose I. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2016, 2016.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, 2011.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint 15Xiv:15 2015.
Simon Schmitt, Jonathan Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.03835, 2018.
Max-Philipp B. Schrader.
gym-sokoban.
/ttps://githune cww
gym-sokoban 2018
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Jonathan Schwarz, Jelena Luketina, Wojeiech M Czarnecki. Agnieszka Yee whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning, arXiv preprint arXiv: a 1805.06370. 2018

---

Published as a conference paper at ICLR 2021
Freek Stulp, Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. Learning to grasp under uncertainty. In 2011 IEEE International Conference on Robotics and Automation, pp. 5703-5708.
IEEE, 2011.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4496 4506, 2017.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23-30. IEEE, 2017.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07933 2018a.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv: /804. /804.0683, 2018b.
Chenyang Zhao, Olivier Siguad, Freek Stulp, and Timothy Hospedales. Investigating generalisation in continuous deep reinforcement learning. arXiv preprint arXiv: 1902.00100, 2019.
12

---

Published as a conference paper at ICLR 2021
A
PSEUDO CODE
Algorithm 1: Pscudo-Code for parallel ITER
Input Length of initial RI training phase limits length of distillation phase Idistal Initialise k + O, policy π. x(8), value function (R)
// Normal RL training at the beginning
for linit steps do
B + collect trajectory data using (0) Update (0)) and ( using standard RL method using B
// Combine further RL training of (1),, ((k) with distillation of 7t (k+t))) vlk+1) while not converged do Initialise student policy π (b++1) and value function (k+1)
for ddsstill steps do
11
Ct, Ot f linear annealing to 0 over Edistiil steps
B <- collect trajectory data using :7(k) Update (*()) and (4) with standard RL method using B
Update (l++1) and (k+1) with eq. 11 using B, av, az (k) and (*k)
14
15
// Housekeeping
Discard π (K) and ((k)
16
17
Set k +
B SUPERVISED LEARNING
Numerical values of results presented in fig. The 'Rel' column shows error normalised hy error of unmodified dataset. The error on the test-data deteriorates worse than on the training data, not only in absolute, but also relativ terms.
Table 1:
Table Hyper-parametent used in the supervised learning experiment on CIFAR-10
Training Error in
Testing Error in %
Rel.
Rel.
Hyper-parameter
Value
Unmodified
0.17 + 0.09
1.0
14.8
0.70
Noisy Labels
0.19 + 0.09
1.13
16.8 = 0.70
1.14
SGD:
Learning rate
3 x 10 0.9
Wrong Labels
0.20 + 0.08
1.22
19.2
0.43
1.30
SGD: Momentum
Dataset Size
0.18 + 0.08
1.05
23.4 + 0.83
1.58
SGD: Weight decay
x 10-4
0.3
86
-066
Accuncy of Final Final Rege.
-1.2
82
-15
80
2.11
Nunyy Labck
Whong Libe's
227
Datuset Size
76
530
0.2
Fraction of correct data
Figure 6: Left: Same results as in fig. 5 (middle), but with the fraction of correct data points f on the x-Axis. Right: Multiroom example layout. The red agent needs to reach the green square, avoiding walls (grey) and passing through doors (blocks with coloured outline..
13

---

Published as a conference paper at ICLR 2021
Here provide additional training details and results for the supervised learning experiments performed on the CIFAR-10 dataset. We used a ResNet18 architecture without Batchnorm, hyper- parameters for the SGD optimiser given in table In table we provide exact numerical values for the results fig. [ We also provide values for the relative change in error rate due to the introduction of non-stationarities, for which the test-performance is also affected than the train performance.
In fig. 6 we show the same results in fig. 5 but here showing the f values used generate f on the X-Axis. The same 'dips' in performance are visible, however from this figure it is clear that Dataset Size experiences it for much smaller values f. which unsurprising, giving the missing influences of diverse input-data distribution.
Lastly, in fig.
we provide the individual training runs used generate fig. 5 middlee and fig. 6
Noisy Labels
Wrong Labels
Dataset Size
90
90
80
70
70
60
60
50
50
40
-30
20
-2
20
10
10
0.0
0.0
0.8
0.0
12
1.4
Epochs
163
Epochs
1c3
Epochs
lc3
85
Tel assurac
Test accuracy
accuracy
30
80
75
75
75
0.0
0.2
0.4
0.6
08 Epochs
1.0
122
14
1.6
0.0
0.2
0.4
0.6
088 Epochs
1.0
1.2
1.4
1.6
0.0
0.2
0.4
0.6
0.8 Epochs
1.0
122
1.4
16
163
Figure Individual training curves the data used fig. midddle and fig. same data as the top row, just 'zoomed in'.
The bottom row shows
B. MULTIROOM
In table 3 we show the hyper-parameters used for the Multiroom experiments which are shared between "PPO" and 'PPO+ITER'. We note that Multiroom environment uses the same modifi- cation that was used in (Igl et al. 2019) to make the environment fully observable. In the original environment, the agent only observed its immediate surrounding from an ego-centric perspective, thereby naturally generalising across various layouts. Instead, full observability introduces the need learn how generalise. Our network consists of three layer CNN With 16, 32 and filters respectively, followed by a fully connected layer of size 64. One max moolino layer is used after the first CNN layer. We use linit = 10" and laistill = 10? (see algorithm for duration of the initial RT. training phase and the following distillation phases.
B.2 BOXOBAN
For Boxoban, we re-use the same architecture and paramtttrseetess as for Multiroom, but with reduced learning rate (1.0e 04) in order stabilise training.
14

---

Published as a conference paper at ICLR 2021
Table 4: Hyper-parameters used for ProcGen
Table 3: Hyper- parameters used for Multiroom
Hyperparameter
Value
Hyper-parameter
Value
PPO: Armmppy Loss
0.01
PPO: AEntropy Loss
0.01
PPO: XTD
0.5
PPÓ: ÀTD
0.5
PPO: éClip
0.2
PPO: (Clip
0.2
PPO Epochs
PPO Epochs
PPO Nr. Minibatches
PPO Minibatch Size
2048
Parallel Environments
64
Parallel Environments
32
Frames per Env per Update
256
Frames per Env per Update
256
AGAE
0.95
AGAE
0.95
0.999
7
0.99
Adam: Learning rate
x 10
Adam: Learning rate
7 X 10
Adam:
X 10
Adam:
10-5
Adam: Weight decay
10
PPO-ITER Too short
51
PPO-ITER Too
PPO
Retum
-5
-10
-15
-20
0.4
Steps
1c9
Figure 8: Left: Boxoban example layout. The green agent needs to push (or pull) yellow boxes on the red targets, avoiding walls. Right: Additional Boxoban results showing consequences of choosing wrong distillation length, either too short or too long. Note that ITKR Tooo ahort uses the same distillating length as the results fig. 2 but continues with distillation past 0.5e9 steps.
In fig. show results for wrongly chosen distillation lengths. Teo short uses the same distillation length as the main results in fig. 2, but continues distilling until the end of training. Because after 0.5e9 steps the policy performance and complexity increase considerably, longer distillation period would be required. On the other hand, using distillation period twice as long from the start (Too long) leads to slower training. We did not experiment with increasing the distillation length over the course of training, since in our experiments earlier stopping of ITER was sufficient for optimal performance.
B.3 PROCGEN
In fig. 9 we show all results on the various ProcGen environment from which the summary plots in the main text (figs. 3 and 4 are computed. We use the same (small) IMPALA architecture as used by (Cobbe et al. 2019b). Training is done on 4 GPUs parallel. One GPU continuously evaluating the test performance, the other three are used for training. Their gradients are averaged at each update step. The hyper-parameters given in table are per GPU, The z-Axis fig, 9 shows the total number of consumed frames. i.e. 250 x 10t per training GPU. The distillation phase takes tdistul = 70 x 10 frames (again per GPU) and we linearly anneal a₂ from to 0 and av from 0.5 to 1. The values of ag and ay were chosen to reflect the relative weight between LPG and LTD in eq. (1) and no further tuning was done. The initial RL training phase takes tinst = 50 x 10 frames. The distillation length was chosen based on preliminary experiments BigFish by increasing its
15

---

Published as a conference paper at ICLR 2021
length in steps of 10 10t frames until no drop in training performance experienced when switching to a new student.
Due to the high computation costs of rumning experiments the ProcGen environment (4 GPUs for about 24h for each run). we decided to exclude environments from the original benchmark based results presented by Cobbe al. (2019a), figures 2 and 4. We excluded environments for two different reasons, either because the generalisation gap was small (Chaser, Miner, Leaper, Boss Fight, Fruitbot) because generalisation did improve all during training after a very short initial jump (CaveFlyer, Maze, Heist, Plunder, Coinrun), indicating that either was too hard, or a very simple policy already achieved reasonable performance.
Climber
Dodgchall
Ninja
12.0
10.0
Architacture
11.5
P2O-ITER
PPO
Toxt Performance
9.0
Tel Pal
10.5
10.0
ITER Sequenial
8.5
9.5
Ensluttioron on
Test levels
9.0
7.5
8.0
7.0
0
7
Steps
108
Steps
1e8
Steps
StarPilot
BigFish
FruitBot
20
Tost Porformance
Tent Petforuncc
15
10
1
2
5
6
7
2
3
6
7
8
3
4
5
8
Stops
le8
Steps
1e8
Stops
1e8
Figure All individual results on ProcGen. Shown is the mean and standard deviation across random seeds.
16


Reviews:
[{"review_id": "vF2ZzSgMQXW", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates an interesting problem that transient non-stationarity can affect the generalization of the neural network. This paper first conducts experiments on a supervised learning task to illustrate that transient non-stationarity can lead to degenerated performance on testing set. Then, the paper proposes an RL algorithm called ITER to avoid the negative impact of such non-stationarity.\n\nStrengths: \n+ This paper observes a novel problem that may appear in RL and designs a new algorithm to prevent such a problem.\n+ This paper investigates this problem through experiments on supervised learning tasks.\n\nWeaknesses:\n- The new algorithm ITER doubles the computational costs.\n- Experiments on Page 7 are trying to further illustrate the mechanism for such a phenomenon. However, this subsection is not very clear.\n- Experiment results on Procgen is fair but not significant.\n\nMinor comments:\n- In Section 2, the advantage function is defined as A^\\pi(s,a,s') but later used as A^\\pi(s,a) without additional explaination.\n- f in $\\mathcal{D}_{f,m}$ is not explained when it first appears. I guess it is the ratio of the modification.", "extra_scores": {}, "extra_text": {}}, {"review_id": "i5xYGGV8pO2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper deals with a relevant issue. The simplified supervised learning setting is a good way of looking at the issue of non-stationarity in isolation and it makes a compelling case that neural networks optimized by SGD can have generalization issues in settings where the data distribution changes over time, even after the data distribution converges. The solution proposed by the paper is simple and can be applied to most off-the-shelf RL algorithms. My main criticism would be that the hybrid objective feels rather ad-hoc and that the proposed method could use a bit more theoretical justification. Also, since the issue being tackled here is quite general to RL, a wider set of benchmarks and base-algorithms (in addition to PPO) would be necessary to get a better picture.\nRegardless, I believe the community would benefit from the inclusion of this paper. \n\n### Pros\n\n* The paper is well-written. The story is easy to follow and well-motivated.\n* The central problem that is being tackled is highly relevant.\n* I like that the problem is illustrated in a supervised-learning setting, which allows to investigate without the noise of a typical RL setup.\n* The proposed modification is simple and can be applied to a wide-range of algorithms.\n* The \"legacy feature\" effect is an interesting phenomenon and the additional experiments inspecting it are useful.\n\n### Cons\n\n* In Figure 5, the middle panel is a bit hard to parse. I don't know how to improve this but in its current state, too many variables are being presented at once.\n* While the method being presented is quite simple, it also seems a bit ad-hoc. For instance, it would be nice if the teacher-student distillation could be supported by some convergence guarantees, e.g. by showing that (under some circumstances) replacing the teacher by the student does not increase the loss.\n* The set of benchmarks considered in the paper is a good proof-of-concept, but additional experiments with different environments and base-algorithms is necessary to better judge the merits of the approach.", "extra_scores": {}, "extra_text": {}}, {"review_id": "JdNx_mlInBy", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents empirical evidence that non-stationarity data typical in deepRL settings can affect the intermediate representation of deep neural network and affect testing performance. The paper is easy to read and the authors provide experiments to support the their observations and claims. Overall I think this is a good paper and in the following I suggest some good to have additions.\n\n(1) The examples for the supervised learning setting clearly demonstrates the impact of non-stationary data. However, given that this is inspired by the problems under DRL setting, it will be interesting to do more analysis of this effect on some DRL tasks. For example, an analysis for offline RL might be a good setting to study this effect.\n\n(2)Imitation learning algorithm like Dagger might be another good example to demonstrate the effect of nonstationarity. The data under the Dagger setting is also changing overtime and it will be interesting to see how it affects the student policy.\n\n(3) The RL experiment is mainly done in the on policy (PPO) settings. Some experiments with off policy RL setting might be useful, and the effect of the non-stationarity might be more pronounced as well.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fbm5bPAM8LM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a mechanism of re-learning (ITER; iterative re-learning) to handle issue from non-stationarity in RL. Some gain of using the proposed method is experimentally presented. My major concern is the limited understanding of the non-stationarity issue and ITER. \n\nPros)\n- The authors propose a new approach to resolve the non-stationarity issue in RL. The intuition itself makes a sense as human often performs such relearning processes to escape from the local minimum. \n\n- Some gain of ITER is empirically demonstrated in various setting.\n\nCons)\n- Although the underlying intuition sounds delightful, the understanding on the non-stationarity is insufficient. In particular, the authors design a setup of \"supervised\" learning to show importance of non-stationarity. However, it is a weak evidence on the importance of non-stationarity in \"reinforcement\" learning. Indeed, the gain of using ITER is not significant.\n\n- The authors do not clearly define the generalization which the relearning aims to improve. It may depend on the neural network architecture and environment and so on. However, presuming that the paper reports the best gain possible, it is hard to accept that ITER always improves generalization.\n\n- I have a concern on the little gain, which is shown by the comparison between PPO vs. PPO+ITER with the same hyperparameters. A fair comparison may use the best hyerparameters for each of PPO and PPO+ITER. Or, at least, there need comparisons with different hyperparameters to claim consistent improvement.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a substantial contribution in identifying novel questions/issues, as this paper certainly does. Neither I nor the reviewers have seen this issue of transient non-stationary before, and the authors make a compelling case for it, especially in the supervised setting with the CIFAR experiments. It is less compelling through the RL experiments. As such, this paper is likely to inspire new work within the field. To me, Figure 1 is the most interesting aspect of the whole paper.

The initial approach by the authors is questionable in its effectiveness, and is likely to be improved by others in the future. Some of the results in Figure 3 are questionable, especially when you look at the individual curves in Figure 8. So overall, this means that the authors have identified a truly novel issue, and proposed an initial method that is just okay.  They've done a nice job investigating this in a supervised setting, and need to push further in the RL setting.

The question is whether the novel contribution of the problem outweighs that the algorithm and its evaluation could use improvement.  The reviewers debated this in the discussion, with points on both sides, but the novelty of the question/issue (even if the investigation could use work) is likely to inspire further work in this direction.

Other notes:
The authors could have evaluated the (impractical) version of their algorithm proposed in the first paragraph of Section 4.2. This would inform 1) whether their parallel training approximation is close to the optimal algorithm, and 2) whether the optimal (impractical) algorithm is capable of improving generalization significantly. If the latter is true, it would leave open a huge avenue of investigation to find better approximate solutions.
Decision: Accept (Poster)