Rethinking Value Function Learning for Generalization in Reinforcement Learning
Authors: Seungyong Moon
JunYeong Lee
Hyun Oh Song
Venue: NeurIPS 2022 Accept (2022)
Keywords: RL generalization

Abstract: Our work focuses on training RL agents on multiple visually diverse environments to improve observational generalization performance. In prior methods, policy and value networks are separately optimized using a disjoint network architecture to avoid interference and obtain a more accurate value function. We identify that a value network in the multi-environment setting is more challenging to optimize and prone to memorizing the training data than in the conventional single-environment setting. In addition, we find that appropriate regularization on the value network is necessary to improve both training and test performance. To this end, we propose Delayed-Critic Policy Gradient (DCPG), a policy gradient algorithm that implicitly penalizes value estimates by optimizing the value network less frequently with more training data than the policy network. This can be implemented using a single unified network architecture. Furthermore, we introduce a simple self-supervised task that learns the forward and inverse dynamics of environments using a single discriminator, which can be jointly optimized with the value network. Our proposed algorithms significantly improve observational generalization performance and sample efficiency on the Procgen Benchmark.

Rethinking Value Function Learning for Generalization in Reinforcement Learning
Seungyong Moon 1.2 Jun'Yeong Lee Hyun Oh Song 1.23+ 12.3+
Seoul National University, Neeurl Processing Research Center, DDepMetrics [symoon11 1,, mas mascheroni99, hyunoh)0mmml 0mllab snu. ac kr
Abstract
Our work focuses on training RL agents multiple visually diverse environments to improve observational generalization performance. In prior methods, policy and value networks separately optimized using disjoint network architecture avoid interference and obtain a more accurate value function. We identify that a value network the multi-environment setting is more challenging optimize and prone to memorizing the training data than in the conventional single-environment setting. addition, necessary to improve both training and test performance. To this end, we propose Delayed- -rattc Policy Gradient (DCPG), a policy gradient algorithm that implicitly penalizes value estimates by optimizing the value network less frequently with more training data than the policy network. This can be implemented using a single unified network architecture. Furthermore, we introduce simple self-supervised task that learns the forward and inverse dynamics of environments using single discriminator, which can be jointly optimized with the value network. Our proposed algorithms significantly improve observational generalization performance and sample efficiency on the Procgen Benchmark.
find that appropriate regularization on the value network is
Introduction
In recent years, deep reinforcement learning (RL) has achieved remarkable success various domains, such robotic controls and games [27 19 32 To apply RL algorithms to more practical scenarios, such as autonomous vehicles or healthcare systems, they should be robust against the non-stationarity of real-world environments and capable of performing well unseen situations during deployment. However, current state-of-the-aar RI. algorithms often fail to generalize to unseen test environments with visual variations, even if they achieve high performance in their training environments [14 9
This problem is referred to as observational overfitting [36].
Training RL agents finite number of visually diverse environments and testing them on unseen environments is the standard protocol for evaluating observational generalization in RL [10] Several methods have attempted to improve generalization in this framework by adopting the regularization techniques that originate from supervised learning or training robust state representations via self- supervised learning 9 20 30 26]. However, these methods have mainly focused on developing new auxiliary objectives on the existing RI. algorithms intended for the conventional single-enviromment setting such as PPO 334 Some recent works have investigated the interference between policy and value function optimization arising from the multiple training environments and proposed new training schemes that decouple the policy and value network training with a separate network architecture obtain an accurate value function
" *Corresponiing author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this paper, we argue that learning an accurate value function on multiple training environments is more challenging than on single training environment and requires sufficient regularization. We demonstrate that value network trained on multiple environments is more likely to memorize the training data and cannot generalize to unvisited states within the training environments, which can be detrimental to not only training performance but also test performance on unseen environments. In addition, we find that regularization techniques that penalize large estimates of the value network, originally developed for preventing memorization in the single-environment setting, are also beneficial for improving both training and test performance in the multi-environment setting. However, this benefit comes at the cost of premature convergence, which hinders further performance enhancement.
To address this, we propose a new model-free policy gradient algorithm named Delayed Delayed-Critic Critic Policy Gradient (DCPG), which trains the value network with lower update frequency but with more training data than the policy network. We find that the value network with delayed updates suffers less from the memorization problem and significantly improves training and test performance. In addition, we demonstrate that provides better state representations to the policy network using single unified network architecture, unlike the prior methods. Moreover, we introduce simple self-supervised task that learns the forward and inverse dynamics of environments using a single discriminator on top of DCPG. Our algorithms achieve state-of-the-ar observational generalization performance and sample efficiency compared to prior model-free methods on the Procgen benchmark [10].
Preliminaries
2.1
Observational Generalization in RL
We consider a collection of environments M formulated as Markov Decision Processes (MDPs). Each environment m E M is described tuple (Sm.A,A Tmm I'm Pm: 7), where 8m is the image-based state space, A is the action space shared across all environments, Tmm : Sm X A + P(S..) is the transition function, I' S.. A R is the reward function, Pm is the initial state distribution, and E [0, 1] is the discount factor. We assume that the state space has visual variations between different environments. While the transition and reward functions are defined as specific to an environment, assume that they exhibit some common structures across all environments. A policy π : S + P(AA) is trained on finite number of training environments Mtrain [m:)?_ where S is the set of all possible states in M. Our goal is to learn a generalizable policy that maximizes the expected return on unseen test environments Mrest = M Mtrain-
In this paper, we utilize the Procgen benchmark as a testbed for observational generalization [0] It is a collection of 16 video games with high diversity comparable to the ALE benchmark [3]. Each game consists of procedurally generated environment instances with visually different layouts, backgrounds, and game entities (e.g., the spawn locations and times for enemies), also called levels. The standard evaluation protocol on the Procgen benchmark is to train a policy on a finite set of training levels and evaluate its performance held-out test levels [10].
2.2 Proximal Policy Optimization
Proximal Policy Optimization (PPO) is a powerful model-free policy gradient algorithm that learns a policy тo and value function Va parameterized by deep neural networks [34]. For training, PPO first collects trajectories 7 using the old policy network mewu right before the update. Then, the policy network is trained with the collected trajectories for several epochs to maximize the following clipped surrogate policy objective Jg designed to constrain the size of policy update:
Te (at 8t) Tanaaa (at, 8t)
Te(ac St)
1 A. }.
(0) ES
min
Åe, clip
TT (ar 88)
where A, an estimate of the advantage function at timestep t. Concurrently, the value network is trained with the collected trajectories to minimize the following value objective Jy:
1 1
(Vo(s)) Rt )
Jy (o) E Egy
where RG = A + Vo())) is the value function target. It is used to compute the advantage estimates via generalized advantage estimator (GAE) (33].

---

Policy phase (=(
(v)
(x)
(Ø)
J+(0)
Cv
Aux phase
C. (0)
Jaux (0)
(v(
(.)
Jv(0)
Jy(0)
Policy Tal (: (8)
Value (s(s)
Policy T
Aux value (*())
Value (a(+)
Policy m
Value
Dynamics
Vo(s)
a,
Policy encoder 0
Value encoder Ø
Encoder
Encoder
PPO network
(b) PPG network
(c) DDCPG network
Figure 1: Network architectures for PPO, PPG, and DDCPG. The objectives Jr, Jv, Jaux. and J, denote the policy, value, auxiliary value, and dynamics objectives, respectively. The regularizers Cx and Cv denote the policy and value regularizers, respectively. The blue and red terms represent optimization problems during the policy and auxiliary phases, respectively.
In practice, the policy and value networks are jointly optimized with shared parameters (i.e., 0 o), especially in image-based based RL [13] For example, they can be implemented using a shared encoder followed by separate linear heads, as shown in Figure Ta Sharing parameters advantageous that representations learned by each objective can be beneficial to the other. It also reduces memory costs and accelerates training time. However, a shared network architecture complicates the optimization as single encoder should be optimized over multiple objectives whose gradients may have varying scales and directions. also constrains the policy and value networks to be optimized under the same training hyperparameter setting, such as batch size and the number of epochs, severely limiting the flexibility of PPO.
2.3 Phasic Policy Gradient
Phasic Policy Gradient (PPG) is an algorithm built upon PPO that significantly improves observational generalization by addressing the problems of sharing parameters [[]]. More specifically, PPG employs separate encoders for the policy and value networks, as shown in Figure lb In addition, it introduces an auxiliary value head Vo top of the policy encoder in order distill useful representations from the value network into the encoder. For training, PPG alternates between policy and auxiliary phases, During the policy phase, which is repeated N,, times, the policy and value networks are trained with newly-collected trajectories to optimize the policy and value objectives from PPO, respectively. Then, all states and value function targets in the trajectories are stored in a buffer During the auxiliary phase, the auxiliary value head and the policy network are jointly trained with all data in the buffer to optimize the following auxiliary value objective Jaux and policy regularizer C+:
Ri) R (x) (Vo(s))-Rt)+
Jaux (0) = E Exj=B 2
Egg=B
84) || To( &())),
where Taaa is the policy network right before the auxiliary phase and DKL denotes the KL divergence. In other words, the value network is distilled into the policy encoder while maintaining the outputs of the policy network unchanged. Moreover, the value network is additionally trained with all data the buffer to optimize the value objective from PPO to obtain a more accurate value function. It is worth noting that the training data size in the auxiliary phase N₂ times larger than the policy phase. has been claimed that the distillation of a better-trained value network with separate architecture and the additional training for more accurate value network can improve observational generalization performance and sample efficiency |
3

---

PPG
DCPG
BigFish
Chaser
Climber
StarPilot
0.6
0.6
0.4
0.4
Soffness
0.4
Stifneis
Stillness
Siffness
0.4
0.2
0.2
0.2
0
50
200
200
200
10
# traiming levels
training levels
training levels
# training levels
Figure 2: Average stiffness of value networks for PPG and DCPG on 4 Procgen games while varying the number of training levels.
3 Motivation
3.1 Difficulty of Training Value Network Multiple Training Environments
We begin by investigating the difficulty of obtaining an accurate value network across multiple training environments. Indeed, learning value network that better approximates the value function on the given training environments can result in improved training performance (37]. However, even in simple setting where an agent is trained on single environment, it has been shown that a value network is likely to memorize the training data and unable to extrapolate well to unseen states even in the same training environment [23 12 15]. This problem can be exacerbated when the number of training environments increases. Intuitively, given the fixed number of environment steps, the value network will be provided fewer training samples per environment and rely more on memorization.
To corroborate this claim, we measure the stiffness of the value network between states (s, 8') while varying the number of training environments [16,6] which is defined by
V VoJv(ó; Vo Jo
s').s')
(o: s)) l2l (v s')ll2
Low stiffness indicates that updating the network parameters toward minimizing the value objective for one state will have a negative effect the minimization of the value objective for other states [6]. That is, the value network is less able to adjust its parameter to predict the true value function across different states and instead tends memorize only the states it has encountered. More specifically, we train PPG agents on the Procgen games while increasing the number of training levels from 200 and compute the average stiffness across all state pairs in mini-batch of size 2 (=16,384) throughout training. The detailed experimental settings and results can be found in Appendix A.
The green lines in Figure 2 show that the stiffness of the value network decreases as the number of training environments increases, as expected. implies that the value network trained on multiple environments is more likely to memorize the training data and cannot accurately predict the values of unvisited states from the training environments. This memorization problem brings us to train a value network with sufficient regularization.
3.2 Training Value Network with Explicit Regularization
Next, we examine the effectiveness of value network regularization in the nulti-environment setting. We consider applying two existing regularization techniques developed to prevent the memorization problem in the single-enviromment setting, especially when training data is limited. The first method discount regularization (DR), which trains a myopic value network with lower discount factor / 28 The second method is activation regularization (AR), which optimizes value network with La
penalty on its outputs:
( = E Eggor
2 Vo(st) - Rd + a V. (8)))
2

---

PPG
PPG+DR
PPG+AR
True
Predicted
BigFish, Train
BigFish, Test
PPG+DR. BigFish
PPG+AR, BigFish
30
30
AmAn
Average return
Value
Average
0
0
10
20
25
10
20
25
10 15
25
0
10
20
25
Environment step (10")
Environment (10")
Environment stop (10")
Environment step (10")
(a) Average training and test returns.
(b) Truc and predicted values.
Figure 3: (a) Training and test performance curves of PPG, PPG+DR, and PPG+AR BigFish. (b) True and predicted values measured at the initial states of training environments for PPG+DR and PPG+AR on BigFish. The mean is computed over 10 different runs.
where a >> 0 the regularization coefficient 3 We train PPG agents with each of these two methods using 200 training levels on the Procgen games. We reduce the discount factor from 0.999 to Y 0.995 for PPG+DR and use a 0.05 for PPG+AR. We measure the average training and test returns to evaluate the training performance and its transferability to unseen test environments.
As shown in Figure Sal the value network regularization improves the training and test performance of PPG on BigFish to some extent. implies that explicitly suppressing the value network also helps mitigate memorization in the multi-environment setting. We also observe that these regularization methods improve the training and test performance across all Procgen games on average. The detailed experimental setting and results can be found in Appendix B.
Despite its effectiveness, explicit value network regularization can lead to a suboptimal solution as the number of environment steps increases. Figure 3b shows the true and predicted values measured at the initial states of the training environments for PPG+DR and PPG+AR on BigFish. The predicted values with explicit regularization reach plateau too quickly, suggesting that excessive regularization later hinders the value network from learning an accurate value function. This motivates us to develop more flexible regularization method that boosts training and test performance while allowing the value network to converge to true values.
4 Delayed-Critic Policy Gradient
In this section, we present novel model-free policy gradient algorithm called Delayed- -Citic Policy Gradient (DCPG), which effectively addresses the memorization problem of the value network in simple and flexible manner. The key idea is that the value network should be optimized with larger amount of training data to avoid memorizing a small number of recently visited states, based on the stiffness analysis in Section 3.I Furthermore, the value network should be optimized with a delay compared to the policy network to implicitly suppress the value estimate, based on the regularization
analysis in Section 3.2
4.1
Algorithm
DCPG follows similar procedure to PPG by alternating policy and auxiliary phases. Still, it employs a shared network architecture the same way as PPO and does not require any additional auxiliary head, shown in Figure Ic During the policy phase, which occurs more frequently but with less training data than the auxiliary phase, the policy network is trained with newly-collected trajectories optimize the policy objective from PPO. In contrast, the value network is constrained preserve its outputs by optimizing the following value regularizer Cv:
(v) = Esyor
2 (Vo(se) - Volu Vonn (8t)))
5

---

True
Predicted
DCPG, BigFish
DCPG, Chaser
DCPG, Climber
DCPG, Dodgeball
10
Value
Value
Value
6
0
5
15 20 25
10
15
25
0
10
20
25
10
20
25
Environment step (10")
Environment stop (10")
Environment step (10%)
Environment (10")
Figure 4: True and predicted values measured at the initial states of training environments for DCPG agents on 4 Procgen games, The mean is computed over different runs.
where Voss is the old value network right before the policy phase. Then, all states and value function targets in the trajectories are stored a buffer.
During the auxiliary phase, the value network is trained with a larger number of data in the buffer to optimize the value objective from PPO. In contrast, the policy network constrained preserve its outputs by optimizing the policy regularizer. Note that since the value network shares the same encoder as the policy network, optimizing the value objective directly plays the role representation learning for the policy network. Thus, unlike PPG, DCPG does not require an additional auxiliary head for feature distillation. The overall procedure of DCPG can be found in Algorithm T
Algorithm 1 Dynamics-aware Delayed-Critic Policy Gradient (DDCPG)
Require: Policy network Ts. value network Va, dynamics discriminator fa
1: for phase = .. do
2: Initialize huffer 1
3: for iter 1,2,..., . do
Policy phase
4: Sample trajectories 7 using Te and compute value function target Hi for each state st E
5: for epoch = 1,2,... ... E. do
Optimize policy objective (0) 7: end for
(0) and value regularizer Cv (0) with
8: Add (St, oc, R₂) to B
9:
end for
10: for iter 1, 1,2..... Euux do
Auxiliary phase
Optimize value objective Jv (0), dynamics objective (0), and policy regularizer C+(0) with B
12: end for
13:
end for
4.2
Value Network Analysis of Delayed Critic Update
The delayed critic update in DCPG can function as implicit regularization of the value network. Since the policy improvement is not immediately reflected in the value network, it encourages smaller value estimates than the true values. To validate this claim, we train DCPG agents using 200 training levels on the Procgen games and measure the true and predicted values at the initial states of the training levels. Figure 4 shows the value estimates are kept lower than the true values at the early stage of training but recover the true values as training progresses. More results can be found Appendix C.
We also evaluate the effectiveness of the delayed critic update in mitigating the memorization problem by measuring the stiffness of the value network as done Section 3.1 The blue lines in Figure ssow that DCPG has higher stiffness than PPG, implying that the value network of DCPG is more robust against memorization. We conclude that the delayed critic update serves as an effective regularizer for the value network. More results can be found in Appendix A.

---

4.3 Learning Forward and Inverse Dynamics with Single Discriminator
In the context of multi-task learning, learning an auxiliary task can operate as good regularizer that prevents memorization in the main task [1]. Motivated by this, we consider learning an auxiliary task using the representations from the encoder and jointly learning the value network with the auxiliary objective. Suppose the buffer B used in the auxiliary phase contains transition tuple (st, dt, S(+1) Since the transition function from each environment assumed to have the same underlying structure, it is natural to train neural network that models the forward dynamics of environments in the latent space of the encoder. Concretely, a dynamics head fe takes the embedding of the current state 8t and the current action as inputs and predicts the embedding of the next state s;, However, it is well known that a forward dynamics model using a neural network tends to overfit a small number of training data and make poor predictions unseen states (8) To address this, we consider learning the forward dynamics by training a discriminator that determines whether the next state is valid, given the current state and action:
fo(s,, ae. St+1) = fo at $6+1) = 0, S E B {s4+1}-
Such a discriminator might discard information about the action to distinguish whether a transition is valid. For example, the discriminator can predict the valid transition by using only the proximity of the current and next states (see Appendix F for more details). To avoid this, we train the discriminator to jointly distinguish whether the current action is valid, given the current and next states (i.e. inverse dynamics):
Sa(St, Ot. 81+1) = 1, fo(Se; âe, 81+1) = 0, âe E B {at). Finally, we optimize the following dynamics objective during the auxiliary phase:
(y(0) B log (fo(se, ae St+1)) + log (1 fo ae 81+1))
+ log (1 fo âr, 8t+ [e)])]],
where 81+1 U( {st+1)) à ~ u( (ae)). and rj > 0 is the coefficient for the inverse dynamics. We name the resulting algorithm Dynamics- aware Delayed- CrCriii Policy Gradient (DDCPG), The network architecture and algorithm for DDCPG are shown in Figure Ic and Algorithm I respectively
(the differences with DCPG are marked in eyan).
5 Experiments
5.1 Experimental Settings
We evaluate the observational generalization performance of our methods on the Procgen benchmark. We the "easy" difficulty mode, which most prior works have focused on. We train agents 200 training levels generated by seeds from 0 to 200 for 25M environment steps and test them on held-out test levels, following the practice in Cobbe et al. [10] We measure the average return of 100 test episodes and report its mean and standard deviation over 10 runs with different initialization.
We compare methods with PPO and 4 other baselines that use PPO as the backbone algorithm: UCB-DrAC, PLR, PPG, and DAAC 33 22 I1 22). UCB-DrAt is a data augmentation algorithm for training policy and value networks to be robust against various transformations. PLR is sampling algorithm for the procedural generation that selects the next training level based on its future learning potential. PPG is the previous state-of- of-the-art method on the Procgen benchmark. DAAC is motivated by PPG and distills the advantage function into the policy network instead. For each method, we use the same set of hyperparameters across all games, following the standard protocol in the ALE and Procgen benchmarks [2 10] For our methods, we use the same hyperparameter setting as PPG for fair comparison. To compare the performance of each method with single score, we calculate the PPO-normalized score averaged over all Procgen games, which is computed by dividing the average return of each method by PPO, following the practice in Jiang al. [22], Raileanu al. [30). For the implementation details and hyperparameters, please refer to Appendix D. The code can be found at httpp://gii thub. wtw.tttt.boooooeeeeeoommmttttttt snu- mllmbbllbbbb aallaabbDCCCCC
5.2 Observational Generalization Performance Procgen Benchmark
Table T the average test returns of each method on all 16 Procgen games. DCPG significantly outperforms all the baselines, yiclding a 24%p improvement on average compared to the previous
7

---

PPO
DAAC
PPG
DCPG
DDCPG
BigFish
Chaser
Climber
20
Average
10
20
25
10
20
25
10
20
25
Environment step (10")
Environment step (10")
Environment step (10")
Dodgeball
Miner
StarPilot
40
Averge
Average
Average
20
10
20
25
0
10
20
25
0
10
20
25
Environment step (10%)
Enviromment step (10%)
Environment step (10%)
Figure 5: Test performance curves of each method on 6 Procgen games. Each agent is trained on 200 training levels for 25M environment steps and evaluated on 100 unseen test levels. The mean and standard deviation are computed over 10 different runs.
Table 1: Average test returns of each method on all 16 Procgen games. Each agent is trained on 200 training levels for 25M environment steps and evaluated on 100 unseen test levels. The mean and standard deviation are computed over 10 different runs.
Environnent
PPO
UCB-DrAC
PLR
DAAC
PPG
DCPG
DDCPG
BigFish
3,4 1.0
6.6 2.5
10.8 2.5
15.3 : 3.0
16.4
23.1 4 2.3
25.9 = 2.3
BossHight
7.4 0.7
12+10 1.0
88 = 0.7
10
10.2 0.4
106:0.5
CaveFlyer
0.6
0.
6.7
5.1
0.7
0.4
6.3
Chaser
5,5 0.8
6.7
7.
6.3
8.9
9.7
9. 0.6
Climber
0.6
6.2
6.=005
7.4
8.1
8.5
9.
CoinRun
0.
8.4
8.
9.
8.9
8.5
8.4
Dodgeball
1,9
1.
22
3.2
3.6
6.7
9.0
FruitBot
275
2.79
23
2.
2..1
29.0
28.8
Heist
26 0.6
1.+
3.6
3.5
2.+
3.3
4,4
Jumper
5.7
5.6
=4
5.8
6.0
6.2
6.4
Leaper
-3-
Maze
0.7
1.1
56=
5.6
5.3
5.7
6.5 0.6
Miner
9.
1.71
9.
4.4
7.6
9.1
10.6
Ninja
0.7
0.7
6.7
6.7
6.6
6.6
Plunder
5.0
1.1
8.3
5.6
13.7
13.7
12.9 2.5
StarPilot
25.8
2.0
27.0 32
24.9
3.6
35,9 + 3.3
4488 2.6
46.1 4 2.1
46.6 4.1
PPO-norm score (%)
100.0
120.7 4 10.7
130.0 6.0
136.7 7.6
160.3
184.5 +
5.2
202.2 10.2
state-of-the--rt PPG. Adding dynamics learning to DCPG can further improve the test performance with an 18%p increase in the PPO-normalized score. We also provide the average training returns on all Procgen games in Appendix E, showing that our methods achieve better sample efficiency in the multi-environment setting. Furthermore, we evaluate the performance of our methods using the Min-Max normalized score and report mean, median, and interquartile mean (IQM) scores in Appendix G, following the recent practice in Agarwal et al. (2]. We observe that the performance improvements of our methods statistically significant for all the metrics consider.
Figure shows the test performance curves of each method on Proegen games throughout training. Our methods achieve superior final test performance and acquire generalization ability with fewer environment interactions. For example, our methods reach the final performance of PPG using only
Results reported in the original paper different hyperparameter setting for each game.

---

PPG
Separate DCPG
DCPG
BigFish
Chaser
Climber
Dodgeball
10
10
Average return
Average roturn
Averago return
20
Average
10
23
0
0
10
20
25
10
20
25
5
15
20
15
25
Environment step (10%)
Environment step (10")
Environment step (10")
Environment stop (10")
Figure 6: Test performance curves of PPG. Separate DCPG, and DCPG 4 Procgen games. Each agent is trained on 200 training levels for 25M environment steps and evaluated on 100 unseen test levels. The mean and standard deviation are computed over 10 different runs.
20% of total environment steps on BigFish and Climber and 10% on Dodgeball and Miner. More experimental results can be found in Appendix E.
5.3 Ablation Study for Delayed Critic Update
The delayed eritic update not only alleviates the memorization problem but also provides better state representations for generalization to the policy network. To validate this, we train DCPG agents with an additional value network V6 using a separate encoder (Separate DCPG). The separate value network Vo is aggressively optimized in the same way as PPG and only used for calculating advantages in the policy objective. The original value network Vo is only used for learning representations for the policy network with delayed updates. Figure 6 shows that Separate DCPG achieves better test performance than PPG, implying that the delayed critic update produces more generalizable representations. For the detailed experimental settings and results, please refer to Appendix F.
5.4 Ablation Study for Dynamics Learning
We conduct an ablation study show the effectiveness training forward and inverse dynamics using single discriminator. We train DCPG agents with only forward dynamics learning (DCPG+F) and only inverse dynamics learning (DCPG+I). In addition, we train DCPG agents with forward and inverse dynamics learning using two separate discriminators (DCPG+FI). Table Taole that DDCPG, which jointly optimizes the forward and inverse dynamics with single discriminator, achieves better test performance than the other methods with dynamics learning. Note that inverse dynamics provides performance gain only when jointly trained with forward dynamics using a single discriminator, We provide the detailed experimental settings and results in Appendix F.
Table 2:
of each method with dynamics learning all 16 Procgen games. Each agent is trained on 200 training levels for 25M environment steps and evaluated on 100 unseen test levels. The mean and standard deviation are computed over 10 different runs.
DCPG
DCPG+F
DCPG+1
DCPG+FI
DDCPG
PPO-norm score (%)
184.5 + 5.2
195.9 + 7.7 185.5 + 6.5 194.6 + 6.2 202.2 + 10.2
Related Works
Observational generalization in RL One prominent approach for improving observational gener- alization in model-free RL is to employ regularization techniques developed for supervised learning. such as batch normalization, weight regularization, and information bottleneck [4 20 Several works adopt data augmentation strategies commonly used in computer vision to address the general- ization problem 38 For example, UCB-DrAC proposes method that automatically finds an effective augmentation and introduces new regularization terms to learn robust state representations

---

under various imgge transformations [30]. Recently, it has been shown that MuZero Reanalyse, the state-of-the -he-t art model-based RL algorithm, achieves better observational generalization performance than model-free approaches, though it requires a larger network architecture 4
Another line of work uses self-supervised learning to learn invariant representations that can generalize to unseen environments, Some methods attempt to learn state representations that capture long-term behavior proximity between states using bisimulation or policy similarity metrics discard irrelevant information for generalization 40 26 The idea of learning dynamics as self-supervised task, originally developed for sample efficiency, has also been used to improve observational generalization DIM learns a state representation that can predict the representations of successive timesteps
[8 35 using supervseee learning [25]. It has also been tested that predicting future state representations conditioned on the current state and action improves the generalization performance of MuZero [4]. Our work aims to enhance dynamics learning further by jointly modeling the forward and inverse dynamics objectives using a single discriminator.
PPG is most closely related to our work, which proposes better training algorithm to improve both observational generalization and sample efficiency in the multi-environment setting [II. It trains the policy and value networks independently using a separate architecture and different levels of sample reuse to avoid interference and obtain a better-trained value network. DAAC also uses decoupled policy and value networks for a more accurate value function but distills the advantage function into the policy encoder under the assumption that it is less prone to overfitting to euvironment-spectin features than the value function [29]. In contrast, we focus on regularizing the value network while obtaining better representation, which distinguishes our work from the prior methods.
Learning value networks with regularization Discount regularization is one common regulariza- tion method to mitigate overfitting in the value network 7 A lower discount factor can lead to better
performance in approximated dynamic programming when the approximation error is large (8 It is also helpful for preventing overfitting when dealing with limited data in POMDPs [7]. Activation regularization penalizes the outputs of the value network and has a regularization effect similar to that of discount regularization [3]. It has been shown that activation regularization is equivalent to discount regularization in batch TD learning and also effective in online RL
7 Conclusion
We have investigated the diffieulty of learning accurate value network in the multi-environment setting and shown that suppressing the value network with appropriate regularization can be helpful to improve both the training and test performance. Based on this observation, we propose Delayed-Critte Policy Gradient (DCPG), where the value network is implicitly regularized to have lower predicted values at the early stage of training with delayed updates compared to the policy network. We find that the delayed value update prevents the memorization of training data and produces more generalizable representations that can be extended to unseen test environments. Furthermore, we introduce simple self-supervised task that jointly learns forward and inverse dynamics using single discriminator, which can be casily combined with DCPG. Our algorithms exhibit state-of-the performance in observational generalization on the challenging Procgen benchmark.
Acknowledgements
This work was supported in part by Samsung Advanced Institute of Technology, Samsung Electronics Co., Ltd., Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-01371, Development of brain-inspired AI with human-like intelligence, No. 2020-0-00882, (SW STAR LAB) Development of deployable learning intelligence via self- eulf-uustaie and trustworthy machine learning, and No. 2022-0-0044 -024000 Development of Training and Inference Methods for Goal-Oriented Artificial Intelligence Agents), and a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: H121C1074). This material is based upon work supported by the Air Force Office of Scientific Research under award number FA2386-22 -4010. Hyun Oh Song is the corresponding author.
10

---

References
[1] Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Mare Bellemare. Con- trastive behavioral similarity embeddings for generalization in reinforcement learning, In ICLR, 2021.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In NeurIPS, 2021.
[3] Ron Amit, Ron Meir, and Kamil Ciosek. Discount factor regularizer in reinforcement learning. In ICML, 2020.
[4] Ankesh Anand, Jacob C Walker, Yazhe Li, Eszter Vértes, Julian Schrittwieser, Sherjil Ozair, Theophane Weber, and Jessica Hamrick. Procedural generalization by planning with self- supervised world models. In ICLR, 2022.
[5] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environment: evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-277 2013.
[6] Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. In ICML, 2020.
[7] D. P. Bertsekas and J. N. Tsitsiklis. Neuro-dynamic programming. pranoodmamicrorrammnnnn Athena Scientific, Belmont, MA, 1996.
[8] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In NeurIPS, 2018.
[9] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In ICML, 2019.
[10] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning, In ICML, 2020.
[11] Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In ICML, 2021.
[12] Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G Bellemare, and David Silver. The value-improvement path: Towards better representations for reinforcement learning. In AAAI, 2021.
[13] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: :/gi thub. www/ http://peaaaaallnnnncce 2017.
[14] Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:1810.0000 2018.
[15] Yannis Flet-Berliac, reda ouhamma. adalric-ambryn maillard, and Philippe Preux. Learning value functions in deep policy gradients using residual variance. In ICLR, 2021.
[16] Stanislav Fort, Pawel Krzysztof Nowak, Stanislaw Jastrzebski, and Srini Narayanan. Stiffness: new perspective generalization in neural networks. arXiv preprint 1rXiv:1901. 999999 2019.
[17] Vincent François-Lavet, Guillaume Rabusseau, Joelle Pineau, Damien Ernst, and Raphael Fonteneau. On overfitting and asymptotic bias in batch reinforcement learning with partial observability. Journal of Artificial Intelligence Research, 65:1-30, 2019.
[18] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. In ICML, 2019.
[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018.

---

[20] Maximilian Igl, Kamil Ciosek. Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmannn Generalization in reinforcement learning with selective noise injection and information bottleneck. In NeurlPS, 2019.
[21] Andrew Blyss, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look at deep policy gradients. In ICLR, 2020.
[22] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In ICML, 2021.
[23] J Zico Kolter and Andrew Y Ng. Regularization and feature selection in least-squares temporal difference learning. In ICML, 2009.
[24] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In NeurIPS, 2020.
[25] Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. In NeurIPS, 2020.
[26] Bogdan Mazoure, Ahmed M Ahmed, R Devon Hjelm, Andrey Kolobov, and Patrick MacAlpine. Cross-trajectory representation learning for zero-shot generalization in RL. In ICLR, 2022.
[27] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-555 -33,,,,,,,,,,, 2015.
[28] Marek Petrik and Bruno Scherrer. Biasing approximate dynamic programming with lower discount factor. In NeurlPS, 2008.
[29] Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforce- ment learning. In ICML, 2021.
[30] Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. In NewrIPS, 2021.
[31] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
[32] Julian Schritwwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-699 2020.
[33] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In ICLR, 2016.
[34] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint 170iv:177.00644 2017.
[35] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self- preffcrreietive representations. In ICLR, 2021.
[36] Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. In ICLR, 2020.
[37] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NewrlPS, 1999.
[38] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforeement learning from pixels. In ICLR, 2021.
[39] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In AAAI, 2021.
12

---

[40] Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforeement learning without reconstruction. In ICLR. 2021.
[41] Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning. arXiv preprint arXiv:7804.06892 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We clearly reflect the main claims in Section and
Section
(b) Did you describe the limitations of your work? [N/A]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines and checked our paper has no conflict.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We provide the code for reproducing the main results in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] We provide the implementation details and hyperparameters of all baselines and our methods in the supplementary material.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We conduct experiments over 10 training seeds and report mean and standard deviation.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We describe the computational resource in the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the benchmark used for our experiments.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "zaTKm2bd-o", "reviewer": "Reviewer_SbES", "summary": "This paper proposes a new approach for RL in procedurally generated environments with empirical evidence. At its core is the problem of the critic learning problem and potential overfitting. The proposed approach mainly builds on the Phasic Policy Gradient method by unifying it. The method is compared against natural baselines and other environment-relevant methods. The experiment section also shows several ablation studies.", "questions": "The above comment does not detract from the pertinence of the method, but I have a few comprehension questions or some comments about the paper, please see below:\n\n1. Some relevant works tackling the accuracy of the critic and its impact on the agent's learning seem to be missing: [1] with an empirical analysis of the phenomena and [2] with the introduction of a critic trained only using the residual variance to speed up learning.\n2. Could the authors elaborate on the motivation for adding the Forward/Inverse Dynamics Learning task? The paper's focus is mainly on value function overfitting and its regularization - if any, could the author help understand to what extent does using this auxiliary task help avoid overfitting?\n3. Have the authors considered applying this Forward/Inverse Dynamics Learning task to PPG? It would be interesting to know if using this component also helps PPG and to what extent (maybe even more than DCPG).\n4. Table 2 is quite interesting. Could the authors provide an intuition about why learning the inverse dynamics separately seems to be worse than learning it jointly with the forward dynamics with a single discriminator? Is the total number of parameters for the “dynamics learning module” kept the same? i.e. how many parameters does DCPG+FI have vs. DDCPG?\n\n\n[1] Ilyas, A., Engstrom, L., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., & Madry, A. (2020). A closer look at deep policy gradients. In International Conference on Learning Representations.  \n[2] Flet-Berliac, Y., Ouhamma, R., Maillard, O.-M., & Preux, P. (2021). Learning value functions in deep policy gradients using residual variance. In International Conference on Learning Representations.", "limitations": "The potential negative societal impact or the limitations of the work have not been addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The contribution of this paper is original and appears relevant to the community. It nicely builds on a battery of previous work, which has some overfitting limitations. The empirical analysis is quite furnished and gives very some interesting insights. However, the improvements provided by DCPG or DDCPG over the baselines appear to be less important than what is claimed by the authors. In particular, the statistical significance in Tables 1, 2, and 5 depicted by bold text and the empirical analysis in Section 5.2 cannot be conclusive. Statistically speaking when compared to the baselines, DCPG appears to improve on 2/18 tasks and DDCPG on 4/18. Indeed, unfortunately, the stds appear in many instances to be generally too significant to draw any statistical conclusion. We develop more points below. Regarding the quality, overall, the paper is also mostly well written and develops its ideas clearly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "xOG-T1mslf", "reviewer": "Reviewer_M6vK", "summary": "In this paper, it is first shown that the value function is easier to overfit compared to the policy function.\nBased on this observation, a new policy gradient algorithm called Delayed-Critic Policy Gradient (DCPG) is proposed, which reduces the overfitting of the value function by less frequent updates with more training data.\nMoreover, a single discriminator is applied to learn both forward and inverse dynamics of environments, which helps to generate better representations and improve the performance of DCPG across different tasks.", "questions": "- In Figure 4, how are the true values computed? How to make sure that these computed values are true values during training?\n- Do DCPG and DDCPG use a larger replay buffer than the baseline algorithms, such as PPO and PPG?", "limitations": "In Table 2, the error bars of some results overlap with each, making the claim in Section 5.4 less convincing.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work presents two novel policy gradient algorithms --- DCPG and DDCPG.\nCompared to PPG, DCPG has a more compact model structure with only one encoder, while achieving comparable or better performance on different tasks.\nMoreover, by combining the discriminator that learns environmental dynamics, DDCPG boosts the performance of DCPG furthermore.\nThis paper is clearly written with a good flow.\nThe analysis experiments and the ablation studies are also helpful in understanding the effectiveness of DCPG and DDCPG.\n\nThere are several minor issues.\nIn Section 2.2, the clip operation is not explained though it might be obvious to readers familiar with PPO.\nIn Section 4.3, both the encoder $e_\\theta(\\cdot)$ and the dynamics head $f_\\theta(\\cdot, \\cdot, \\cdot)$ are parameterized by $\\theta$. Is the encoder the same one as shown in Figure 1(c)? If so, how could the same $\\theta$ be used to parameterize a different function, i.e. $f_\\theta(\\cdot, \\cdot, \\cdot)$? It is better to present the model structure of DDCPG as well to clarify.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lmFoYsJSl2", "reviewer": "Reviewer_FX6f", "summary": "This paper claims that training policy and value networks separately for actor critic algorithms like PPO or PPG can lead to overfitting of the value network in procedural generalization tasks like Procgen. To get around the overfitting issue, it proposes to update the value network less often and with more data. It also proposes to add a self-supervised objective to further improve generalization performance, and observes modest gains on some games in Procgen. ", "questions": "* The state-of-the-art claim might not stack well against MuZero++ (https://arxiv.org/abs/2111.01587) given the gap b/w MuZero++ and PPG/PLR/etc. However, it looks like the authors were not simply aware of this work, and DDCPG being model-free is not directly comparable to MuZero++ anyways, so I don't hold any negative points for this one if its mentioned in the revisions. \n* It would be nice to see if and how does the performance changes as the model size and data size is increased. Perhaps you can try increasing the IMPALA network size systematically or using a ResNet, and similarly train on more amounts of data. \n* Generalized in RL is an overloaded term, and this paper studies a very specific form of generalization (i.e. observational or procedural generalization). There are other forms of generalization such as changing the task or reward functions. Make sure that this distinction is clear. \n* The self-supervised objective should be appropriately positioned with respect to prior work listed above. \n* It would be a good idea to use the rliable colab (https://arxiv.org/abs/2108.13264) to plot the results since it has been found that standard reporting of results can be misleading. Specifically, please include the IQM and probability of improvement plots. Also, following the recommendation its better to report Min-max rather than PPO normalized scores.\n* The bolding of scores in Table 1 relies on mean scores it looks like, which can be very misleading. If two methods have overlapping mean and std devs, they should both be bolded or neither since variance can be high for these scores. ", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper makes an interesting empirical observation of overfitting in separate update PPG agents that could harm their generalization performance. \n* The paper is very clearly written except for section 4.3 and gets the idea across clearly. Figure 1 is a great summary of the proposed modification with clear differences to PPG and PPO. \n* The fix for the overfitting issue is straightforward and easy to implement. \n* The experiments are performed on a well benchmarked environment suite (Procgen) and include informative ablations. There are also a bunch of valuable auxiliary experiments investigation value function learning.\n\nWeaknesses: \n* The contribution of delayed value updates itself seems not quite significant, very specific to PPG and its unclear how well it would generalize to other algorithms. \n* The overfitting issue might just vanish as we scale the model and data size for these agents. So I fear the delayed value update contribution might end up being a \"regularization game\" for this specific PPG setup and end up getting \"bitter lesson'ed\" in the long run.\n* Note: The above two are subjective opinions on significance of the contributions. \n* The improvement for DCPG over PPG seems marginal, especially when we look at individual performance curves in Figure 6 of the appendix. It seems like there are only 3-4 out of 16 games where the regularization has a clear and significant benefit.\n* The contribution a self-supervised objective with learned dynamics has been explored multiple times in the literature (SPR (https://arxiv.org/abs/2007.05929), PBL (https://arxiv.org/abs/2004.14646), DeepMDP (https://arxiv.org/abs/1906.02736) etc. Moreover, it has been explored explicitly in Procgen as well in MuZero++ (https://arxiv.org/abs/2111.01587) and DRIML (https://arxiv.org/abs/2006.07217). The idea of using a discriminator for this task was proposed in CPC|Action, and evaluated in Procgen in DRIML and ablated in MuZero++. Using an inverse modelling objective along with the forward objective was also proposed in SGI (https://arxiv.org/abs/2106.04799). The paper unfortunately doesn't mention or appropriately position itself against these works.\n* The reporting of results does not follow the recently proposed recommended practices as described in rliable (https://github.com/google-research/rliable), and thus might be prone to making incorrect conclusions. More details below. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept