Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards
Authors: Xingyu Lu
Pieter Abbeel
Stas Tiomkin
Venue:  (2020)
Keywords: reinforcement learning
representation learning
reward shaping
predictive coding

Abstract: While recent progress in deep reinforcement learning has enabled robots to learn complex behaviors, tasks with long horizons and sparse rewards remain an ongoing challenge. In this work, we propose an effective reward shaping method through predictive coding to tackle sparse reward problems. By learning predictive representations offline and using these representations for reward shaping, we gain access to reward signals that understand the structure and dynamics of the environment. In particular, our method achieves better learning by providing reward signals that 1) understand environment dynamics 2) emphasize on features most useful for learning 3) resist noise in learned representations through reward accumulation. We demonstrate the usefulness of this approach in different domains ranging from robotic manipulation to navigation, and we show that reward signals produced through predictive coding are as effective for learning as hand-crafted rewards.

Under review as a conference paper at ICLR 2020
PREDICTIVE CODING FOR BOOSTING DEEP REINFORCEMENT LEARNING WITH SPARSE REWARDS
Anonymous authors
Paper under double-blind review
ABSTRACT
While recent progress in deep reinforcement learning has enabled robots to learn complex behaviors, tasks with long horizons and sparse rewards remain an on- going challenge. In this work, we propose an effective reward shaping method through predictive coding to tackle sparse reward problems. By learning predic- tive representations offline and using these representations for reward shaping, we gain access to reward signals that understand the structure and dynamics of the en- vironment, In particular, our method achieves hetter learning by providing reward signals that 1) understand environment dynamics 2) emphasize on features most useful for learning 3) resist noise in learned representations through reward accu- mulation. We demonstrate the usefulness of this approach in different domains ranging from robotic manipulation to navigation, and we show that reward signals produced through predictive coding as effective for learning as hand-crafted rewards.
INTRODUCTION
Recent progress in deep reinforcement learning (DRL) has enabled robots to learn and execute com- plex tasks, ranging from game playing (Jaderberg et al. 2018 OpenAl 2019), robotic manipulations (Andrychowicz al. 2017 7 Haarooja et al. 2018), navigation (Zhang al. 201 However, in many scenarios learning depends heavily on meaningful and frequent feedback from the environ- ment for the agent to learn and correct behaviors. As a result, reinforcement learning (RL) problems with sparse rewards still remain difficult challenge (Riedmiller et al.) 2018 Agarwal et al. 2019).
In a sparse reward setting, the agent typically explores without receiving any reward, until it enters a small subset of the environment space (the "goal"). Due to lack of frequent feedback from the environment, learning in sparse reward problems is typically hard, and heavily relies on the agent entering the "goal" during exploration. A possible way to tackle this is through reward shaping (Devlin & Kudenko 2012) Zou et al., 2019 Gao & Toni 2013), where manually designed rewards added to the environment to guide the agent towards finding the "goal"; however, this approach often requires domain knowledge of the environment, and may bias learning if the shaped rewards
not robust (Ng et al. 1999).
RL problems often benefit from representation learning (Bengio et al,) 2013), which studies the transformation of raw observations of an environment (sensors, images, coordinates etc) into more meaningful form, such that the agent can more easily extract information useful for learning. In- tuitively, raw states contain redundant irrelevant information about the environment, which the agent must take time to learn to distinguish and remove; representation learning directly tackles this problem by either eliminating redundant dimensions (Kingma & Welling 2013 van den Oord al., 2017) or emphasizing more useful elements of the state (Nachum et al. 2018a). Much of the prior work on representation learning focuses on generative approaches to model the environment, but some recent work also studies optimizations that learn important features (Ghosh et al. 2018).
In this paper, we tackle the challenge of DRL to solve sparse reward tasks: we apply representation learning to provide the agent meaningful rewards without the need for domain knowledge. In par- ticular, we propose to use predictive coding in an unsupervised fashion to extract features that max- imize the mutual information (MI) between consecutive states in a state trajectory. These predictive features are expected to have the potential to simplify the structure of an environment's 's state space: they are optimized to both summarize the past and predict the future, capturing the most important

---

Under review as a conference paper at ICLR 2020
elements of the environment dynamics. We show this method is useful for model-free learning from either raw states or images, and can be applied on top of any general deep reinforcement learning algorithms such as PPO (Schulman et al. 2017).
Although MI has traditionally been difficult to compute, recent advances have suggested optimizing on tractable lower bound the quantity (Hjelm et al., 2018: Belghazi et al. 2018 Oord et al., 2018). We adopt one such method, Contrastive Predictive Coding (Oord et al. 2018 J. to extract
features that maximize MI between consecutive states in trajectories collected during exploration (one thing worth noting is that this method is not restricted to specific predictive coding schemes such as CPC). Such features are then used for simple reward shaping in representation space to provide the agent better feedback in sparse reward problems. We demonstrate the validity of our method through extensive numerical simulations in a wide range of control environments such as maze navigation, robot locomotion, and robotic arm manipulation (Figure B. In particular, we show that using these predictive features, we provide reward signals as effective for learning as hand- shaped rewards, which encode domain and task knowledge.
This paper is structured as follows: We begin by providing preliminary information in Section and discussing relevant work in Section []] then, we explain and illustrate the proposed method in Section 4 lastly, we present experiment results in Section S and conclude the paper by discussing the results and pointing out future work in Section 6
2 PRELIMINARIES
This paper assumes finite-horizon Markov De- cision Process (MDP) (Puterman 1994), defined by a tuple (S, p.r. T). Here, S E Rd denotes the state space, A E I denotes the action space, P : S x A + R$ denotes the state transition distribution, r : S x A + R denotes the reward function, y E [o, 1] is the discount factor, and finally T is the horizon. At each step I, the action a, A is sampled from a policy distribution To(a (5x) where & E S and 0 is the policy parameter. After transiting into the next state by sampling from p(s |t, st), where p E P, the agent receives a scalar reward r(s,, at). The agent continues performing actions until it enters a terminal state or t reaches the horizon, by when the agent has completed one episode. We r denote the sequence of states that the agent enters in one episode.
Reinforcement Learning and Reward Shaping:
With such definition, the goal of RL is to learn a policy TT- (azls;) that maximizes the expected discounted reward E,, r. [R(TToTTT 1)] Er, 1E I'1 7'r(st, at)], where expectation is taken on the possible trajectories r and starting states vi-- In this paper, we assume model-free learning, meaning the agent does not have access to P.
Reward shaping essentially replaces the original MDP with a new one, whose reward function is now rs(s,, az). In this paper, reward shaping is done train policy n. that maximizes the expected discounted reward in the original MDP, i.e. Ex,,, [R(TToTT-)]
Mutual Information and Predictive Coding: Mutual information measures the amount of informa- tion obtained about one random variable after observing another random variable (Cover & Thomas 2012). Formally, given two random variables X and Y with joint distribution p(z, y) and marginal densities p(x:) and p(y), their MI is defined as the KL-divergence between joint density and product of marginal densities:
p(z, 9) p(z)p())
MI(X;Y) DKL. ()ll())))) Re(a..) [log
Predictive coding in this paper aims to maximize the MI between consecutive states in the same state trajectory. As MI is difficult to compute, we adopt the method of optimizing on a lower bound, InfoNCE (Oord al. 2018), which takes the current context et to predict future state St+R:
f f((44, C:) MI(s(+u:Ce) 2 Es(log f(utk. a) + Eejesf(zj,0
(2)
Here, f(x y) is optimized through cross entropy to model a density ratio: fx, x P(rly)
P(z) 244k
is the embedding of state Zz+* by the encoder, and e is obtained by summarizing the embeddings of previous states in segment of trajectory, 1: through gated recurrent unit (Cho al.)
2

---

Under review as a conference paper at ICLR 2020
Figure 1: Clustering states in embedding space. We start with the original states (left most), obtain and cluster the embeddings (middle two), and finally label original states by clusters (right most).
2014). Intuitively, the context C pays attention to the evolution of states in order to summarize the past and predict the future; thus, forces the encoder to extract only the essential dynamical elements of the environment, elements that encapsulate state evolution.
RELEVANT WORKS
Our paper uses the method of Contrastive Predictive Cod- ing (CPC) (Oord et al. 2018), which includes experi- ments in the domain of RL. In the CPC paper, the In- foNCE is applied the LSTM component (Hochreiter & Schmidhuber 1997) of an A2C architecture (Mnih et al. 2016 Espeholt et al. 2018). The LSTM maps every state observation to an embedding, which is then directly used for learning. This differs from our approach, where we train on pre-collected trajectories to obtain embeddings, and only use these embeddings to provide rewards to the agent, which still learns the states. Our approach has two main advantages: 1) Preprocessing states allows us collect exploration-foeusee trajectories, and obtain embeddings that are suitable for multi-tasking. 2) Using embeddings to provide rewards is more resistant to noises in embeddings than using them training features, since in the former case we care more about the accumulation of rewards across multiple states, where the noises are di- luted.
Figure 2: Illustration of policy trained through clustering. The first-step pol- icy guide the agent towards the correct cluster (white arrow), and the second- step policy guides the agent towards the goal (grey arrow).
Applying representation learning to RL has been studied many prior works (Nachum et al. 201 8a Ghosh et al. -uueel Dupré al. 2018). In a recent paper on actionable representation (Ghosh et al. plied providing the agent useful reward signals. In actionable representation paper, states treated as goals, and embeddings are optimized in way that the distance between two states reflects the difference between the policies required reach them. This is fundamentally different from our approach, which aims to extract features that have predictive qualities. Furthermore, computing actionable representation requires trained goal-conditioned policies as part of the optimization, which is strict requirement, while this paper aims to produce useful representations without need- ing access trained policies.
2018 Oord al. 2018 Caselles-
2018), representation learning is also ap-
Lastly, VQ-VAE (van den Oord et al. 2017) is generative approach that provides principled way of extracting low-dimensional features. contrast to VAE (Kingma & Welling 2013), it outputs discrete codebook, and the prior distribution is learned rather than static. VQ- VQEV could be useful for removing redundant information from raw states, which may speed up learning; however, since the goal of VQ-VAE is reconstruction, it does not put emphasis on features that are particularly use- ful for learning, nor does it attempt to understand the environment dynamics across long segments of states. Our of predictive coding is thus be a better fit for reinforcement learning, size on features that help understand the evolution of states rather than reconstruct each individual state.
empha-
3

---

Under review as a conference paper at ICLR 2020
4 METHOD
4.1 LEARNING PREDICTIVE FEATURES
In this step, the key idea is train, in an unsupervised fashion and prior to learning, an encoder that extracts predictive features from states, We begin by collecting state trajectories through initial exploration. While there no requirement for any specific exploration strategies, we used random exploration with manual resets to collect diverse trajectories without need for pre-trained policies.
From these trajectories, segments of consecutive states are sampled and used to train a CPC encoder: for each segment, fixed number of beginning states (zz-na) are encoded into latent embeddings (zt-nct) and summarized by a GRU (c, = fcnu (zt=na)); the output of the GRU was referred to in the original paper as "context", which is then used to predict the embedding of each remaining state (2(+&) in the segment through a score function S4+4 exp(2144 W20)). More architectural details can be found in Appendix A
4.2 APPLYING PREDICTIVE FEATURES TO RL
The trained embeddings then used for reward shaping in 2 ways:
We sample random states from the environment and cluster their corresponding em- beddings. We found that clustering these embeddings provide meaningful information the global structure of the environments, i.c. states that are naturally close to each other (Figure [ We then these clusters to provide additional reward signals to the agent, in particular awarding the agent a positive reward for entering the cluster that contains the goal. This way, the agent more likely enter the subset of state space that is close the goal (Figure 2 and learning faster and more stable.
Clustering:
Optimizing on Negative Distance: environments with large state spaces, we directly optimize on the distance between the current state and the goal state in represen- tation space (or embedding space, both of which will be used interchangeably for the rest the paper). That is, add additional negative distance term to the reward at each step t, || 2g where is the embedding of the current state 8t, and 2g is the embedding of the goal 4g. This simple approach leads surprisingly good improvements for learning, even for environments with non-linear structures such mazes (Figure 3
B.
Figure 3: Illustration of policy trained through optimizing on negative distance embedding space. Note that moving straight in embedding space (right) may correspond to moving around a wall in
In the next section, we study both the embeddings ob-
tained from initial training as well as both of the applica- tions discussed above.
the maze (left).
5 EXPERIMENTS
In this section, we will address the following questions:
1. Does predictive coding simplify environment structure?
2. Do these simplified representations provide reward feedback to agent in sparse reward tasks?
We show the result of applying predictive coding to learning in five difffrrnnt environments: Grid- World, HalfCheetah, Pendulum, Reacher, and AntMaze (Figure 4 These environment form a rich set of standard DRL experiments, covering both discrete and continuous action spaces.
GridWorld environment (Chevalier-Boiie) Boisvert et al. 2018) is used as the primary experiment for dis- crete settings. Although it has simple dynamics, Grildorld environments have variety of maze structures that pose an interesting representation learning problem: two points between a wall are close distance wise, but they might require the agent take dramatically long route reach one
4

---

Under review as a conference paper at ICLR 2020
Figure 4: Visualization of environments used in this paper. The environments GridWorld, HalfCheetah, Pendulum, Reacher, and AntMaze respectively.
point from the other. In our experiments, we demonstrate that predictive coding is able understand the global structure of any arbitrary maze, and map states in the latent space according to their actual distances in the maze.
We use Mujoco (Todorov et al., 2012) and classic Gym (Brockman et al. 2016) environments for continuous control settings, HalfCheetah, Pendulum, and Reacher are environments in continuous setting with richer dynamics than GridWorld. While they have simpler global structures (e.g. pendu- lum moves circle), we show that predictive coding is able to understand a hierarchy of features in these environments, and that these features be directly incorporated to speed up learning. AntMaze environments (Nachum et al. 201 have continuous control dynamics as well as in- teresting maze structures similar GrrdWorld. As result, representations learned by predictive coding in AntMaze show both understanding of global structure and formation of hierarchies of features.
To show the generality of our approach, we use an open-sourced implementation of CPC and stan- dard DRL baselines Saselineeoforrr for training. A compilation of all code used in this paper will be made publicly available. Further details on the experimental setup and architectural details can be found in Ap- pendix A
5.1
DISCRETE SETTINGS
5.1.1 GRIDWORLD
GridWorld environments are 2D 17-by-17 square mazes with different layouts. We include differ- ent layouts: one with a U-shaped barrier (U-Maze), one with 4 rooms divides walls (Four-Room), and one with square blocks (Block-Maze).
The result of applying CPC repre- sentation learning GridWorld is shown in Figure In all three exper- iments, CPC learns representations that reflect the true distance between two points. For instance, as seen in the plot, states between the barrier (blue and green states) in U-Maze are mapped to points far from each other in the representation state, although being distance-wise close. Similarly, the states in the blue room and green room of Four-Roome are mapped to two ends of a long band in the repre- sentation space, with states in the red and pink rooms located in the middle. This reflects the need for the agent to go through the red and pinks rooms to reach the blue room or the green
Figure 5: CPC representation of GridWorld environment: U-Maze (left), Four-Rooms (middle), and Block-Maze (right). By clustering embeddings (bottom, all embeddings are visualized by T-SNE), we are able to recover clusters corresponding to natural structures of the mazes (top).

---

Under review as a conference paper at ICLR 2020
room. Lastly, representation learned in Block-Maze restores the true structure of the maze from imaged-base based observations.
We assess the quality of the embeddings by analysing how much they reflect the true distances between states. For each maze environment, we sample random pairs points from the maze, and plot the true distance (obtained by running A in the original maze between the pair against their distance in the representation space (L-2 norm). Additionally, we run linear regression to obtain a line of best fit for each plot. The result for Barrier Maze is shown in Figure and for all three mazes we observe strong correlation between the true distances and the distances in latent space.
We show the result of applying clustering sparse reward problems in Grid the agent is randomly spawn and navigates in the maze, and it only receives positive re- ward when reaching the goal. Without additional reward signals, the agent might not able reach the goal it is spawn in f'a--wway location. To make use of clusters obtained from CPC representations, we train a two step policy: the agent first goes the cluster that contains the goal, and then to the goal. We reward the agent for reaching the cluster in the first step, and use environment reward for the second step. This way, the agent receives more signal in all locations in the maze. An illustration of policy trained this way is shown in Figure
6--
Figure 6: Plot of true distance be- tween states in maze vs distance in rep- resentation space. The x-axis is the true distance obtained by A", while the y- axis is the distance in the representation space. A line of best fit is provided for illustration.
We find that this approach leads to better learning in all three mazes (Figure 7) In all three experiments, the re- ward (adjusted remove cluster reward) converges faster and to higher values with clustering. This is likely be- cause the additional reward for entering the cluster guides the agent towards states that are naturally close to the goal, allowing the agent to reach the goal more frequently during exploration. Table shows that the policy learned through clustering significantly outperforms the policy learned in standard setting.
Feaffoarm
Soc-Moos
Ear 2050
32
48
Bey Came
2003 3008
2000
1200
280 30:0
680 9008
11:00 5000 100R 1R800
Figure 7: Learning curves of GridWorld environments, averaged over 3 runs. The reward curve with clusters is shifted down by to remove the cluster reward.
Table Success Rate GridWorld Environments. A successful run indicates that the agent reaches the goal from random stating potion in the maze within 100 steps.
Setup
Without Clustering
With Clustering
U-Maze
61%
98%
Four-Rooms
71%
98%
Block-Maze
98%
100%

---

Under review as a conference paper at ICLR 2020
5.2 CONTINUOUS SETTINGS
We study four different control environments: two moved to Appendix A).
Pendulum, AntMaze, Reacher, HalfCheetah (latter
Environments used this section have much richer dynamics than GridWorld. We show that the learned representations simplify these environments both by understanding the global structure of the environment (AntMaze) and encoding meaningful hierarchies of features.
Unlike GridWorld, simple clustering strategies are less effective because of the large state space. Instead, we directly optimize on the agent's 's distance to goal in representation space. We show that this simple approach can lead improvement in learning as much as using hand- shaped rewards. For each environment, we include 4 setups, each setup using 3 random seeds:
1. Sparse reward (blue): Providing an agent small positive reward when it reaches the goal
2. Hand-shaped reward (pink): Providing an agent hand-shaped reward at each step (hand- shaped reward)
3. Raw distance (green): Providing an agent negative penalty on the distance between cur- rent state and goal state (plus sparse reward to goal for AntMaze only)
4. Embedding distance (orange): Providing an agent negative penalty on the distance be- tween current state and goal state in the representation space (plus control penalty on action norm for HalfCheetah only).
The details of the hand-shaped reward schemes can be found in Appendix
5.2.1 PENDULUM
The Pendulum is classic control problem where rigid arm freely swings about fixed center. To imi- tate the swing-up task, we set the goal states to have angles 0 [-0.1, 0.1]. where an angle of 0 means the arms pointing straight up. Additinally, we consider "the goal is reached" only after the agent manages to stay among goal states for 5 consecutive steps. For hand-shaped reward, we penalize the magnitude of the angle encourage the arm to maintain an up- ward position.
19%
Figure 8: Illustrations of embeddings of Pendulum and learning curves for different reward schemes. Embeddings (bottom-left) cluster primarily by angle of the arm (top-left). Orange curve converges significantly faster than others.
As shown in Figure 8 clustering the embeddings produces clusters pri- marily by angle. However, there is still lot of overlapping between clusters when only consider the position of the arm, suggesting that the arm's 's velocity also plays a less important role. This hierarchy between position and velocity was very beneficial for learning, as the agent would learn to swing up the arm first before decelerat- ing the arm to maintain top positions. Indeed, optimizing on distance in embedding space (orange) led to much faster learning than all other setups, including the hand-shi hand-shaped rewards (pink), where as optimizing on distance in the original space (green) leads to sub-optimal behaviors such as reducing velocity early.
5.2.2 ANTMAZE
Finally, in AntMaze, four-legged robot navigate in maze-like environment until it reaches the goal area. Naturally, AntMaze has both the rich dynamics robot as well as the structure of a maze environment, and is helpful for illustrating the power of predictive coding both reflect the global structure of an environment while picking the most important features. In our experiment setup, use thin wall to block passage in the maze, that state the other side of the wall

---

Under review as a conference paper at ICLR 2020
may appear close the agent, but is in reality very far. We set the goal state be the lower left corner of the maze; for hand-shaped rewards, we assign each state in the maze correct direction to move in and award the agent for moving in that particular direction.
Figure 9 shows the visualization of embeddings and learning curves. One immediate observation is that clusters properly divide the maze into 4 sections, and states between the wall are now more separated; this is similar to the embeddings in Grid- World, where the embeddings under- stand the true distance between dif- ferent positions in the maze. At the same time, even though full states are provided (positions as well as full joint dynamics), the clusters reflect that embeddings learn to use the po- sition of the agent as the most im- portant feature. As a result, learn- ing using these embeddings (orange) achieves performance on par with
Illustrations of embeddings of AntMaze and learning curves for different reward schemes. Embeddings (bottom-leftt cluster primarily by position of the agent (top- left). Orange curve achieves similar performance to pink curve with smaller variance. hand-shaped reward (pink).
Figure 9:
6 DISCUSSION
6.1 EMBEDDINGS AS FEATURES VS REWARD SHAPING
Mutual information maximization notori- ously difficult optimize, and may easily pro- duce noisy embeddings without sufficient train- ing data. Our approach mitigates this problem in two aspects. Firstly, we preprocess the em- beddings instead of training them online, so that the agent avoids learning on noisy embeddings that are not fully trained. Secondly, instead of using the embeddings as features to train on,
we use them to provide reward signals to the
agent, who still learns using the raw features. This approach more resilient to noises in em- beddings, especially for policy gradient meth- ods, since we care more about total rewards across trajectories than the rewards of individ- ual states.
Aeg
20400 330000
3000000
135000
200000 1755000
We illustrate the above points by comparing training with epe features and our approach in the Reacher environment. Both experiments use the same architectural and algorithmic set- tings, and the raw states used for training the embeddings contain information about the goal. As shown Figure 10 the use of cpc embed- dings as features lead to insignificant improvements to learning, where as using these embeddings to only provide reward signals led to the best performance.
Figure 10: Illustration of learning curves for dif- ferent setups. Learning the original sparse ward problem with embeddings as features (yel- low) did not lead to significant learning improve- ment, while using embeddings to provide rewards (orange) achieved the best learning.
6.2 TEXTURE AGNOSTIC PREDICTIVE CODING
In this section, we discuss an important advantage of predictive coding: since embeddings are opti- mized maximize their predictive abilities, less meaningful information such as the texture of the
8

---

Under review as a conference paper at ICLR 2020
background from the observations ignored. This property of predictive coding differentiates itself from other unsupervised learning methods such as autoencoder or VAE, which inevitably pay attention to the background in order to reconstruct the original states.
This property of predictive coding makes it possible for an agent to learn in constantly chang- ing environment, such game (Bellemare et al. 2012). We showease this property by training the encoder with states from the Pendulum environment with multiple backgrounds (bricks, sand, cloth), and assess the encoder's generalizability tains examples of textures used for training and validation, as well as the clustering results of their particular,
new textures (such wood). Figure [1] con-
textures resulted in very similar embeddings, even though the encoder had never seen the wooden texture during training. We conclude that predictive coding has learned to ignore the background, which contains less important information about the state dynamics.
corresponding embeddings.
Figure 11: Illustrations two background textures and clustering of their corresponding embed- dings (clustered by embeddings, illustrated by mapping to position of arm). The cloth texture (left) was used during training, while the wood texture (right) was only used during validation. Both textures led to embeddings properly clustered by the angle of the arm.
6.3 EXPLORATION
Our proposed method relies on the quality of trajectories collected at the beginning, which in turn depends the initial exploration. Although in most cases, exploration with random policies or simple goal-conditioned policies enough to produce trajectories that expose the environment dy- namics, there environments with extremely long horizons large state spaces that effective exploration without learning the task is difficult. An example is Montezuma's Revenge, which is currently unsolvable without algorithms designed tackle hard exploration (Ecoffet al. 2018) or expert demonstration data (Salimans & Chen 2018). For future work, a direction is to train the embeddings online, i.c. during training the agent. This way, trajectories collected may be more relevant to the particular training task, and we could obtain high-quality embeddings (high-quality in the sense that they are useful for the particular training task) without thorough exploration of the environment. As discussed in the first paragraph, learning on intermediate embeddings may be un- desirable, so agent should initially rely purely on environment rewards, and only start receiving rewards shaped by embeddings after the embeddings reach certain quality mark (for CPC, this could be checked by the InfoCEE loss, which indicates a lower bound on mutual information).
6.4 NEGATIVE DISTANCE AS A POTENTIAL FUNCTION
One of the major issues with reward shaping that could potentially bias learning, leading the agent to learn suboptimal policy. In a previous work on policy invariance under reward trans- formation (gg al.) 1999), the notion of potential-based reward shaping function establishes the conditions for guaranteeing unbiased learning. With the use of predictive coding, our negative dis- tance reward scheme a potential function if the latent space induced by the encoder preserves the metric properties of the original state space. A rigorous formulation of this setting necessitates mathematical analysis, which out the scope of this preliminary study.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In International Conference on Machine Learning, pp- 130-140, 2019.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Picter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. In Advances in Neural Information Processing Systems, pp. 5048-5058 2017.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Devon Hjelm, and Aaron Courville. Mutual information neural estimation, In International Conference on Machine Learning. pp. 530-539, 2018.
Mare G Bellemare, Joel Veness, and Michael Bowling. Investigating contingency awareness using atari 2600 games. In Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 37(8):1798-188)) 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Hugo Caselles- Michael Garcia Ortiz, and David Filliat. Continual state representation learn- ing reinforcement learning using generative replay. arXiv preprint / 2018.
Maxime Chevalier- Chevelier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: pettps//githuss htt.:/uuu.coomccooommmmmmomo wtt.c/om/mmaaaam matomm/aximmma
http:///wwaaaec//eaaooaoeaecee/e mmebb http://wwww.mmm/..ymmmmigi.g.../ gym-minigrid 2018.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decodes approaches. Syntax, Semantics and Structure in Statistical Translation, pp. 103, 2014.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Sam Michael Devlin and Daniel Kudenko. Dynamic potential- -baed reward shaping. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, pp. 433- 440. IFAAMAS, 2012.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Montezumas revenge solved by go-explore, a new algorithm for hard-exploration problems (sets records on pitfall, too). 2018. BtLLhttps:/// :/tb:p:://eggg wberr cww/ wacpn/gooepploeeeeee explore/
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.0111 082.00161. 2018.
Yang Gao and Francesca Toni. Potential based reward shaping for hierarchical reinforcement learn- ing. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal- conditioned policies. arXiv preprint arXiv: 1811.:181 2018
Tuomas Haarnoja, Aurick Zhou, Schoon Ha. Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. arXiv preprint 1812:18112.11000 2018
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schul- man, Szymon Sidor, and Yuhuai Wu. Stable baselines. hnttpa://gitan com/hill a/
stable-basel nes 2018.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808 06670, 2018.
10

---

Under review as a conference paper at ICLR 2020
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Max Jaderberg, Wojeiech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari Morcos, Avraham Ruderman, et al. Human- level performance in first-person multiplayer games with population-based deep reinforcement learning. arXiv preprint arXiv:1807.01281, 2018.
Diederik Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Sameera Lanka and Tianfu Wu. Archer: Aggressive rewards to counter bias in hindsight experience replay. arXiv preprint (809:02077000 2018.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arxiv:/810.011257 2018a.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3303-3313 2018b.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. ICML, volume 99, pp. 278 777, 1999.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv:1807.03744 2018.
OpenAL. OpenAl Five, 2019. URL https:// hwt.mnaa..cmmffvv//////
Martin L Puterman. Markov decision processes: Discrete stochastic dynamic programming. 1994.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. International Conference on Machine Learning, pp. 4341-4350, 2018.
Tim Salimans and Richard Chen. Learning montezuma's 's revenge from single demonstration. arXiv preprint 1t.2:v:88/2.03330. 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint wxXiv:1707.06347 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep rein- forcement learning with successor features for navigation across similar environments. In 2017 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2371-2378. IEEE, 2017.
Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, and Jun Zhu. Reward shaping via meta- learning. arXiv preprint 1900i 2019.
11

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 ENVIRONMENT DESCRIPTIONS
The agent is point that can move horizontally vertically in a 2-D maze structure. Each state observation is compact encoding of the maze, with each layer containing information about the placement of the walls, the goal position, and the agent position respectively. The goal state is one in which the goal position and the agent position are the same.
GridWorld:
The agent 2-d two-legged robot freely moving on a horizontal plane. Each state observation encodes the position of the agent, the angles and the velocities of each joint. The goal states are any states where the agent position æ 2 10.
HalfCheetah:
The agent is rigid arm moving about fixed center by applying force to its tip. Each state observation encodes the angle and the velocity of the arm. The goal states are any states where the agent achieves an angle 0 E [-0.1, 0.1].
Pendulum:
The agent a robotic arm with two rigid sections connected by a joint. The agent moves about fixed center the plane by applying force to each rigid section. Each state observation encodes the angles of two sections, the position of the tip the arm, and the direction to goal. The goal state is where the tip of the arm touches certain point the plane.
Reacher:
The agent is four-legged robot freely moving in a maze structure. Each state observa- tion encodes the position of the agent, the angles and the velocities of each joint. Instead of learning from scratch, we pre-train a simple direction-condition this environment. The goal states square area with side and center (0,0).
AntMaze:
walking policy and learn to navigate in
Table 2: Environment dimensions and horizons
Environment
State/Goal Dimensions
Action Dimensions
Maximum Steps
GridWorld
(17, 17, 3)
(4.)
100
HalfCheetah
(18,)
(6)
1000
Pendulum
(3.)
(1.)
200
Reacher
(11,)
(2.)
50
AntMaze (113.)
(2.)
600
A.2 NETWORK PARAMETERS AND HYPERPARAMETERS FOR LEARNING
For all experiments in this paper we use standard PPO baseline (Hill et al, 2018) to train. We use two fully-connected layers with output size 64 for the actor critic. We provide hyperparameters below, and refer to Hill et al. (2018) for all other implementation details.
Table 3: Hyper parameters for PPO
Parameter
Value
gamma
0.99
entropy coefficient
0.01
leaning rate
2.5 10
clip range
[-0.2,0.2]
max gradient norm
0.5
batch size
128
12

---

Under review as a conference paper at ICLR 2020
A.3 TRAINING CPC
We follow the approach proposed in Oord et al. (2018) to obtain predictive features from states. The details provided in the tables below. All hyperparameters for training CPC are tuned through performing grid search.
Table 4: Network details for training CPC
Environment
Encoder Type
Autoregressive Model Type
GridWorld
FC layers with output size 64 Cov layers with (3, 3) kernel, and stride 2
GRU with output size 256 GRU with output size 256
All others
Table 5: CPC training details
Environment
No. Trajectories
Length of Trajectory
Context Length
Predict Length
Epoch
GridWorld
200
100
10
10
2
HalfCheetah
500
1000
50
50
5
Pendulum
200
300
10
10
10
Reacher
200
500
20
20
10
AntMaze
500
1000
30
30
5
A.4 HAND-SHAPED REWARD SCHEMES
For continuous environments, we show that using predictive features provides reward signals as informative hand-shaped rewards, which encodes domain and task knowledge about the environ- ment. In particular, each hand-shaped reward scheme contains information about where the goal is and how to get there. We provide details below for each environment.
(Ie+1 - 22) - a ||ae]|, where v; is the x position of the agent at time i, and ag the action input the agent time t.
HalfCheetah:
Pendulum: - -(10,1 + a | + [|a,]),, where 0r, wi, and a, are the angle, angular velocity, and action input time respectively.
Reacher: -(1pp: pyll
|aoll), where pa is the position of the tip of the arm at time t, is the position of the goal, and a, is the action input at time t.
AntMaze: - ((x++1 - Iy cos(()) + (ve+1 - y/ sin())))) where ze. Ye are the x, y positions of the agent at time f, and 0, is the hard-coded direction to travel in.
13

---

Under review as a conference paper at ICLR 2020
A.5 EXPERIMENT RESULT FOR HALFCHEETAH
Figure 12 shows the visualization of embeddings of random states well as the comparison between different reward setups. The predictive fea- tures focus on the most significant el- ement of the environment: the x po- sition of the agent, allowing us to re- cover horizontally spaced clusters.
As the plot shows, optimizing on the negative distance embedding space (orange), optimizing the nega- tive distance in raw space (green), and optimizing hand-shaped rewards (pink) all lead to similar perfor- mances. While this is less convinc- ing than other environments, we ob- serve that optimizing negative dis- tance in representation space is not worse; rather, it is likely that optimizing on negative distance raw space already good enough, since the x-position of the agent has the largest variance among all other features.
30033
200000
Tromel
Figure 12: Illustrations of embeddings of HalfCheetah and learning curves for different reward schemes. Embeddings (bottom-left, visualized by T-SNE) cluster by position of the agent (top-left, with x-aixs being x position and y-axis being y-position of the agent).
A.6 EXPERIMENT RESULT FOR REACHER
In the Reacher environment, a robotic arm has two sections with a joint in the middle. The arm's one end is fixed at the center of the plane, and its goal is typically to reach a certain point on the plane with the other end of the arm. This can be naturally for- mulated as sparse reward problem (Lanka & Wu 2018), where the agent receives no reward until reaches the goal state. For hand-shaped reward, we penalize the distance between the tip of the arm and the goal point plus the L-2 norm of agent's ' action for sta- bility.
10.
33000 1U006 BEROH 22N00
Figure 13: Illustrations of embeddings of Reacher and learning curves for different reward schemes. Embeddings (bottom-left) cluster primarily by angles of sections of the arm (top-left). Orange curve achieves almost-perfect per- formance 750k steps.
Similar to Pendulum, embeddings of Reacher achieves clusters primarily by the position of the arm, as shown in Figure (note that each axis is the angle of one section of the arm). Consequently, optimizing on the distance in embedding space (orange) allows the agent to quickly learn to move directly towards the goal. This turned out be more stable than using hand-shaped reward (pink), which sometimes led the agent to occasionally overshoot and miss the goal.
(note
14


Reviews:
[{"review_id": "SkgcmjEaFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "  *Synopsis*:\n  This paper proposes using the features learned through Contrastive Predictive Coding as a means for reward shaping. Specifically, they propose to cluster the embedding using the clusters to provide feedback to the agent by applying a positive reward when the agent enters the goal cluster. In more complex domains they add another negative distance term of the embedding of the current state and goal state. Finally, they provide empirical evidence of their algorithm working in toy domains (such as four rooms and U-maze) as well as a set of control environments including AntMaze and Pendulum.\n\n  Main Contributions:\n  - Using the embedding learned through Contrastive Predictive Coding for reward shaping.\n  - A reward shaping scheme that seems generally applicable to any embedding.\n\n\n  *Review*:\n\n  I think the paper provides a compelling motivation, and is well written. I think using the embedding learned through CPC could provide meaningful semantics for representation learning and for reward shaping (as done in the current paper), and encourage the authors to continue down this line of inquiry. Unfortunately, I have several concerns over the method as currently implemented and the empirical comparisons (specifically with the chosen competitors) which I detail below. Given these concerns I am unwilling to recommend accepting this paper, unless several of these concerns are addressed.\n\n  1. This algorithm, by nature, is purely offline as the CPC and clustering all are currently done offline. Furthermore, the clustering portion of this approach requires states to be randomly sampled from the environment to create a nice set of clusters which are representative of the environment's full state space. These two requirements significantly limit this approach, especially when looking at domains where simulation is not possible, or the underlying state distribution is unknown. By and all, I don't think this means we should completely discount this method entirely and the authors do mention this as a detriment to their algorithm in section 6.3. I'm wondering if this paper should look at implementing an online version before publication, but think this is less prescient to the other concerns.\n\n  2. I am concerned about the current policy learning scheme (i.e. tiered policy (1) go to the correct cluster (2) go to the goal) which seems to only be used by your approach. This invalidates the comparisons made with the other reward shaping schemes as this goes beyond reward shaping (you are learning two separate policies).\n\n  3. The current competitors are unsatisfactory as they don't include other reward shaping techniques from the literature. Also the related works section seems to completely disregard this part of the literature. I would recommend comparing your approach to (methods you even cite!):\n     - \"Reward Shaping via Meta Learning\" by Zou et. al https://arxiv.org/pdf/1901.09330.pdf\n     - \"Potential based reward shaping\" by Gao et. al. https://www.aaai.org/ocs/index.php/IJCAI/IJCAI15/paper/viewPaper/10930\n     (I'm sure there are others beyond these)\n\n  4. I'm also concerned with the current significance of the results, particularly AntMaze, Half Cheetah, and Reacher. I'm especially concerned because I don't feel your competitors are not relevant/representative of the current state of reward shaping.\n\n  5. It would be worthwhile to try this method on several RL algorithms (i.e. Q-learning, TRPO, SAC, DDPG, etc...). This will help readers understand if this approach is a general method, or only applicable to PPO.\n\n  6. I quite like the idea of predictive coding (albeit the original scheme presented by Rao and Ballard 1999) as an unsupervised representation learning scheme, but am unsure this is critical for your method and the current approach is not really predictive coding in a sense (or at least the ideas I'm familiar with from cognitive computational neuroscience). I am concerned with the predictive coding idea being highlighted here as a key ingredient, but none of the papers containing the originating idea of predictive coding are mentioned. Rao and Ballard is one, but there is a rich literature following from this work into the free energy formulation (Friston) and active learning. These ideas should appear in your introduction, as you heavily rely on them. Also there are other predictive coding schemes for unsupervised representation learning (such as PredNets from David Cox https://coxlab.github.io/prednet/), which I believe should be mentioned. In light of these other methods, I'm not sure your discussion on only using predictive features for reward shaping is accurate, and instead these claims should be softened for only features learned through CPC.\n\n  Missing experimental settings:\n  - Number of runs tested\n  - What are the error bars in your plots?\n\n  I would also like to recommend \"Deep Reinforcement Learning that Matters\" by Henderson for a reference on how to conduct meaningful Deep RL experiments using policy gradient methods (https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewPaper/16669). I think this paper could benefit from the recommendations made there.\n\n\n  More questions/clarifications:\n  \n  Q1: What constitutes success in the grid world domains for table 1?\n\n  Q2: If you were to train a single policy using your reward shaping scheme (i.e. not the tiered approach currently employed) how would the new policy perform? Are there problems with the current scheme where you have to have the tiered policy?\n\n  Q3: Why CPC and not say an autoencoder or VAE? A comparison over many types of unsupervised embedding learning algorithms could be interesting and make your method more general than currently presented. It could also strengthen your argument for using CPC.\n\n  Q4: How long were the trajectories for training CPC?\n\n  *Other minor comments not taken into account in the review*\n  - I think your labels are backwards in table 1.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJg3kV22FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a reward shaping method which aim to tackle sparse reward tasks. The paper first trains a representation using contrastive predictive coding and then uses the learned representation to provide feedback to the control agent. The main difference from the previous work (i.e. CPC) is that the paper uses the learned representation for reward shaping, not for learning on top of these representation. This is an interesting research topic. \n\nOverall, I am leaning to reject this paper because (1) the main contribution of the paper is not clear (2) the experiments are missing some details and does not seem to support the claim that the proposed methods can tackle the sparse reward problem. \n\nFirst of all, it would’ve been better to have a conclusion section, so the readers can see the contributions of the paper. After reading the paper, I still do not understand what are the contributions of the paper and what're from the previous works. The paper does not provide well justification why CPC feature can provide useful information for reward shaping. The paper does not provide a new method to learn predictive coding. It does not provide a novel reward shaping method (the “Optimizing on Negative Distance” method is very similar to [1]). So, I am not sure what’re the contributions of this paper. \n\nMoreover, I am not convinced that the proposed method can tackle long horizon and sparse reward problems. As the paper discuss in introduction, learning in sparse reward environment is hard because it relies on the agent to enter the goal during exploration. However, the proposed approach seems only able to work in environments where exploration with random policy can generate trajectories that contain sufficient environment dynamics (e.g. dynamics near the goal states). How can the method learn that information without entering the goal? \n\nFurthermore, it seems that the proposed approach only works for goal-oriented tasks (since we need to know the goal state for reward-shaping). I think this should be clearly stated in the paper. \n\nThere are some missing details which makes it difficult to draw conclusions: \n1. How is the ‘success rate’ computed (e.g. in figure 7 and table 1).\n2. How were the parameters selected (e.g. table 5 in the appendix). Why did you use the default the parameters?\n3. How many runs are the curve averaged over and what’s the shaded region (e.g. one standard error)? Most of the results in the paper seem not statistically significant. \n4. In figure 4, five domains are mentioned but only three of them are tested in the section 5.\n5. Section 6.2 seems irrelevant to the paper. What’s the purpose of this section?\n6. Figure 10 shows the result of using CPC feature directly vs. reward shaping. Are both feature using the same NN architecture, same PPO parameters, and same control setting? Moreover, the reward shaping method assumes we know the goal state but using CPC feature does not. Is it a fair comparison?\n\nThe paper has some imprecise parts:\n1. The definition of MDPs (in section 2) is imprecise. For example, how is the expectation defined? how is the initial state sampled? What does $p\\in\\mathcal{P}$ (last line in the first paragraph) mean where $\\mathcal{P}$ is the state transition function? \n\nMinor comments which do not impact the score:\n1. Figure 1 should come before figure 2. \n2. It would have been better if there is a short description of how the hand-shaped rewards is designed for each domain in the main text. \n\n[1] The Laplacian in RL: Learning Representations with Efficient Approximations\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byeo7VT3tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method to derive shaping rewards from a representation learnt with CPC. They propose learning a CPC representation from some random data and fix it. They assume exploration is not too difficult so that rewards are achievable without additional mechanisms. Once a reward is achieved they can compute the embedding of the corresponding state as a goal under the CPC learnt representation either directly ie. distance or via a clustering step. The paper is clearly written and presents a nice idea. The paper is correct and part of the approach seems novel. I find the analysis of the results well executed though I think that they should be improved for publication.\n\nMajor points: \n* I think this is a valid contribution and it seems like something the RL audience might be interested in. \n* I am very surprised that CPC does such a good job given that the main object for learning in CPC is the distribution of trajectories which should be quite different in random exploration and the optimal policy. I presume this is because the environments are quite simple. This issue is of interest to the readers so an example of a simple failure case should make that point. \n* Secondly, as nice as those embeddings in figure 5 look I wonder what happens in larger mazes with more structure i.e. somewhere where random walk will not be a uniform distribution and thus CPC will (most likely) not work as intended.\n* The clustering bonus how do you prevent it from staying at the edge of the goal region and deriving infinite rewards from that ?\n* Since these are not potential functions how do you prevent the rewards from biasing learning ? Ng et al. -- Policy invariance under reward transformations. This should be discussed in the paper because it is of practical importance.\n* Contrastive learning has been used to find goal embeddings before Warde-Farley et al. Unsupervised control through non-parametric discriminative rewards. In that paper they don't need the CPC future state predictors but instead contrast the goal and the final state of the trajectory. They use the resulting embedding to learn a reward function and ignore the extrinsic reward. Interestingly, they show that the rewards can be learnt online (maybe ideas from that paper can be applied here).\nMinor points:\n* please make legends on plots readable size.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use the representation learned via CPC to do reward shaping via clustering the embedding and providing a reward based on the distance from the goal.

The reviewers point out some conceptual issues with the paper, the key one being that the method is contingent on a random policy being able to reach the goal, which is not true for difficult environments that the paper claims to be motivated by. One reviewer noted limited experiment runs and lack of comparisons with other reward shaping methods.

I recommend rejection, but hope the authors find the feedback helpful and submit a future version elsewhere.
Decision: Reject