Temporal abstractions-augmented temporally contrastive learning: an alternative to the Laplacian in RL
Authors: Akram Erraqabi
Marlos C. Machado
Harry Zhao
Sainbayar Sukhbaatar
Ludovic Denoyer
Alessandro Lazaric
Yoshua Bengio
Venue: ICLR 2022 Submitted (2022)
Keywords: Representation learning
Laplacian
self-supervised
exploration

Abstract: In reinforcement learning (RL), the graph Laplacian has proved to be a valuable tool in the task-agnostic setting, with applications ranging from option discovery to dynamics-aware metric learning. Conveniently, learning the Laplacian representation has recently been framed as the optimization of a temporally-contrastive objective to overcome its computational limitations in large or even continuous state spaces (Wu et al., 2019). However, this approach relies on a uniform access to the state space S, and overlooks the exploration problem that emerges during the representation learning process. In this work, we reconcile such representation learning with exploration in a non-uniform prior setting, while recovering the expressive potential afforded by a uniform prior. Our approach leverages the learned representation to build a skill-based covering policy which in turn provides a better training distribution to extend and refine the representation. We also propose to integrate temporal abstractions captured by the learned skills into the representation, which encourages exploration and improves the representation’s dynamics-awareness. We find that our method scales better to challenging environments, and that the learned skills can solve difficult continuous navigation tasks with sparse rewards, where standard skill discovery methods are limited.

Under review as a conference paper at ICLR 2022
TEMPORAL ABSTRACTIONS-AUGMENTER TEMPORALLY CONTRASTIVE LEARNING: AN ALTERNATIVE TO THE LAPLACIAN IN RL
Anonymous authors Paper under double-blind review
ABSTRACT
In reinforcement learning (RL), the graph Laplacian has proved be valuable tool in the task-agnostic setting, with applications ranging from option discovery to dynamics-awan metric learning. Conveniently, learning the Laplacian repre- sentation has recently been framed as the optimization of temporally-contrane objective to overcome its computational limitations in large or even continuous state spaces (Wu et al. 2019). However, this approach relies to the state space S, and overlooks the exploration problem that emerges during the representation learning process. In this work, reconcile such representa- tion learning with exploration in a non-uniform prior setting, while recovering the expressive potential afforded by a uniform prior. Our approach leverages the learned representation to build a skill-based covering policy which in turn pro- vides better training distribution to extend and refine the representation. We also propose to integrate temporal abstractions captured by the learned skills into the representation, which encourages exploration and improves the representation's ss dynamics-awarenes,
uniform access
We find that our method scales better to challenging en- vironments, and that the learned skills can solve difficult continuous navigation tasks with sparse rewards, where standard skill discovery methods are limited.
INTRODUCTION
Representation learning has been at the core of many recent machine learning advances (c.f. [Bengio et 2013). With the advent of deep reinforcement learning (Mnih et al. 2015), representation learning has also become one of the main topics of interest in reinforcement learning (RL). For example, in the goal-conditioned hierarchical setting (Vezhnevets et al. 2017 Nachum et al. 2019a),
one learns a representation that maps observations to an abstract space, the representation space, in which the higher-level policy defines the desired behavior of the lower-level policy. Distance in the representation space can then be used to reward and guide the lower-level policy towards specific goal states. Moreover, environments with rich observations and complex dynamies (e.g., [ellemaal eet al. 2020) have motivated recent works on learning representations as controllable or contingent features (Bengio al. 2017 Choi al. 2019), on top of which one can potentially learn latent models in the perspective of planning (Hainer et al.] [2019b] Nasiriany et al., 2019 Schrittwieser al. 2020) and control (Watter al. 2015 Banijamali al., 2018 Hafner et al. 2019a).
In this work, we are interested in the aggori- setting in which an RL agent first interacts with the environment to build a representation, ø, of the state space, S, without relying on any task-specific reward signal. This representation can later be used to solve task posed in the envi- ronment the form of reward function. In this setting, the environment dynamics are the only informative interaction channel available to the agent. This has naturally motivated graph Laplacian- based methods to address the task-agnostic phase; where the graph vertices correspond to the states and its edges to the transitions probabilities. The Laplacian's 's eigenvectors can been leveraged holistic state representation, termed the Laplacian representation, which captures the environment's dynamics structure and geometry (Mahadevan 2005 Mahadevan & Maggioni] 2007).
Wu et al. (2019) recently proposed an efficient approximation of the Laplacian representation (LAP- REP) by framing the graph drawing objective temporally-connrastive loss (see Section 2.2

---

Under review as a conference paper at ICLR 2022
While this formulation works around potentially prohibitive eigendecomppsitions, which extends the representation's 's applicability to large and continuous state spaces, it assumes access to a uniform sampling prior over S. In practice, this translates in the ability to reset the agent to uniform random starting state in the environment, which artificially alleviates the exploration problem. As we will show in Section + the uniformity of that prior is crucial for the quality of the learned representation. However, such sampling is not trivial in the absence of the uniform prior privilege, since the agent has to learn to explore the state space to be able access arbitrary states. In effect, one must handle the exploration along the representation learning in order to preserve the representation's 's quality. In this work, we propose a representation learning framework that conciliate a similar temporally- contrastive approach with exploration in the task-agnostic setting.
In practice, the representation is trained on data collected with a uniformly random policy, Tp (random walk trajectories). Without uni- form access to the state space, the collected data is concentrated around accessible starting states. To achieve better data collection, tie the representation learning problem to that of learning covering strategy. Briefly, our method consists in using the available repre- sentation to learn a skill-based (hierarchical) covering policy that is in turn used to discover yet unseen parts of the state space, provid- ing novel data to refine and expand the rep- resentation. Our approach, illustrated in Fig- ure [1, is inspired by the cyclic option discov- ery framework (Machado) 2019), which moti- vated several related methods (Machado et al.] 2017] 2018]] Jinnai et al.) 2020). In addition, propose to integrate the temporal abstrac- tions learned by the skills in the contrastive representation learning objective encourage temporally-extendee exploration and enforce i.e.
Train radiom walk trajectanies
@ -Skil trajectory
RRnndom Walk
(d) Using the updateed extend the skills the newly discovered arca
(b) Use the lesmed train skills travel across the explored area
(e) Use skills to discover regions and update o
Our representation is trained to encode the area that the agent has learned to cover. Skills are continuously trained on the representation to discover new areas where novel data is collected to refine the representation, progressively extend- ing its coverage. Similar incremental discovery is at the core of several works (Ecoffet et al., 2021 Pong et al. 2019 Machado 2019).
Figure 1:
the representation's dynamicx-awarenest how representative the ø-induced euclidean dis- tance is of distances in the state space.
We empirically show our agent's 's ability to pro- gressively explore the state space and to consistently extend the representation covered domain in non-uniform prior setting. We show that our representation leads to better value predictions than LAP-REP, and that it recovers the representation quality expected from a uniform prior. We also evaluate our representation in shaping rewards for goal-achieving tasks, and we show it outper- forms LAP-REP, confirming both its superior ability in capturing dynamics and in scaling to larger environments. Finally, the skills learned in our framework also prove to be successful at difficult continuous navigation tasks with sparse rewards, where other standard skill discovery methods are limited.
2 PRELIMINARIES
2.1 TASK-AGNOSTIC REINFORCEMEN LEARNING
We describe a task-agnostic RI environment as task-agnostic Markov decision process (MDP)
M = AS da) where S is the state space, A the action space, P S x A + A(S) the transition dynamics defining the next state distribution given current state and action taken, 7 e [0, 1) the discount factor, and d, the initial state distribution. A policy : S - (AA) maps states s E S to distributions over actions.
Knowledge acquired from task aanosigo interactions with the environment (e.g., a representation or a policy) can then be leveraged for specific tasks. A task is instantiated with a reward function, R S R. which is combined with the task-agnostic MDP. The task objective is to find the
2

---

Under review as a conference paper at ICLR 2022
optimal policy maximizing the expected discounted return, En,do do EE,r ai)] starting from state So dg and acting according to at TT-)s))).
2.2 THE LAPLACIAN REPRESENTATION
The Laplacian representation (LAP-REP), proposed by Wu et al. (2019) can be learned with the following contrastive objective:
LLap (x,)) (u.0)~Dry
[ll((u) < +B E [(ő(u) o(v)) l (( (1)
u-Dapp
where is hyperparameter, Tp the uniformly random policy, p. set of trajectories from Jy (random walks). We use (u, v) ~ Dr, to denote the sampling of a random transition from Dr,, and similarly u Dap for a random state, Wu al. (2019) showed the competitiveness of the Laplacian representation when provided with uniform proor over S during the collection of DA,. Their objective (Eq. [ is a cemporally-contrastive loss: it is comprised an attractive term that forces temporally close states have similar representations and repulsive term that keeps temporally far states' representations far apart. Here, the repulsive term was specifically derived from the orthonormality constraint of the Laplacian eigenvectors.
2.3 THE NON-UNIFORM PRIOR SETTING
In RI., representation learning is deeply coupled to the problem exploration. Indeed, the induced state distribution defines the representation's 's training distribution. However, LAP-REP (Wu al. 2019) has been learned in the specific uniform prior setting that alleviates the exploration challenge. In this setting, Dr,, from Eq. T collection of random walks with uniformly random starting states, which provides uniform training distribution to the representation learning objective. In the case of non-uniform prior, the induced visitation distribution can be quite concentrated around the start state distribution when solely relying random walks, hence the need for an exploration strategy for a better covering distribution.
To study the problem described above, we investigate the setting in which the environment has fixed predetined state 5p to which resets with a probability p, every steps; with K of the order of diameter of S. With uniformly random behavior policy, this setting is equivalent to a initial state distribution that concentrated around Su and whose density decays exponentially away from it. We will refer this setting as the non-uniform prior (non-p4 setting, opposed to the uniform prior (1)) setting where the agent has access to the uniform state distribution.
TEMPORAL ABSTRACTIONS-AUGCAAAA
REPRESENTATION LEARNING
In this section, present Temporal Abstractions-augmenice (TATC), a representation learning approach in which the representation works in tandem with a skill-based covering policy for better representation learning in the non-uniform prior setting. We first propose an alternative objective to Eq. [ that suits this setting, then describe the exploratory pol- icy training. Finally, we introduce an augmentation the proposed objective based on the learned temporal abstractions to improve exploration and enforce the representation' 's dynamics-awand awareness...
Contrastive learning
3.1 CAMPORALLY-CONTRASEE
REPRESENTATION OBJECTIVE
mentioned in Section 2.2 the repulsive term in LAP-REP'S objective (Eq. T derives from the eigenvectors' orthonormality constraint. However, because the environment is expected to be pro- gressively covered the non-uniform prior setting, the orthonormality constraint can make online representation learning highly non-stationary]I For this reason, adopt the following objective with a generic repulsive term that is more amenable to online learning:
Leant (d:; D.,.)) (u,
E
[l]c(u) ((v) UND
(exp(-ll(ux) -(v((())()))l))))
D
In general, even within given matrix's 's perturbation neighborhood, its eigenvectors show highly nonlinear sensitivity (Trefethen & Bau 1997).
3

---

Under review as a conference paper at ICLR 2022
3.2 REPRESENTATION BASED COVERING POLICY
In the non-uniform prior setting, exploration is required to provide the representation with a better training distribution. To this purpose, adopt hierarchical RL approach leverage the ex- ploratory efficiency of options (Sutton et al.] 1999 Nachum et al.) 2019b), or skills. The agent acts according to a bi-level policy (Thi Tlow). The high-level policy Thi S + A(12) defines, at each state 8, a distribution over a set 52 of directions (unit vectors) in the representation space (5 {8 5 E R ([=1|==11) Each direction corresponds to fixed length skill encoded by the low-level policy Tlow S x f2 - A(A). These skills are expected to travel in the representation space along the directions instructed by Thi In short, given a sampled direction mh(()s) ~ å E Ω the low-level policy executes the directional skill Tow (- 8) for fixed number of steps before new direction is sampled.
Now, we describe the intrinsic rewards used to train the policies Thi and Mow.
Low-level Policy.
Mow is simply trained to follow directions defined by Thi in the representation Rd, the corresponding skill Tlow (-Is, 5) is trained to maximize the
space. For given 5 E Ω intrinsic reward function:
r (s, cos( (os')) x(s) 8 (6((5)) d(s))
(3)
| o(s)
where (8, is an observed state transition, and ø the representation being learned. We use the cosine similarity a way encourage learning diverse directional skills. Indeed, skills co- specialization is avoided by rewarding the agent for the steps induced along the instructed direction ò regardless of their magnitudes.
High-level Policy. The high-level policy is expected to guide the covering strategy. It should do so by sampling the skills of the most promising directions in terms of exploration: affording new discoveries while avoiding to spend more time than needed in previously explored areas. For this purpose, we design a reward function defined over a sequence of consecutive skills. Let (shi k== be the sequence of their initial states and their respective sampled directions 6д (- ). Since ó is trained to capture the dynamics, the travelled distance in 6's space is good proxy of how far the choices made by Thi eventually brought the agent in the environment. Therefore, for given high-level trajectory, بار = (sT shi ......, S2: ), with s/i the final state reached by the last skill, the ,
high-level policy is trained to maximize the following quantities:
Vk E (l,..,, L}, 1eth (s 8x) = (1x())) (l)
where 5R mai(-1sh) is the direction sampled at sh From the policy optimization perspective, each of these quantities plays the role of the return cumulated along the sampled high-level trajectory and not just single (high-level) step reward. This term looks at reaching s as the result of sequential collaboration of L skills, rewarding them equally. It values how far this sequence of skills has eventually brought the agent.
These policy training choices are closely related to how the representation is trained. Indeed, the exploratory behavior emerges from the interaction between the policy and the representation while training. In the following section, we describe how the representation benefits from the temporal abstractions learned by the covering policy (This Mow).
3.3 AUGMENTING REPRESENTATION LEARNING WITH TEMPORAL ABSTRACTIONS
A skill abstracts temporally-extendee and structured behavior in a single action. As temporal abstraction, it represents a factorized knowledge of the environment dynamics in the form of a policy (the directional skill policy). Here, we propose to leverage these temporal abstractions in the representation objective in the favor of better capturing the environment dynamics. In order preserve the temporal contrast of the base objective (Eq. 2) we augment it with the following contracting term along the skills trajectories:
Σ [(((s)) (s
B(Φ; D.) =
E T&
(5)
D
4

---

Under review as a conference paper at ICLR 2022
where D3 is set of collected skills trajectories. By minimizing this term, ø integrates temporally- extended dynamics: areas connected by skills are brought closer in the representation space. This term will be referred to as the boredom term due to its exploratory virtue, explained in the following.
How does boredom help exploration ? The interaction of the high-level policy reward function (Eq. q and this boredom term (Eq. 55) induces progressive exploration mechanism. In effect, Thi samples skills that travel further more often, i.e. with larger Rhi. The more often a skill is sampled, the less rewarding becomes due the minimization of B(ő). This will increase the probability of sampling the remaining under-sampled skills, hence encouraging the exploration of less visited parts of the state space. In short, the interplay between the policy and the representation dynamically fights what can be considered accumulated boredom along over-sampled skills trajectories which increases the agent curiosity and urge to explore.
Finally, the proposed objective to train the representation & consists in the objective in Eq. 2 aug- mented with the boredom term (Eq. 5). and can be written as
CTATC D,, Px,.) (oo Dr,,) B'B(;; Dz)
(6)
with 3' hyperparameter controlling the strength of boredom term. A detailed description of TATC training is available in the Appendix A (Algorithm 1.
4 EXPERIMENTS
In this section, investigate the behavior of TATC in two types of environments: gridworld en- vironments with discrete state and action spaces, and a continuous navigation environment (Mu- JoCo, Todorov et al. (2012)) for continuous state and action spaces. Implementation details of all the experiments in this section can be found in Appendix
4.1 GRIDWORLD
For gridworld environments, we evaluate our approach in three different domains: U-MAZE, T- MAZE and 4- 4-ROOMS. These environments visualized in Figure 2 raise different explorations chal- lenges. U-MAZE is the simplest but the most relevant environment test the dynamics-awaremt of the representations" T-MAZE raises the challenge of splitting the exploration focus at an intersec- tion while maintaining exploration and coverage both corridors; 4-ROOMS similar to U-MAZE, but requires learning more controlled skills to efficiently move from one room to another.
(a) U-Maze 30
(b) T-Maze 40
(c) 4-rooms 21x
Figure The gridworld domains with the fixed initial state so highlighted in red.
The states are one-hot encoded such that no positional information is provided to the agent. For our method, we learn a representation (d = 2), and define Ω as a set of 8 unit vectors equally spaced on the unit sphere (see Appendix C
4.1.1 PROGRESSIVE REPRESENTATION LEARNING
Figure 3 shows the progression of the representations throughout training. The agent progressively explores the environment starting from Se, builds the representation by continuously integrating newly discovered parts.
showss
"The presence of the wall makes 1.2 1.2-distance in xy-coordinates deceptive. The 1.2-distance in dynamics- aware representation space should correct for that.
5

---

Under review as a conference paper at ICLR 2022
(a)
(b)
(c)
(d)
(e)
(g)
(h)
(a)
(b)
(c)
(d)
(e)
(g)
(h)
(a)
(b)
(c)
(d)
(e)
(n
(g)
Figure 3: TATC representations learned throughout the training. Axes scales were equalized. Top row (R1): U-MAZE. Middle row (R2): T-MAZE. Bottom row (R3): 4-ROOMS. The colors reflect the distances in terms of the dynamics. They can be seen as quantities proportional to the length of the shortest path from so (marked in red) to the represented state.
U-MAZE. The agent starts from the bottom left corner of the maze. Figure 3 shows how the rep- resentation progressively expands until reaching the first corner (Fig R RI.a-c.. During this phase, the agent learns skills to travel further away from $o along the corridor. At this stage, the rest of the environment still not explored. The remaining phase (Fig RI.f-j shows not only the com- plete discovery of the corridor but also the flattening of the full domain representation: placing the last corner further from the starting corner than the intermediate ones indicates the representation's ' success in capturing the maze dynamics.
The agent starts from the bottom left corner of the maze. As in the U-Maze, it starts learn- ing travel along the corridor (Fig R2,a-b) until reaching the intersection. There, the exploration focus is shared between both possible paths whose representations are progressively disentangled (Fig 3 R2.c-f). Eventually, the agent fully explores both corridors and finalizes its representation. Note that, the discovery of one of the corridors did not hinder finishing the discovery of the other. The boredom term proved to be important for such property (see Appendix
T-MAZE.
The agents starts in the first room. It progressively discovers and learns about other rooms. Once the domain is fully explored, and similarly to UZE, the representation straightens, reflecting a holistic understanding of the environment dynamics.
4-ROOMS.
Boredom Ablation Study. Appendix provides an ablation study showing the importance of the boredom term for the agent's ' exploratory behavior and the representation's dynamics-
4.1.2 EVALUATING THE LEARNED REPRESENTATION
We now compare our representation against LAP-REP (Wu et al. 2019) in the non- non-uniform prior setting. First, appreciate the sensitivity of LAP-REP the uniformity of said prior, we trained LAP-REP in two settings: (i) the uniform prior setting where the agent can be set any arbitrary state, as done by Wu et al. (2019), (ii) the non-uniform prior setting defined Section R We show that LAP-REP quite sensitive to this change in distribution while TATC recovers the expressive potential of LAP-REP learned with uniform prior.
Prediction. To evaluate the learned representations, we first consider how well they linearly approx- imate a given task's 's optimal value function. To do so, we train an actor-critic agent (Mnih et al. 2016) with a linear critic top of each representation. In Figure we note significant loss in the expressive power of LAP-REP when the access to the state space is not uniformly distributed any- more. This figure also shows that TATC outperforms LAP-REP in the non-uniform prior setting, and

---

Under review as a conference paper at ICLR 2022
TATC (ours)
LAPRREP, nonr prior, d--2
LAP REP, nonz peior, d-3
LAPREP, proor
Maze
T-Maze
4rooms
15.0
15.0
15.0
12.5
12.5
7.5
7.5
7.
5.0
5.0
5.0
0.0
0.0
0.0
epochs
Figure 4: Learned representation's ability to approximate the value function. LAP-REP was learned in the same non-uniform prior setting (non-j() with d = 2 and d = (no improvement was observed for higher values). The dashed line gives the performance of LAP-REP in the uniform prior setting (μ). TATC outperforms LAP-REP in non-/u setting, and succeeds in recovering its expressive power when learned from the uniform prior. Performances were averaged over different runs.
succeeds in recovering LAP-REP's expressive power when it is learned with the unrealistic uniform prior.
Control. We also compare the representations from the perspective of control, by training deep actor-critic agent on top of each representation to solve goal-reaching task in the same domains as above. The agent only rewarded upon reaching the goal state (r 1). Figure shows Shoows that TATC consistently outperforms LAP-REP, which confirms the competitive quality our representation.
TATC (ours)
LAP-REP, non-H prior, d=2
LAP-REP, 2015 prior, d=3
U-Maze
T-Maze
4rooms
Egisodd Beward
0.6
400
500
epochs
Figure 5: Control performance (episode reward) in the fixed initial state setting (non-uniform prior). Performances were averaged over 5 different runs.
4.2 CONTINUOUS CONTROL
The second set of experiments, which focuses continuous state and action spaces, is conducted on AntMaze which is essentially a MuloCo counterpart of U-MAZE where a four-legged agent has learn control its joints maneuver along U-shaped corridor.
To visualize our learned representation in this enviromment, Figure 6 depicts a grid of positional states in the environment domain and their mapped representations. Similarly U-MAZE, the leamned representation translates the environment dynamics by placing the end of the corridor (top left) away from the initial state (represented in red) than the intermediate corners (top and bottom right).
4.2. REWARD SHAPING WITH LEARNED REPRESENTATION
We first demonstrate how our learned representation is able to improve an RL agent's performance when the distances in the representation space are used for reward shaping, the same setting which Wu et al. (2019) evaluated LAP-REP. We define goal-achieving task by setting goal state g the end of the corridor (top left). The objective is to learn to navigate to state s close enough to the goal area []|s g la < e). We define the reward function based the distance in representation space (TATC and LAP-REP). More specifically, we train a soft actor-critic (SAC) agent (Haarnoja et al.] 2018) to reach the goal with dense reward defined as pensee -110(8++1) Φ(g)]]2. Similarly

---

Under review as a conference paper at ICLR 2022
1.0
apisode sucouss
0.8
0.6
tours)
0.4
TATC-mit (ours)
LAP- RER---eds
0.2
LAP REP wi 3o0-g)
Figure 6: TATC learned representation visu- alized on a grid of positional states. Colors reflect the distance in the representation space from the initial state, highlighted in red. Axes scales were equalized. We can visually appreci- ate how the U-shaped continuous state domain is mapped to flatter manifold reflecting the presence of the wall.
0.0
steps
1e6
Figure 7: Results of reward shaping using learned representations: Performances were av- eraged over 5 different runs and then exponen- tially smoothed (0.9) for better visualization.
Wu al. 2019). we also compare against the half-half mix of the dense reward and the sparse reward PR 0.5 pdense + 0.5 Il [|| gll2 c).
For this evaluation, we used larger environment than those used by Wu al. (2019), making it more challenging task. Unlike our representation, LAP-REP was learned with a uniform prior over as in Wu et al. (2019), but with d = 2 instead of d = 20. 7 that our representation effective in reward shaping, with both mix and dense variants and enjoys comparable if not superior mnamics--waarene
to LAP-REP, REP, even when TATC learned from a non-uniform prior and AAP-REP is learned from uniform one. Note that LAP-REP with non-uniform prior is unable to guide the agent to success.
This result further confirms the conclusions drawn from the GridWorld experiments (Section 4.1.2) and positions TATC as competitive alternative to LAP-REP in this difficult setting.
4.2.2 THE LEARNED SKILLS
We evaluate the exploratory potential of TATC's skills. Here, we compare the learned skills against task-agnostic skill discovery methods, DIAYN (Eysenbach et al. 2019). and DCO Tinnna et al.) 2020). DIAYN learns diverse set of skills by maximizing mutual in- formation between skills and states. Similarly to our skills, DCO training is also based on lemporally-contrastte
oposide success
success
0.6
TATC (ours)
representation. In this case, DCO requires a pretrained I.AP-REP (that approximates the Laplacian's second eigenvec- tor). We train the required representation, as well as DCO, with the advantage of data col- lected from uniform prior over S. For fairness, we train 8 skills for both methods (DCO and DIAYN).
DIAYN
DCO
0.0
10
20
50
epochs
Figure 8: Skills Evaluation: performance gath- ered from 5 independent runs and then exponen- tially smoothed (0.9) for better visualization.
Once trained, the skills learned by each method are fixed and used to train a discrete high-level policy that can select across the available skills to solve a goal-reaching task with a sparse reward function r: ||++ - gl12 S e]. The sparsity of the reward naturally poses challenge as no additional signal can guide the agent towards the goal, unlike the evaluation setting of DCO and DIAYN by Dinnai et al. (2020). The results (Figure show that the skills learned by TATC quickly assist to complete the task while the skills learned with DCO and DIAYN do not. DIAYN's 's limited performance in difficult sparse-rewae navigation tasks was also confirmed by Kamienny al, (202T). These results suggest that in order succeed, DCO
8

---

Under review as a conference paper at ICLR 2022
and DIYAN skills may require richer signal like the distance-based dense reward used by Jinnai et al. (0200) to evaluate both them and where they show similar performances.
5 RELATED WORK
The potential of graph Laplacian representations in capturing functional information about the en- vironment has motivated their use in the task-agnostic RL setting (e.g.. MMahadevan 2005 Machado
et al. 2017: 2018). These are powerful tools that proved to scale to the continuous case (Wu et al., 2019 Machado et al.; 2018, Tinnai al., 2020). While these recent works proved to learn useful representations, they overlooked the challenging exploration problem that emerges when collecting the representation training data. Our framework proposes an alternative approach that explicitly couples the exploration challenge with the representation learning objective.
This work also relates to sulf-supervised learning (Bromley al., 1994, Chopra et al. 2005 J which brought recent advances in representation learning (Bachman et al. 2019; He et al. 22200; Chen al. 2020, Grill al., 2020; Caron et al., 2020). These techniques have naturally been adapted to RI, especially contrastive methods. While some of these benefited from visually contrasting observations (Laskin et al. 2020; Yarats et al.) (022I), others leveraged temporal contrasts to learn representations (Mazoure al. 2020; Stooke et al. 2021 Li et al. 2021), which fall closer to our work.
We designed our covering policy hierarchical agent. This has actually been the default setting to model temporally-extendeee
actions (Sutton et al.] 1999). Our work shares the same motivation as Vezhnevets et al. (2017) for training skills to follow latent directions. Among the large body of work on skill discovery, the eigenoptions framework (Machado et al. 2017) and its extensions (Machado al. 2018 Jinnai al., 2020) are probably the closest to skill training scheme. Moreover, Eigenoptions also fit in the directional skills definition as they are trained to travel along the di- rections defined by the eigenvectors of the Laplacian (dimensionality of [S], potentially large). To contrast, we train directional skills defined by an arbitrarily diverse set of directions in the learned representation space (small dimensionality) to progressively learn about the environment. The incre- mental discovery paradigm has been previously adopted. either for exploration (Ecoffet et al.) [202T) incremental skill discovery (Jinnai et al., 2020 Pong et al., 2019), or even state abstraction (Misra et al. 2020). Finally, we use learned skills to penalize boredom (Schmidhuber) 1991; Oudeyer et al.) 2007 Oudeyer & Kaplan 2009) in the representation space and encourage exploration. The idea of using skills to foster curiosity has also been investigated by Bougie & Ichise (2020).
CONCLUSION
The Laplacian representation proposed by Wu et al. (2019) made the benefits of spectral meth- ods affordable in large state spaces where function approximation is required. Unfortunately. this representation's 's quality is strongly tied to the uniformity of its training data distribution, as shown is Section This has motivated the method proposed in this work where we reconcile similar cemporally-contrastnn
representations with exploration demanding settings. Our approach lever- ages the practical skills' training that such representations allow, and uses the learned skills to better cover the state space and hence learn a better representation. In addition, we propose augment the temporal contrast-based representation objective with temporal abstractions captured by the acquired skills. This has two benefits: enforces the representation's 's dynamics-awarenes. and contributes to exploring the environment by inducing poredom-fighting curiosity in the covering policy. We validate our method in tabular as well as continuous environments. Our representation learned in non-uniform prior setting shows a comparable representational power to the one acquired from a uniform prior, and proves to scale well to challenging settings. Moreover, the skills that emerge from our algorithm show competitive performance in hard continuous control tasks with sparse wards where standard skill discovery methods fail. Thus, with these results, we hope to bring such representations' applicability one step closer to realistic contexts.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. NeurlPSS2019, (rXxv:1906.00900, 2019.
Ershad Banijamali, Rui Shu, Hung Bui, Ali Ghodsi, al. Robust locally- -ineer controllable em- bedding. In International Conference on Artificial Intelligence and Statistics, pp. 1751-1759, 2018.
Mare G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Sub- hodeep Moitra, Sameera S. Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77-822 2020.
Emmanuel Bengio, Valentin Thomas, Joelle Pineau, Doina Precup. and Yoshua Bengio. Indepen- dently controllable features. arXiv preprint arXiv: 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 3588):1798-1888, August 2013. ISSN 0162-8828. doi: 00.100//PAM..2013.000 URL http::://oo..oo////0///// 1ttp:/10..1111PPPP 2013.50
Nicolas Bougie and Ryutaro Ichise. Skill-based curiosity for intrinsically motivated reinforcement learning. Machine Learning, 109(3):493-5)) -129335512221 2020.
Jane Bromley, Isabelle Guyon, and Yann LeCun. Signature verification using a" siamese" time delay neural network. 1994.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato. R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924. Curran Asso- ciates, Inc., 2020. URL https::/tppptttttttl 770fb6226991 6e0238 /ttp:////cceeeee........ neur neur pps. cc httppppprr/////11111 .pdf]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. Contingency-aware exploration reinforcement learning. 7th International Con- ference Learning Representations, ICLR 2019, pp. 1-19, 2019.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR' 055), volume 1, pp. 539-546. IEEE, 2005.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return then explore. Nature, 590 7847:580-555 08600055867778 2021.
Benjamin Eysenbach, Julian Ibarz, Abhishek Gupta, and Sergey Levine. Diversity all you need: Learning skills without reward function. 7th International Conference on Learning Represen- tations, ICLR 2019, 2019.
Jcan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec. Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 2019a.
10

---

Under review as a conference paper at ICLR 2022
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp.
2019b.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6d separate, adaptive learning rate for each connection. 2012.
Yuu Jinnai, Jee Won Park, Marlos C. Machado, and George Konidaris. Exploration in reinforcement learning with deep covering options. In International Conference Learning Representations, 2020. /
net./ www.:eeeeeooomee-eeeeeeeee..e
Viere-AAexandre Kamienny, Jean Tarbouriech, Alessandro Lazaric, and Ludovic Denoyer. Di- rect then diffuse: Incremental unsupervised skill discovery for state covering and goal reach- ing. In ICML 2021 Workshop on Unsupervised Reinforcement Learning, 2021. URL https: //openreview.. 2ete://oooommm21e2--6362
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, PMLR 119, 2020. arXiv:2004.0413:
Siyuan Li, Lulu Zheng, Jianhao Wang, and Chongjie Zhang. Learning subgoal representations with slow dynamics. In International Conference Learning Representations, 2021. URL
hettps://openreeee
.et/ wxwp://ooommmldddwwwbooeeeoooeee
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint MiXi:1509000977 2015.
M Cholodovskis Machado. Efficient Exploration in Reinforcement Learning through line-Based Representations. PhD thesis, University Alberta, 2019.
Marlos C. Machado, Mare G. Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. 34th International Conference on Machine Learning, ICML 2017, 5:3567-3582, 2017.
Marlos C. Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. 6th International Conference on Learning Representations, ICLR 2018 Conference Track Proceedings, pp. 1-22, 2018.
Sridhar Mahadevan. Proto-value functions: Developmental reinforcement learning. ICML 2005 Proceedings of the 22nd International Conference on Machine Learning, pp. 553-560, 2005. doi:
10.114 1102421.
Sridhar Mahadevan and Mauro Maggioni. Proto-value functions: A laplacian framework for learn- ing representation and control in markov decision processes. Journal of Machine Learning Re- search, (774):2169-223)) 2007. URL http:/ /mir imlr wtgppapers/ wwt.////aaaaaaaaa.ammmm
html
Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin pp. 3686 3698, Curran Associates, Inc., 2020. URL wttps::tttttt..tt..ttt / http://ppppooeedinon.eeeee......
in Neural Information Processing Systems, volume 33,

---

Under review as a conference paper at ICLR 2022
Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state ab- straction and provably efficient rich-observation reinforcement learning. In ICML 2020, July 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. 2015.
Nature, 518(7540):529-55555
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.). Proceedings of The 33rd International Conference Machine Learning. volume 48 of Proceedings of Machine Learning Research, pp. 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https: /pproceddings www..p.es.........mm.....m.....e press v48 mwwhalniillmmmmmmmmmmmmmmmmmmmmm html
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. 7th International Conference Learning Representa- tions, ICLR 2019, pp- 1 1-15, 2019a.
Ofir Nachum, Haoran Tang. Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work well in reinforcement learning? arXiv preprint 1909::0699.1111 2019b.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-condition policies. In Advances Neural Information Processing Systems, pp. 14814-14825, 2019.
Pierre-Yvel Oudeyer and Frederic Kaplan, What is intrinsic motivation? a typology of computa-
tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Pierre-Yve Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-286, 2007.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv: wrXiv:9903.0369, 2019.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neu- ral controllers. In Proceedings of the First International Conference on Simulation of Adaptive Behavior on From Animals to Animats, pp. 222-227, Cambridge, MA, USA, 1991. MIT Press. ISBN 0262631385.
Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, 2020.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learn- ing from reinforcement learning, 2021. URL https: SKUm2AJpvN
Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A frame- work for temporal abstraction in reinforcement learning. Artificial Intelligence, 1 11111),11-211, 1999. ISSN 00043702, doi: 10. 10.1016//00044377000000033333
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEFF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Lloyd Nicholas Trefethen and David Bau. Numerical Linear Algebra. SIAM. Philadelphia,
1997. ISBN 08987 13617 9780898713619,, numeri ca 1 nea htt.a//eeeaattttttoooooooooo/aaa
URL htw.lattpps/wwwooooddaaaaa.a
12

---

Under review as a conference paper at ICLR 2022
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference Machine Learning- Voluue 70, pp. 3540- 3549. JMLR. org, 2017.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances neural information processing systems, pp. 2746-2754, 2015.
Yifan Wu, George Tucker, and Ofir Nachum. The Laplacian in RL: Learning representations with efficient approximations, 2019.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. 2021.
13

---

Under review as a conference paper at ICLR 2022
A TEMPORAL DESTRACTTIONS-AUGEENN TEMPORALLY-CCOTRRSTIIE LEARNING (TATC) IN THE NON-UNIFORM PRIOR SETTING
The proposed approach consists in simultaneous training of the representation á and the hierar- chical agent (Tlww, Thi). The idea is to progressively extend the explored area while maintaining the previously collected knowledge. To do so, in the non-uniform prior setting, the agent switches with some probability Prw between following uniformly random policy xy and executing the hierar- chical policy (skills). The latter helps reach further areas, more efficiently, where data collected by Ty would be used to train the representation d. Along their training, the skills would progressively extend to reach newly discovered areas, advancing the exploration frontier. Algorithm C provides pseudocode of the proposed approach, in the non-uniform prior setting.
Algorithm TATC in the non-uniform prior setting
1: Input: L, C, Prw N
2: for ileration 1,2,... do
3: Dr. = 0, D
4: for batch 2...................... N do
5: Reset to So with probability p~ 1n))((
6:
7:
if p < Prw then
8:
Run the uniformly random policy тy to collect L random walk trajectories {T/ of steps each.
9:
D=, t DTT UT
10:
else
11:
Run (Tii klow) ) to collect L consecutive skills' trajectories {(Tk, 5x) and their cor- responding directions
12:
D. + D, U [(TK, 8z)
end if
14:
end for
Optimize the policies (Thi Flow) using their intrinsic objectives and (vanilla actor-critic
update)
Optimize & so to minimize LTXTE ( Da: D...)) (Eq. 6
17: end for
B REPRESENTATION OBJECTIVE AUGMENTATION: ABLATION STUDY
B. BOREDOM AUGMENTATION HELPS EXPLORATION
In order illustrate the importance of the proposed augmentation with the boredom term in the final objective (Eq. 6) we conducted the same representation learning experiments for the three gridworld domains in the non-uniform prior setting, but this time with the non-augmented representation learning objective (8' 0).
Figure shows how the agent failed at exploring the whole domain. In T-MAZE, focuses only on one corridor without getting curious about the other one. Regarding U-MAZE and 4-ROOMS, the agent stops exploring after discovering the end of the first corridor and the second room respectively. This is due to the lack of incentive to visit the yet unseen states, they less rewarding for Thi (i.e. closer in the representation space, hence smaller _ than the furthest explored state. The effect of the proposed augmentation would compress the representation of the explored area, say the first corridor in U-MAZE, which makes the rest of the environment more appealing to explore for mhi (i.e. further in the representation space, hence larger Rhi). This emphasizes the importance of the boredom term in inducing the agent's 's exploratory behavior.
B.2 BOREDOM AUGMENTATION ENFORCES DYNAMICS-AWARES
To verify the benefit the boredom term beyond helping exploration, we train the representation with the non-augmented objective (3' 0) but this time in the uniform prior setting, that to
14

---

Under review as a conference paper at ICLR 2022
(a) U-MAZE
(b) T-MAZE
(c) 4-ROOMS
Figure Learned representations the gridworld domains with the non con-augmented objective. Without the boredom term, the agent fails to cover the state space (cf. Figure 3 and may settle for incomplete representations. The colors reflect the distances terms of the dynamics. They can be seen quantities proportional to the length of the shortest path from the (marked in red) to the represented state.
marginalize the exploration problem. Figure 10 illustrates the learned representations in the three gridworld domains. These representations have failed to capture the dynamics. For example, in the case 4-ROOMS, distances from first room to the fourth and third rooms are comparable in the representation space, which indicates that the representation does not take into account the relative order in which the rooms should visited, when moving from the first room to the last. Similarly, in U-MAZE, the end of the maze is closer to the initial area than the second corner is. However, in order to reach the former on must pass by the latter. This proves that the boredom term is not only important for the desired exploratory behavior (ef. Figure 9), but also enhances the dynamics- aynamics-awareness awareness of our representation.
(a) U-Maze
(b) T-Maze
(c) 4-rooms
Figure 10: Learned representations when uniformly sampling over the state space. Without the boredom term, the representation does not reflect temporally-extendee dynamics. The colors reflect the distances in terms the dynamics. They can be seen as quantities proportional to the length of the shortest path from the so (marked in red) to the represented state.
C IMPLEMENTATION DETAILS
C.I GRIDWORLD
For all the experiments, we defined the representation network as an MLP of two hidden layers of size 128 and tanh activations and linear output layer of the size of representation' 's dimensionality The high-level and the low-level policies are both MLPs of two hidden layers size 128 with tanh activations and logsoftmax output layer of the size of their respective action spaces: the environment's 's 4 actions for the low-level policy and actions for the high-level policy corresponding to the 8 directions Ω {(cos((
sin(2 nn) k {O...., }}} that define diverse skills.
The policies were trained with vanilla A2C with MC returns from the collected trajectories (Monte- Carlo estimates), iee. no bootstrapped values where used. The skills being of fixed size they could be trained without any reward discount (7 = 1). The high-level and low-level policies were entropy entropy-reguland regularized with the coefficients 0.3 and 0.1 respectively.
All of these networks were trained with R.Mpprop (Hinton et al. 2012) and learning rate of 0.001. Environments specific hyperparameters are provided below.
15

---

Under review as a conference paper at ICLR 2022
C.1.1 REPRESENTATION LEARNING
U-MAZE. Our representation learned in the non-uniform prior setting with p.3..3, Prw 0.4 and K K99 (around the number of steps between so and the furthest state in the maze). We learn a 2-dimensionna representation (d 2) using the representation learning objective o with = 0.2 and 3' = 2. We fix the skills length to representation Φ and the policies (Thi, Tlo by collecting, for each update, a batch of N 32 trajectories of length to fill D) and D=, as described in Algorithm [ We train them for 700 epochs where each epoch corresponds to 10 updates (convergence to the complete representation required around 500 epochs).
30 steps (so L = K/c 3), and jointly train the
T-MAZE. Our representation is learned in the non-uniform prior setting with p,-0.2, Prw 00.4 and K=40 (around the number of steps between 8n and the furthest state in the maze). We learn a 2-dimensional representation (d = 2) using the representation learning objective O with 3 = 0.2 and 3 = 2. We fix the skills' length the representation o and the policies (Tm. Flow) by collecting, for each update, a batch of N = 48 trajectories of length to fill D, and Dr,, as described in Algorithm We train them for 700 epochs where each epoch corresponds to 10 updates (convergence to the complete representation required around 350 epochs).
c 20 steps (so L K/c 2). and jointly train
4-ROOMS. Our representation is learned in the non-uniform prior setting with 0.25, Pr = 0.5 and K0 (around the number of steps between 8o and the furthest state in the maze). We learn a 2-dimensional representation (d 2) using the representation learning objective 6 with 3 = 0.2 and 3 = 2. We fix the skills' length to c = 20 steps (so L = K/c 3). and jointly train the representation and the policies (mhi Tow by collecting, for each update, a batch of N 32 trajectories of length to fill D, and Dr,, as described in Algorithm [1] where each epoch corresponds 10 updates (convergence to the complete representation required around 350 epochs).
We train them for 700 epochs
The Laplacian representation (LAP-REP) was trained in the same environments' settings described above, for both the uniform and non-uniform prior settings (of course no policy is trained here so Prw 1, and (so: Pr) are not relevant for the uniform prior setting). We used the representation learning objective and the associated hyperparameters proposed by Wu et al. (2019). For the uniform prior setting, our online data collection does not cause any discrepancy compared to the offline scheme used in Wu et al. (2019). Indeed, for minibatch size large enough, the stochastic minibatch based training of LAP-REP when using a uniform prior is agnostic to the data collection sheme (offline online) since in both cases the minibatches are sampled from the exact same uniform distribution over the state space.
C.1.2 PREDICTION AND CONTROL
In the prediction and control experiments, we evaluate each pretrained representation by training an actor-critic agent to solve aahh-achieiing task with sparse reward (r = 1 upon reaching the goal). The episode size was set to 100 steps for all the gridworld domains.
For the prediction, the critic head is linear function in the given representation, while the actor is a MLP with two hidden layers of size 64 and tanh activations, logsoftmax output layer of size (discrete gridworld actions) and the actor's input is the state one-hot code. For the control experi- ments, the actor-critic agent is defined on top of the representation as a MLP of two hidden layers of size 64 with tanh activations that feed two output heads: linear critic head and a logsoftmax action head for the 4 actions. The agent is trained with A2C with MC returns and a discount of y = 0.98, batchsize of 80 episodes, an entropy regularization with a 0.01 coefficient and Adam optimizer (Kingma& & Ba 2014) with a learning rate of 0.001.
C.2 MuJoCo: ANTMAZE
In this navigation task, the environment is composed of x 4 x 4 blocks defining a U-shaped corridor. The environment's action space is 8 dimensional. For the sake of simplifying the RL training algorithm, we mapped each dimension values interval to a discrete set of 5 values equally spaced over the interval. We used the same architectures for the representation and the policies as
16

---

Under review as a conference paper at ICLR 2022
for the gridworld, with the only difference that for the low-level policy, the action head was adapted to the discretization of the action space by having 8 logsoftmax output heads of size 5, one for each action dimension and the corresponding discrete values. This choice makes the training algorithm simpler as it allows using A2C here as well.
Our representation is learned in the non-uniform prior setting with p- 0.2, Pri 0.3 and K 500. We learn a 2-dimensional representation (d = 2) using the representation learning objective 6 with 3 = 0.2 and 3' the representation Φ and the policies (Thi, Tlw) by collecting, for each update, a batch of N = 32 trajectories of length epochs where each epoch corresponds to 10 updates (convergence to the complete representation required around 650 epochs).
We fixed their length to c 100 steps (so L K/c- 5). and jointly train
fill D, and Dx,, as described in Algorithm I We train them for 1000
The policies were trained with the same A2C used in gridworld domains and the same RMSprop hy- perparameters. The high-level and low-level policies were entropy- reguuarregl with the coefficients 0.15 and 0.1 respectively.
C.2.1 REWARD SHAPING
Regarding the Laplacian representation baseline, LAP-REP was learned in the same non-uniform prior setting described above. with the representation objective and its associated hyperparameters proposed by Wu et al. (2019). In this setting, the data collection and the representation training performed simultaneoush in online fashion. We have also tested the offline representation training. replicating the training scheme in Wu et al. (2019). Still in the non- non-uniform prior setting, we collected 500000 training samples (10 times more than in Wu et al. (2019) according to uniformly random policy, then we trained the representation on the large dataset built this way. For all other hyperparameters, used the same as provided in Wu et al. (2019). Both trainings ended up giving the same performance for the reward shaping task.
Now, for the reward shaping, we train Soft Actor-Critic (SAC) (Haarnoja et al. 2018 agent to reach a goal area (neighbourhood around the goal state) with episodes of size 1000 steps. We use the following hyperparameters:
Discount 7 0.99
Entropy coefficient (temperature) a = 0.1
Soft critic updates with smoothing constant = 0.005
Replay buffer of size 5 10 (equal to the number of training steps).
Adam optimize with learning rate of 0.0001
As SAC sensitive the reward scale (Haarnoja et al. 2018), we grid-searched this hyperparameter in (10-5, 10-1 1........ 1.0, 2.0}. and the best performing one for our representation was 1.0, while for LAP-REP the SAC agent didn't succeed with any of these values solve the task.
C.2.2 SKILLS EVALUATION
To train DCO, we first collect dataset to estimate the second eigenvector and then use the same dataset train policy the option using DDPG (Lillicrap al. 2013). Each DCO option is tied to its own cigenvector estimate and its own training set of size 500000 (10 times the size used in Jinnai et al. (020)). As suggested by the authors of DCO (Jinnai al.) (200), the remaining hyperparameters to estimate the eigenvectors and train their corresponding options were taken from Wu et al. 2019). DIAYN skills were trained as recommended by Eysenbach et al. (2019). For fair comparison, we train 8 skills for both DCO and DIAYN.
For the skills evaluation stage, we freeze the learned low-level policies and train a high-level policy to use the 8 skills as the only available actions to reach the goal on the other end of the AntMaze environment using a sparse reward It = 1 ||] glla I c] within a finite horizon of 1000 steps Note that this tasks is quite challenging given the type of reward and the length of episode especially in a continuous state space. As our skills offer some flexibility in their execution (can be started everywhere and run for arbitrary number of steps), this episode length was decomposed to 5 skills
17

---

Under review as a conference paper at ICLR 2022
of 200 steps each. The high-level policy was trained with A2C with MC returns (no discount given the finite horizon) a batch size episodes, and RMSprop optimizer with a learning rate of 0.001.
D THE SWITCHING UTILITY OF THE BOREDOM TERM
Note that D., in Eq. may contain trajectories from skills that are not yet duly trained; for example early in the training in a freshly discovered area. Since that stage, these skills' trajectories are close to random walks, their contribution in the boredom term is similar to the first attractive term, in Eq. 2 which is based on random walks. This means that new skill trajectory initially tributes to the temporal similarity term (attractive term) in training the representation, thus making the most of the sampled skills' trajectories while these still early in their training. The more a skill is trained, the more structured its trajectories become and the more they contribute to the intended "boredom" effect (Section 3.3). that encouraging exploration and dynamics awareness (Appendix B
18


Reviews:
[{"review_id": "tTTEEZreBb3", "reviewer": "Reviewer_73Mt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Significance: This work is related to the exploration issue, which is an important topic in RL and will be intriguing to the community.\n- Soundness: The proposed method is sound to me.\n\n\nWeakness:\n- The exhibition of the paper should be improved and some aspects are hard to follow. For example, there are too many overlapping contents in the section of the introduction, related work, and conclusion. However, the presentation on the proposed method is unclear, such as the problem formulation, method design, and how it works in the procedure.\n- Limited novelty: this work extends the previous work of LAP-REP (Wu et al., 2019) and it is incremental. \n- Empirical evaluation is weak. The paper claims that the proposed method can scale better to challenging tasks, and the learned skills can solve difficult tasks. \nHowever, the current results don't match its claims. I didn't see the results for the challenging or difficult tasks (e.g., atari games), although gridworld and AntMaze are good testbeds in demonstrating the efficiency of exploration.\n\n\nMinors:\n- Is there a typo in this sentence? \"hence the need for an exploration strategy for a better covering distribution\".\n- Figure 3: it's better to give more detailed descriptions, such as what's the meaning of the shape and the different colors. Besides, how (a) can be evolved to (j), how does this work?", "summary_of_the_paper": "This work focuses on the problem of uniformity in representation learning, especially Laplacian representation. To address the issue, the authors propose TATC method, which leverages the skills training that representations allow, and uses the learned skills to better cover the state space, i.e., learn a covering policy. Finally, the proposed method is evaluated on (discrete) gridworld and (continuous) navigation environments, with results and analysis demonstrating its effectiveness.", "summary_of_the_review": "I prefer to weak reject (marginally below the acceptance threshold) this paper due to:\n- the work is incremental.\n- the empirical evaluation is weak.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qM-azN8aTw-", "reviewer": "Reviewer_XcUD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper's topic is not really specific to Laplacian RL but proposes an exploration method which seems specific to the exmployed evaluation environments of Laplican RL. As Laplacian RL only works on state spaces with fully reversible actions to allow modelling the state space as an undirected graph, the method has only been evaluated on relatively somply deterministic 2D routing tasks. In general, one could generally argue how practicall important fully reversible environments are as they basically do not yield probabilistic interesting decisions.\nThe proposed method for  using directions to push exploration away from the known part of the state space, does not really seem task-agnostic. Furthermore, the contributions should have been discussed in the light of other methods that improve exploration based on intrinsic rewards. An open question is whether approach could be extended to more general environments where the definition of directions is less obvious.  \nAdmittedly, the two compared task-agnostic skill discovery methods are clearly beaten on the task. However, given that the proposed exploration scheme does not seem to be straight forward to extend to more complicated environments that is not too suprising. \n\n---- Post author response----\nThough the authors claim that exploration is not the main goal of the proposed approach, it is not clear what the difference between a task agnostic exploration method like curiosity with intrinsic and the proposed skill-based covering policy is.\nAfter reading author response I reviewed the description of the  AntMaze and have to admit that in this setting the state space seems considerably larger as the ant's joints need to be moved.  However, I did not find a proper description of the environment as the corresponding reference is MuCojo and the description in the paper and the appendix do not really offer all details on the environment.\n\n\n", "summary_of_the_paper": "The paper proposes an exploration strategy for adapting Lapliacan RL to settings without the possibility to uniformely sample from the state space. In particular, the work builds on the Wu et al. discussion of estimating a spectral decomposition of the state space. However, the estimation of the Laplacian's eigenvectors is based on uniform sampling from  states and possible transitions. \nTo address this restriciton the authors propose a hierarchical exploration scheme, which follows directions on a higher level and optimizes actions of the lower level. Based on these levels the authors propose to employ intrinsic rewards to improve exploration and achieve a better cover of the state space. Experiments are conducted on grid graphs and continous navigation tasks and demonstrate that the new sampling policy performs better than sampling from random walks. More interestingly the authors compare to two other task-agnostic skill discovery\nmethods.\n", "summary_of_the_review": "The contribution of the paper does not seem to be strongly connected to Laplacian RL but seems to propose an hierarchical temporal abstraction method for improving exploration. The result that the approximation scheme of Wu et. al does not work well if the sampling is biased around the starting state and thus, non-uniform is not too surprising.\n\nThus, the paper should discuss and compare its contribution to other exploration schemes based on intrinsic rewards. \nFurthermore, the authors should address the limitations of their approach w.r.t. the properties of the environment. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_3qqiIK-7T", "reviewer": "Reviewer_2jVQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\n* The requirement of a uniform prior in (Wu et al., 2019) is indeed a big limitation. Overcoming this limitation is an interesting topic and will extend the applicability of Laplacian representation.\n* The writing is good and clear. Most notations are well defined. The figures are also informative.\n* Most experiments are well executed.\n\n\n**Weaknesses:**\n* One of my concerns about this paper is the lack of theoretical support. After reading the first half of the abstract, I was very excited and expecting some provable results that overcome the uniform limitation (or say something about the inevitability of the uniform prior, which would also be interesting). Unfortunately, all evidences are empirical. I am not saying that TATC does not work well. TATC is effective, as demonstrated in the experiments. It is just that I feel without theoretic support, the paper is a little bit weak.\n* Also, I feel that the motivation and the conclusion are somewhat mismatched. The motivation is that approximating Laplacian representation (Wu et al., 2019) suffers the limitation of requiring a uniform prior. The conclusion is that TATC can learn a better representation without access to a uniform prior. However, the learned representation by TATC may (probably) NOT be the Laplacian representation. I think it would be better to say \"Motivated by Laplacian representation, we propose a new temporal contrastive representation learning framework in RL\" rather than \"We overcome the uniform prior limitation in (Wu et al., 2019)\".\n* Quite a few details about the experiments are missing. I will explain below.\n\n**Other comments:**\n\n* What is the state (or observation) space for AntMaze environment?\n* How are the colors in Figure 3 and Figure 6 computed? My guess is that they are not calculated in the representation space but manually specified instead.\n* For the prediction and control experiments (Section C.1.2), is the starting state fixed at a specific state $s_0$ across episodes? Where are the goals located?\n* How does the low-level policy network take in the direction vector $\\delta$ ?\n* I am wondering how the hyper-parameters in Section C.1.1 and C.2 (e.g., $p_r$,  $p_{rw}$, $\\beta'$) are chosen? Are the results robust to different hyper-parameter choices?\n* In this paper, the authors use $|\\Omega|=8$ unit vectors as the representative directions in a 2-dimensional representation space. I am wondering how to deal with higher dimensions, since the number of representative directions grows exponentially with the dimension of the representation space.\n* In Section 4.1, the authors emphasize that \"no positional information is provided to the agent\". I do not see the particular necessity of hiding position information. Even with the (x, y)-position, it is still challenging to learn a dynamic-aware representation, due to the presence of walls.\n* In Section 4.1.1, why the AntMaze environment used in this paper is larger compared to the one used in (Wu et al., 2019)? I feel the AntMaze here is pretty similar to the ant-maze2 in (Wu et al., 2019).\n* For the experiments in Figure 7, can you add comparisons with Lap-Rep (trained with total number of dimensions d=2)? Though it still suffers from the uniform prior limitation, I believe Lap-Rep (d=2) can achieve comparable performance as TATC.\n* I am not quite convinced by the results in Section B.2. This setting is similar to the original Lap-Rep paper, except that the implementation of the repulsive term is different. Therefore, I would expect similar results as Lap-Rep. For Lap-Rep (here d=2), the learned representation would span the same subspace as 2 smallest eigenvectors. Since the smallest eigenvector is constant, the distance between two states would then depend on the 2nd smallest eigenvector (the Fiedler vector), and accordingly reflect the environment dynamic (at least for a chain-like environment). However, for U-Maze and 4-Room, it seems that the learned representations in Figure 10 do not have this property. For example, the distance between two ends of the U-Maze is smaller. I would like to hear opinions from the authors.\n* Section 2.1: It might be clearer to give a definition of $\\Delta$.\n\n* Please consider including following two related works:\n  * Wang, K., Zhou, K., Zhang, Q., Shao, J., Hooi, B., & Feng, J. (2021, July). Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing. In *International Conference on Machine Learning* (pp. 11003-11012). PMLR.\n  * Machado, M. C., Barreto, A., & Precup, D. (2021). Temporal Abstraction in Reinforcement Learning with the Successor Representation. *arXiv preprint arXiv:2110.05740*.", "summary_of_the_paper": "This paper introduces a representation learning framework for reward-agnostic RL, which is based on temporal contrastive learning. The proposed method (TATC) is motivated by the Laplacian representation approximation approach (Wu et al., 2019) and overcomes the limitation of requiring a uniform prior. The experiments demonstrate the effectiveness of TATC.", "summary_of_the_review": "Overall, I do not think this paper addresses the uniform prior limitation of Lap-Rep. It is an ok paper if it positions itself in proposing a new representation learning framework. If so, experiments on more challenging environments are necessary. Therefore, I lean towards rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The topic of this paper is non-uniform priors and exploration in reinforcement learning with the graph Laplacian.

All reviewers appreciated several aspects of this work but they all also have several reservations. 

Looking at the paper, reviews and discussions, I see the potential a very nice more general contribution. This potential is not fully realised as the paper stands now. Acceptance can therefore not be recommended.
Decision: Reject