Discrete Compositional Representations as an Abstraction for Goal Conditioned Reinforcement Learning
Authors: Riashat Islam
Hongyu Zang
Anirudh Goyal
Alex Lamb
Kenji Kawaguchi
Xin Li
Romain Laroche
Yoshua Bengio
Remi Tachet des Combes
Venue: NeurIPS 2022 Accept (2022)
Keywords: goal conditioned RL
discrete bottleneck
vq-vae
self-supervised representations
hierarchical RL

Abstract: Goal-conditioned reinforcement learning (RL) is a promising direction for training agents that are capable of solving multiple tasks and reach a diverse set of objectives.  How to \textit{specify} and \textit{ground} these goals in such a way that we can both reliably reach goals during training as well as generalize to new goals during evaluation remains an open area of research. Defining goals in the space of noisy, high-dimensional sensory inputs is one possibility, yet this poses a challenge for training goal-conditioned agents, or even for generalization to novel goals. We propose to address this by learning compositional representations of goals and processing the resulting representation via a discretization bottleneck, for coarser specification of goals, through an approach we call DGRL. We show that discretizing outputs from goal encoders through a bottleneck can work well in goal-conditioned RL setups, by experimentally evaluating this method on tasks ranging from maze environments to complex robotic navigation and manipulation tasks. Additionally, we show a theoretical result which bounds the expected return for goals not observed during training, while still allowing for specifying goals with expressive combinatorial structure.

Discrete Factorial Representations as an Abstraction for Goal Conditioned RL
Riashat Islam 1,5,7+ Hongyu Zang"; Anirudh Goyal Alex Lamb
Kenji Kawaguchi', Xin Li Romain Laroche', Yoshua Bengio².7 Remi Tachet Des Combes
McGill University, Montreal, Canada, 2 University of Montreal, Montreal, Canada 3 Beijing Institute of Technology, Beijing, China, Harvard University, Cambridge, USA Microsoft Research, Montreal, Canada, Microsoft Research, New York, USA Mila Quebec AI Institute, Montreal, Canada $ DeepMind
Abstract
Goal-conditioned reinforcement learning (RL) is a promising direction for train- ing agents that are capable of solving multiple tasks and reach a diverse set of objectives. How to specify and ground these goals such a way that we can both reliably reach goals during training as well as generalize to new goals during eval- uation remains an open area of research. Defining goals in the space of noisy and high-ddmensiioal sensory inputs poses a challenge for training goal-conditioned agents, or even for generalization to novel goals. We propose to address this by learning factorial representations of goals and processing the resulting represen- tation discretization bottleneck, for coarser goal specification, through an approach we call DGRL. We show that applying discretizing bottleneck can im- prove performance in goal-conditioned RL setups, by experimentally evaluating this method on tasks ranging from maze environments to complex robotic navi- gation and manipulation. Additionally, we prove a theorem lower-bounding the expected return on outoo--distribution goals, while still allowing for specifying goals with expressive combinatorial structure.
Introduction
Reinforcement Learning popular and highly general framework [25, 60) focusing how to select actions for an agent to yield high long-term sum of rewards. An important question is how control the desired behavior of RI agent, both during training and evaluation [24]. One way to control this behavior is by specifying a reward signal [52, 56]. While this approach is very general, the reward signal can be hard design and may not be the most informative form of feedback. The credit assignment problem in RL can become difficult when the reward signal is sparse [62, 63, 36, 37, 57], such as policy gradients becoming nearly flat regions where reward is almost never achieved. Generalization can also suffer if the agent only learns one way to achieve high reward rather than learning diverse set of skills for coping with novel challenges [20].
One potential way to flexibly specify and ground the desired behavior of RL agents is by training agents that receive reward when they reach goal specified explicitly to them [23]. In this ap- proach, called Goal-Conditionel RI., a single agent is trained to reach a diverse set of goals, and is given a reward only when it reaches the goal was instructed to reach [61, 48. 44]. This provides a richer signal for the agent than simply collecting more samples oriented around a single goal, as reaching multiple goals requires the agent to learn more diverse and robust of skills. It also allows for more flexible and tightly constrained control over its desired behavior [12, 5, 27]. Fi- nally, the diversity of goals seen during training should help improve both credit assignment and generalization [48, 43].
"Corresponding Author. E-mail: riashat islam @ww.ilmmmillll...mmmmmm anirudhgoyal91 19 19@ gmail.com
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

While this framework is promising. it introduces two new challenges: goal grounding [8, 2] and goal specification [4]. Goal grounding refers defining the goal space, and goal specification refers to selecting what goal the agent should try to reach in a given context. The agent is only rewarded in goal-conditioned RI. when reaching the reward was instructed to reach, whereas in goal-free RI. reward is provided regardless of any such specification, which makes the nature of the agent's task fundamentally different.
What makes grounding and specifying goals challenging? Consider trying to train goal- conditioned RL agent to pick up various fruits from table. For example, we may want it to pick up red apple or a green pear (illustrated
VQ codeboak C
Robot arm
Geal to pick up fruits
Factor Colour
VQ
Figure 1). The number of possible goals of interests may be fairly small, such as the set of all valid combi- nations of fruits and their colors, while the number of possible observations of goals is extremely large when working in rich ob- servation space (e.g images from a camera). Goal Grounding refers to this challenge of relating high-dimensional observations and the space of relevant goals. Goal Specifica- tion refers to picking a suitable goal for the appropriate reward when it is reached. It also implies specifying goals reachable in the agent's 's current context [39, 26]. Goal specification can be done either man- ually by a developer or by another RL agent, such as a high-level agent which generates goals lower-level agent then tries to reach [12, 5, 27, 22]. Goals specified in language are an excellent fit for these desiderata, language is a compressed discrete representation which is useful for out-of- distribution generalization, while being compositional and expressive [19, 11, 22, 17, 1, 67]. At the same time, connecting language feedback for an agent is non-trivee non-trivial, requiring special assumptions or a labeling framework [9].
Factor Shape
Agen observation
Learnt represctiation structure of relevant goals
Figure 1: Illustration of learning discrete and factorial goal representations.
agent reach and computing
We propose to learn the goal representations with self-supervised learning (either trained on their own, or jointly with the downstream RL objective) while forcing them to be discrete and factorial. To perform this discretization, we use Dector-Quantiztti ous representation using a codebook of discrete and learnable codes. The approach proposed here, called DGRL, serves two complementary purposes. First, it provides structured representation of the raw visual goals. By representing the visual goals as a composition of discrete codes from a learned dictionary, it simplifies the grounding of unseen goals, i.e., goals not seen during training, to novel compositions of the trained discrete codes. We show empirically that this improves the gen- eralization performance of goal-reaching policies while remaining expressive enough. Second, the learned discrete codes can be used by another agent (like higher-level policy in hierarchical RL) to specify sub-goals to lower-level policy, and eventually complete the task (i.e., reach the final goal). In this case, goal-inference is learned end-to-end The effectiveness of goal-conditioned HRL relies on the specification of semantically meaningful sub-goals. Using factorial discrete sub-goals allows the higher-level policy to specify semantically meaningful objectives to the lower-level policy.
[64, 47, 33] which discretizes continu-
2 Preliminaries
Goal-conditioned RL. We consider goal-conditioned Markov Decision Process, where the goals g E G live in the state space S, i.e., S. We denote a goal-conditioned policy as T [a]s,, (either stochastic or deterministic). and its expected total return as J(x) = Σ R(st. g) where the goal g is either sampled from a distribution Pg provided by another higher level policy Th (g|s). The value function V= is additionally conditioned on goals, and is trained to predict the
R(st, expected sum of future rewards conditioned on states and goals; V=(s,g) E R(s,s I (, |
8g s; π As in standard RL, the objective in goal-conditioned RL is to maximize the expected discounted returns induced by the goal-conditioned policy.

---

Self-Supervised Representation
Discretization
522
Learning
(14) Factor
gool
Factot 2
Encodde
Shary weightt
Ersoderr
Embedding Space
G-2
(
(ig)
Downstream RL
subguuls x (e)
subgoals: No. 2
goaal g
obs
High policy
rajectory
Policy wxtwork.
Injectocy:
Low-lovel pocicy
action: a x (-))
goal:
Hierarchical RL
action: d- x[- (o,22)
Goal reaching policies
Figure 2: Summary of Proposed DGRL Model for improving goal grounding and goal specifica- tion by making goal representations discrete and factorial. We learn latent representation for both observations and goals using a self- sel--supervieed learning method (sec. 3). We convert the learnt latent representation into discrete latents based on a VQ-VAE quantization bottleneck with multiple factor outputs (sec. 3). We use the resulting discrete representations for downstream RI. tasks: (i) to train a goal-conditioned policy or value function, and (ii) in the context of goal-conditioned hierarchical reinforcement learning (sec. 3.1).
Hierarchical Reinforcement Learning. We consider goal-conditioned settings in which the goals are specified the observation space. In the hicrarchical reinforcement learning (HRL) setup, goals provided by higher level policy T3j (gls,). The higher level policy operates at coarser time scale and chooses goal 9. Tég (g]s;) to reach for the lower level policy every K steps. The lower level policy executes primitive actions Té. (alst, 9r) to reach the goals specified by the high-level policy and is trained maximize the intrinsic reward provided by the high-level policy, The higher level policy is trained to maximize the external reward i.e., the reward function specified by the MDP. Both the higher and lower level policies can be trained with any standard RL algorithms, such as Deep Q- Q-Learning (DQN) [36] or policy optimization based algorithms [50, 51, 30). Alternately, one can also consider another setup for goal-conditioned RL, where the goals are provided by the environment .1.... -L. and are part of the state or observation space. At each episode of training, one of the goals sampled from the distribution of goals Pg and the policy is trained to reach the sampled goal. At test time, the agent can be evaluated either on its ability to reach goals within the distribution Py. or for its out-of-distribution generalization capability to reach new kinds of goals. We consider both the HRI. and goal-conditioned setups, and evaluate the significance of learning factorial representation of discrete latent goals in a series of complex goal goal-conditioned tasks.
Vector Quantized Representations. VQ- VAE [64, 47,33) 33] discretizes the bottleneck representation of an auto-encoder by adding a codebook of discrete learnable codes. The input passed through an encoder. The output of the encoder is compared to all the vectors in the codebook, and the codebook vector closest the continuous encoded representation fed to the decoder. The decoder is then tasked with reconstructing the input from this quantized vector.
Self-supervised learning of representations. Several papers [31, 58, 53, 34] have demonstrated the benefits of using a pre-training stage where the representations of raw states are learned using self-supervised objectives in task-agnostic fashion. After the pre-training stage, the representa- tions can be used for (and potentially also fine-tuned on) downstream tasks. These self-supervised representations have been shown improve sample efficiency.
3 Discrete Goal-Conditioned Reinforcement Learning (DGRL)
In this section, we provide technical details on the proposed framework, DGRL, which consists of three parts: (a) learning representations of raw visual observations through self-supervised repre- sentation objectives, (b) processing the resulting representations via a learned dictionary of discrete
3

---

codes, and (c) using the resulting discrete representations for downstream goal- condiiondi and HRL tasks. We later describe, in Section 6, how discrete goal representations can accelerate learning in complex navigation and manipulation tasks. We emphasize that these representations can be learned at the same time as the downstream-RI objective or pre-trained with self-supervised learning, and then used as fixed representation for RI..
Self-Supervised Goal Representation Learning. One can use any off-the shelf self self-supervise method for learning representations of the raw state and the goal observations. We denote by e the encoder network that takes as input the raw state and maps to a continuous embedding: F. ((x). Here, we explore two different self-supervised techniques for learning representations. For simpler environments, we use a simple autoencoder with its standard reconstruction objective. For more complex environments,
use the Deep InfoMax approach [34] which optimizes for contrastive objective as a proxy to maximizing the mutual information between representations of nearby states in the same trajectory.
Processing continuous representations via a discrete codebook. We learn discrete representa- tions by using the cector-quantzzation method from the VQ-VAE paper [64], and follow the multi- factor setup used in Discrete- Value Neural Communication [33]. The discretization process for each vector Ze E H C I is described as follows. First, vector ze is divided into G segments C12 c....., Ca.
where &e = CONCATENATE(CI,, C2 ....... Cg). and each segment C; Rm//G (imply- ing that m is divisible by G). Each continuous segment & is mapped independently to a discretized latent vector e R. (mm))) where L is the size of the discrete latent space (i.e., an L-way categori- cal variable):
Ro: DISCRETIZE(G),, where o =
arg min ||q eyll- jE(I....))
These discrete codes, which we call the factors of the continuous representation 2e, are concatenated obtain the final discretized vector 2g:
2q = CECECATENNTTEIISSEETTZZEEEE DISCEETZEE)))) .... DISCRETIZE(CO)).
(1)
The loss for vector quantization is: Ldiserettizatinn = 8 Σ. 1 sg(e,,,
The training procedure closely follows both [33] and [64]. Here, sg refers to a stop-gradient op- eration that blocks gradients from flowing into co., and [3 is a hyperparameter which controls how strongly we move the codes toward the encoded values. Unlike [33], we used a moving average to update the code embeddings rather than learning them directly as parameters. We update eo, with an exponential moving average to encourage it to become close to the selected output segment c. This update sets the new value of co2 to be equal ne, + (1-1))c, where the value of 1) is fixed hyper- parameter controlling how quickly the moving average updates. 2 The term Σi ||e; sg((,,))l 1 is often called the commitment loss. We trained the VQ-quantization process together with other parts of the model by gradient descent. When there were multiple se vectors to discretize in a model, the mean of the commitment loss across all 2e vectors was used.
Summary. The multiple steps described above can be summarized by 2g = [(E, L, G), where is the codebook size, G the number of factors per vector, and q(-) the whole discretization process. We train the representations for both the state and goal observations with this discretization bottleneck applied to the continuous representations resulting from the self-supervised training. The number of factors G is a hyper-parameter. In our experiments, we explored different values: G = 2, 4, 8, 16, and found that G 16 worked the best. Discretizing with more factors slightly increases compu- tation but reduces the number of model parameters due to the codebook embeddings being reused across the different factors.
3.1 Using representations for downstream RL
We use the discrete representations for downstream RL tasks: (i) to train a goal-conditioned policy, and (ii) in the context of hierarchical reinforcement learning.
Goal-conditioned RL. Defining goals in the space of noisy, high- high-dimensional sensory inputs poses a challenge for generalization to novel goals because the encoder that maps the goal observations to
Note that this could also he thought of as gradient step co, taken in the direction C: Cor-

---

the low dimensional latent representation may fail to generalize. One way to address this is to embed the continuous latent representation into discrete representation such that the representation of the novel goal is mapped to the fixed set of latent discrete codes. This facilitate generalization to new combinations of these codes while making easy for downstream learning to figure out the meaning of each discrete code. In this setu, instead of feeding the continuous state and goal embeddings to the agent, we use their discretized versions, thus grounding goal representations in the input space.
We use the resulting representations to train a goal-conditioned policy a; ~ Té. (a|st, It) or a goal- conditioned action value function Q(st, 4z, (t,9t). At each training episode, a goal is sampled from the goal distribution py. and the agent gets rewarded for reaching it. This reward can either be extrinsic, Le., part of the environment, or intrinsic, i.e., part of the algorithm. In DGRL, we define the intrinsic reward the fraction of discrete factors which match in the respective representations of the goal observation and of the state observation. At test time, the agent can either be evaluated on reaching goals within the distribution Pg. or for its generalization capability to goals not seen during training.
Hierarchical RL. The higher level policy gr Th (g st) outputs a continuous representation of goals g by conditioning on the states every K time-steps, it can also output a sub-goal 8g by conditioning on both states s and environment goals g. i.e., n8 (sg I 8t, gt). The effectiveness of goal-conditioned HRL relies the specification of semantically meaningful sub-goals. Learned codebooks (Section 3) consisting of a set of discrete codes can be used by a higher level policy to specify which goal to reach to lower level policy. The use of learned codebooks ensures that the goal specified by the higher level policy is grounded in the space of raw-observations.
In Section 6, we empirically show the benefits of the proposed approach for training goal-reaching policies or goal-conditioned value functions, as well as in a goal-conditioned hicrarchical RL setup.
4 Theoretical Analysis
In this section, discretization is shown to improve generalization to novel goals by enhancing the concentration of the goal distribution within each neighborhood of diseretized goal values; i.e., by decomposing the goal probability p(g) into p(g) = Σx p(glg E Gk)p(g E 9k) with the neigh- borhood set {Gx) it improves the overall performance in p(g) by increasing the concentration in plolg E Gs). Intuitively, this is because the discretization removes varieties of possible goal val- ues 9 E G% for each neighborhood Ga- To state our result, we define (g) Em, [V* (so.g)], where 0 E Rm is the vector containing model parameters learned through n goals observed dur- ing training phase, I1.-... In- We denote the discretization of g by 4(g), and the identity function
by id as id(g) = g. Let Q = {q(g)= 9 E g} and d be a distance function. We use Q to de- note the i-th element of Q (by ordering elements of Q with arbitrary ordering). We also define
[n] (I....,.2), n). GR {g E G k arg min 21 d(((g), 2.)) Ig {i E [m] gi GR]. and Io {k E [|@]] [ 2 1}. We denote by constant in (n. 8, 0, 5,
The following theorem (proof in Appendix D) shows that the goal discretization improves the lower bound of the expected sum of rewards for unseen goals Eg-n, Iyyo o <)(g)) by the margin of (0): Theorem 1. For any ó > 0. with probability at least 1 - 5, the following holds for any 0 E R and < E {id, q): Sorp,lli00 <)(g))] 2 Σ (0o s)(s)) - -ov
n
221(2/0))
1ls id)
n
where w0) = n. EikTo [zal Eiez, po(gi) - Eg-o, (oo())lg E Gul). Moreover, for any compact θ C RR,, ()())) continuous at each 0 θ for almost all g and is dominated by
function X as (po(g) S x(g) all 0 E O with Ey Ey(x())) < co, then the following holds:
(0) 0 when 00.
sup SEB
Proof. Detailed proof provided in the Appendix D
Without the goal discretization, we ineur an extra cost of w()), which is expected to be strictly positive since [x Σiez, so (gr))) is maximized during training while Ey-e, (g)(g)) E Gx] is not.
5

---

Thus, the goal discretization can improve the expected sum of rewards for unseen goals by the degree of (0), which measures the concentration of the goal distribution in each neighborhood. This extra cost (0) goes to zero when the number of goal observations n approaches infinity.
Related Work
Learning with multiple hierarchies has long been proposed in the RL literature, where goal condi- tioned HRL implements high level planning and low level control using sub-goals. Often in goal conditioned HRL, the higher level policy specifies goals which may not have good specification and grounding. Several prior works focus on gondicond RL to improve sample efficiency in deep RL tasks [38, 40J. These build on ideas that were proposed years back to solve long horizon tasks by hicrarchical RL specifying goals [23, 13, 14, 68]. The goal is to learn to solve sub-goals pro- vided to the policy, by learning to predict a sequence of actions that can reach each of the sub-goals 166, 49, 41]. Additionally, in existing HRL literature, distance measures are often used based on goal-conditioned value functions, allowing to measure distances between states and the sequence of sub-goals to reach [16, 71], for planning [42], or exploration [29]. Since the set of goals specified in the state space can be arbitrary, an additional constraint is often also learnt to tie the distribution of selected goals those the lower level policy can reach [72]. We tackle this problem by proposing DGRL, for better grounding and specification of sub-goal representations.
In previous works, mutual information based objectives have been proposed for goal conditioned RL. They perform goal-based representation learning. in order to improve stability of training goal- conditioned value functions [38, 40], to provide goal representations allowing to identify decision states for better exploration [18]. However, for most of these settings, the sub-goals are based on an external reward and are lacking in terms of specification, which can lead inefficient training. [32, 72] have proposed approaches that penalize the high level controller for generating sub-goals that are too difficult for the lower level policies, through the use of additional constrained objectives [72]. We highlight that our proposed DGRL can be generically applied to any goal conditioned RL literature for better grounding and specification of the sub-goals. Furthermore, recent work has shown significance of learning representations through self supervised objectives in RL [3. 54], often as pre-training phase [55, 69]. which can help for both exploration [35] and control [70].
In the context of goal conditioned RL, it can be a challenging problem as it additionally requires learning reliable representations of goals in parallel, purely from high dimensional observations [15]. Previous works [32] have often used the entire observation space as goals, which not scalable for complex tasks. Other works have used pre-defined space of sub-goal or self-play for sub-goal representations [59] to reduce the complexity of goal space design. Most recently, [42, 41] utilized unsupervised representation learning learn goal representation space, which can further be used for planning and control. In this work, we show additionally that using a bottleneck can further lead to factorial representation of goals, while helping with goal specification via learning a latent space of discrete goals usable for planning and control. We emphasize that DGRL can be integrated on any existing goal conditioned approach that utilizes learning a sub-goal representation.
domain knowledge [38],
Experiments
The main goal of our experiments to show that goal discretization can lead to sample efficient learning and generalization to novel goals, in goal-conditioned RI. First, we directly study this by training on environments with a set of goals (such as 8 positions within a gridworld) and then eval- uating the agent's ability to reach a position within the gridworld which it was not trained to reach. Second, we consider hicrarchical goal-conditioned RL, in which a higher-level agent generates goals that a lower-level agent is tasked with reaching. In this case, the task of reaching novel goals occurs organically as the higher-level model selects new goals. This setup also shows the advantages of DGRL for goal specification. A secondary goal of our experiments is to show that using many discrete factors is often critical for optimal performance, which proves the value of factorization in grounding goals.
We evaluate our proposed method DGRL, by integrating into existing ofate-of-the-art goal- conditioned and hierarchical RI. tasks. Experimentally, we analyse DGRL, on several challeng-

---

ing testbeds that have previously been used in the RL community. DGRL in principle can be ap- plied to any existing downstream goal-conditioned RL tasks. We demonstrate improvements on five such tasks. We consider maze navigation where images are used as observations and we show im- proved generalization to novel goals. We integrate DGRL to an existing goal-conditioned baseline for navigating pncedurrally-genrran it outperforms state-of-the-art exploration baselines. We also show improvements with DGRL on continuous control (Ant) navigation and manipulation tasks, where goals come from high-level controller. Finally, we show that discrete representations also significantly improve sample efficient learning challenging vision-based robotic manipulation environment.
hard exploration Minigrid environments [10] and find that
Demonstrating Factorized Representation Learning We first pick color-mnist supervised learn- ing example to support the idea that DGRL can learn factorized or compositional representations. Figure 3 displays reconstructed images from trained decoder operating on discretized 2-factor
4
0
9
4 F 1
4 7
9
4 9
9
S
2
2
5
8
6 2
2
5
7
8 2 2
3 5
0
9
3 3
9
6 2 8
5
5
1
>
3
2 2
1 6 5
3
4 9
6
1
8 P 3
8
4
9 6
1
6 4 6
9
4
S
6
4
8
4 9
5
2 8
8
6 x 9
5
2
3
2
O
2
oon
2
9
5
0
4
6
C
5
&
448
/ 150 8 8
4
8 5 4
6
4 8
7
7
4
7
7 1
7
6
4
4
2
7
O
4
7
D
V
6 9
5
6
8 7 605 9 3 / 9 6
8
8
6
5
8
8
8
L
2
7
s
7 6 6
5
3
6
6 5
3
8 5
0
i 1
3
8
0
0 4
3
7
0
7
0
/
6
0
7
0
6
0
(a) Original Image
(b) Both factors recon.
Factor I recon.
(d) Factor recon.
Figure 3: Color-MNIST example to demonstrate factorized representations; reconstructing the origi- nal images with two factors. Leftmost : Original Image; Left-Middle: Reconstructed Image without substitution; Right-Middle Reconstructed Image with one groups of discrete codes substituted by zero vectors; Rightmost: Reconstructed Image with the other groups of discrete codes substituted by zero vectors
representation. We find that different factors capture information of different semantic nature. More precisely, factor tends to encode the shape of the digit, while factor 2 specialized in its color. This empirically suggests the emergence of "factorization" in the learnt representations. Further experimental details are provided in section C.I.
Learning to Reach Diverse and Novel Goals. We study gridworld navigation task in which an agent is trained to reach a goal from a small finite set of training goals, and during evaluation is tasked with reaching novel goal unseen during training. This is navigation task with pixel-level observation space showing the position of the agent and the goal in a gridworld. We consider two mazes spiral and single-loop topology. Experiment setup given in Appendix C.2.
For this task, we train a goal-conditioned Deep Q-Learning (DQN) agent, and use a pre-trained representation (6-) where the encoder is trained using data from random rollout policy. Because the gridworld is small the random rollout policy achieves good coverage of the state space, so we found this was sufficient for learning good goal representation. At each episode, specific goal is randomly sampled from a distribution of goals, and the DQN agent is trained to reach the specified goal for that episode. During evaluation, test the learned agent goals either from the training distribution, or not seen during training.
Furthermore, for this task, we additionally use an intrinsic reward to promote exploration of the goal-DQN agent. Since we learn a discrete factorial representation of the goal, we compute an exploration bonus based on the discrete latent codebooks; i.e., we embed the states and goals using

---

the learned codes and then compute an intrinsic exploration bonus based on the fraction of learned factors that match. For the baseline goal-DQN agent, we provide an additional reward bonus based on the cosine distance between continuous embeddings of the state observation and goal. Figure 4 shows that DGRL significantly outperforms a continuous baseline goal DQN agent, when trained on either four goals or eight goals. We evaluate generalization to 4 novel goals unseen during training (Figure 5) and demonstrate improved generalization.
4-Gow LoopWord, Eva usted in Dimributio
4-Gool Spint/Word, Evaruated In Datributin
8-Goe Loop/Wons. Eviluated in Distriuttonn
Exolate Retums
EX0 L000 0cal Reauiss
DGRI, Gmd
DiRi, (Gmm
DORI, DGRs,
DCRU,
DIGRt. (ee8
DCNL
DCRL G-18
Gool Boscline
DDN
Ooal Bascline
Eperades
Epocdes
164
Epeeaden
104
Figure 4: Loopworld maze environment. We show that for 4,8 and 16 different discrete factors, DGRL outperforms goal-DQN baseline agent with continuous goal representations. As we in- crease the number of factors to 16, the expressivity of the discrete goal representation increases, lowering the odds of the factors being the same. This provides better intrinsic reward signal for exploration, resulting in faster convergence for DGRL integrated on a goal-DQN agent.
8-Cua SpiraWWorld, Generalized to New Goals
Sleps to Reath Goa (Eve usiton)
16-Cua SpiraWWorld, Ceneralized lo New Coals
COR,
(Eve
DCNL G-d
DCRL G-16
400
conL Gid
EGRt: Ge10
dual DDN
D2N Roodion
300
200
200
100
Siepe
Epleoded
165
Epieodes
165
Figure 5: SpiralWorld World environment (left). Generalization to test distribution containing 4-goals in a SpiralWorld environment (left). We show the total number of steps to solve all test set goals, when trained either an 8-goal 16-goal training distribution.
In the previous experiment, we evaluated the generalization ability of DGRL by showing that learn- ing discrete factorial representations of goals can improve generalization novel goals. Now, we consider various setups which a goal generating agent specifies goals using the learned codebook and goal-conditioned agent is tasked with reaching the goals specified by the goal generating agent. We test various settings, where the goal generating agents is parameterized as an adversarial teacher [6]. or higher-level policy in the case of hierarchical RL.
Procedurally Generated MiniGrid Exploration Task. We follow the experimental setup of [6] and [46] and evaluate DGRL on procedurally generated MiniGrid environments [10]. In [6],
goal-generating teacher proposes goals to train goal-conditioned "student" policy. We integrate DGRL on top of AMIGO [6] and compare DGRL on a hard exploration task with state-of-the-eeee exploration baselines. Experimental results are summarized in Table 1 and more details provided in
Appendix C.4. Note that unlike RIDE and RND, we do not provide an additional exploration bonus to DGRL, and find that DGRL can still solve this hard exploration task more efficiently.
Goal Grounding in KeyChest Maze Navigation Domain. We consider a simple discrete state action KeyChest maze navigation task, following [72], where discrete goals in the state space are provided by higher level policy. For this task, to integrate DGRL, we learn an embedding o(-) ((-) of the goals, then discretize the representation with a learned codebook. We compare with a baseline HRAC [72] agent (details in Appendix C.3). Figure 7 shows an illustration of the KeyChest environ- ment and a performance comparison of DGRL with different group factors G. Using fewer factors
(G 4) performs worse than the HRAC baseline, whereas using larger number of factors (G = 8 or = 16) improves the sample efficiency of the goal reaching agent, providing evidence for the benefits of factorization.

---

Model
KCmedium
Disorez 0oa
AMIGO + DGRL, G=16
.96 +: .01
Success Rate
ADRG Uaesithe
AMIGO DGRL, G=8
.70 .16
AMIGO RIDE
.93 + .06
.00 + .00
RND
.89 .00
ICM
42 = .21
2.0
Figure 6: Performance comparison of the Amigo baseline [6, adversarially intrinsic goals] with and without DGRL for goal discretization.
Table 1: We added DGRL on top of the Amigo baseline implementation provided by the authors.
KeyChest
KeyChest, the agent (A) starts from random stochastic position, picks up the key (K) and then
Figure 7:
Evaluation Retums
the key to open the chest (C). We find
DGRL Gnt DGR, G-5
DGRL, G-16
DGRL improves sample effi
ciency over the HRAC baseline.
Eolsodes
1e4
Ant Manipulation Control Domains. We employed DGRL on three different continuous control tasks: AntMazeSparse, AntFall and AntPush. We emphasize that these tasks the more challeng- ing counterparts AntGather and AntMaze tasks, typically used in the hierarchical RL community [38, 40]. Figure provides illustration. We evaluate goal discretization by integrating DGRL to the state-of-the-an art HRAC baseline. Details of the experimental setup are provided in Appendix 6. Figure 8 shows that specifying the goals using the learned codebook helps DGRL achieve higher success rate compared to the HRAC baseline.
Ant Navigation Maze Tasks. We consider Ant navigation tasks that require extended temporal rea- soning, following the setup in Reinforcement learning with Imagined Subgoals [7, RIS]: U-shaped maze, and an S-shaped maze (the S-shaped maze is shown in Figure 9). The ant navigating in the maze is trained reach any goal in the environment. The agent evaluated for generalization in extended temporal setting with difficult configuration, integrated on top of RIS with several baselines. We emphasize the difficulty of these tasks, where existing baselines like soft actor critic [20, SAC] and temporal difference models [45, TDM] fail completely. Results in Figure 9 show that DGRL improves the sample efficiency over the RIS base- line. Additional experimental setup and environment configurations are provided in Appendix C.6.
compare the success rate of DGRL
ArttAszaSpane
ArtPah
ActFall
DGRL
-7200
DGRL
DORL G#16
HRAC
-7000
DGRL, SGRt, Get
0000
Ged
DGRL, HRAC
-000
HRWC
Eplosdioo
Figure 8: Comparison of DGRL with baseline HRAC [72] on 3 different navigation tasks.

---

U-Shaped Environment
ArtMarreMed mainnrest
DaR. Gerl
RG
Succces
VGS 400
200 00 0.2 0ul
Episodes
Figure 9: Performance comparison with the success rate of reaching goal positions during evaluation in extended temporal configuration of the U-shaped and maze-shaped Ant navigation tasks. We find that integrating DGRL with RIS can lead to more sample efficient convergence on these tasks, while baselines such SAC and TDM (not shown) fail completely both AntU and AntMaze as reported by [7]. The RIS baseline is based on raw data provided by the authors.
DGRL hard vision-based robotic manipulation task, and use the same setup as in Section 6 to integrate DGRL with the state-of- the-art RIS baseline on the Sawyer task in Figure 10. This manipulation task is adapted from [42], where the baseline RIS is already shown to be superior to previous goal conditioning methods. The task of the agent control a 2-DoF robotic arm from image input and move puck positioned on the table. The Sawyer task is designed for training and generalization. At test time, it evaluates the temporally extended configuration. This
Vision Based Robotic Manipulation. Finally,
agent's success at placing the puck in desired positions is a challenging vision-based complex motor task, since test time generalization requires temporally extended reasoning. Results Figure show that DGRL improves the sample efficiency over the RIS baseline. Details of the experimental setup are provided in Appendix C.7.
Success Rate Sawyer Environment
Figure 10: Sawyer Robotic Manipula- tion Task. Integrating DGRL with RIS (with larger num- ber of factors, G=8 and G=16, while G=4 fails), improves over the RIS baseline
60
252 Timesteps (x1000)
860
Discussion
Conclusion Our work provides direct evidence that performance of goal-conditioned RL be improved when the representations of the goals are both discrete and factorial. We show that an instantiation of this idea using multi-factor discretization significantly improves performance diverse set of benchmarks.
Limitations and Future Work An interesting question that arises from our work is how to the- oretically ground and specify goals, which might be helpful for efficient structured exploration in tasks where goal seeking crucial. Additionally, while we demonstrate that the factorial represen- tations learnt by DGRL can be beneficial, it would be interesting to explore whether can also enforce compositionality in the latent embeddings. Finally, exploring in more details the structure of the discrete factors forming the goals, their coverage, and to which extent they semantically capture the underlying factors of the environment a promising research avenue.
Acknowledgement
The authors would like thank Nicolas Heess, John Langford, Yonathan Efroni, Manan Tomar, Dipendra Misra, Akshay Krishnamurthy, Pierre-Yves Oudeyer, Harm Van Seijen and Doina Precup for valuable discussions and insightful comments related this work. Hongyu Zang and Xin Li were partially supported by NSFC under Grant 62276024.

---

References
[1] Ashutosh Adhikari, Xingdi Yuan, Marc-Alexandre Côté, Mikuláš Zelinka, Marc-Antoine Ron- deau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and Will Hamilton. Learn- ing dynamic belief graphs to generalize on text-based games. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Sys- lems, volume 33, pages 3045-3057. Curran Associates, Inc., 2020.
[2] Ahmed Akakzia, Cédric Colas, Pierre-Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language to autonomously-acquired skills via goal generation. arXiv preprint arXiv:2006.077 07185, 2020.
[3] Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R. De- von Hjelm. Unsupervised state representation learning in atari. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bus Buc, Emily B. Fox, and Roman Garnett, ed- itors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8766-8779, 8779, 2019.
[4] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprini 1 1806001966 2018.
[5] Andrew Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learn- ing. Discrete event dynamic systems, 13(1):41-77, 2003.
[6] Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B. Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[7] Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learn- ing with imagined subgoals. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 1430-1440. PMLR, 2021.
[8] Crystal Chao, Maya Cakmak, and Andrea L TThomaz. Towards grounding concepts for transfer in goal learning from demonstration. In 2011 IEEE International Conference on Development and Learning (ICDL), volume 2, pages 1-6. IEEE, 2011.
[9] Maxime Chevalier-Boisverr Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Sa- haria, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample effi- ciency of grounded language learning. arXiv preprini
2018
[10] Maxime Chevalier-Boisvedd Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for openai gym.
2018.
[11] Ugo Dal Lago, Marco Pistore, and Paolo Traverso. Planning with language for extended goals. In AAAI/IAAI, pages 447-454, 2002.
[12] Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.
[13] Peter Dayan and Geoffrey E. Hinton.
Feudal reinforcement learning. In Stephen Jose Hanson, Jack D. Cowan, and C. Lee Giles, editors, Advances in Neural Information Processing Systems
5, [NIPS Conference, Denver, Colorado, USA, November 30 December 3, 1992], pages 271-
278. Morgan Kaufmann, 1992.
[14] Thomas G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition. J. Artif. Intell. Res., 13:227-303. 2000.
[15] Zach Dwiel, Madhavun Candadai, Mariano J. Phielipp, and Arjun K. Bansal. Hierarchical policy learning is sensitive to goal space design. CoRR, abs/1905.01533. 2019.

---

[16] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning Learning to achieve goals via recursive classification. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[17] Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprinl arXiv:/90.007442, 2019.
[18] Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew M. Botvinick, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the information bottleneck. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[19] Herbert P Grice. Logic and conversation. In Speech acts, pages 41-58. Brill, 1975.
[20] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[21] Robert I Jennrich. Asymptotic properties of non-linear least squares estimators. The Annals of Mathematical Statistics, 40(2):633-64) 1969.
[22] Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstrac- tion for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[23] Leslie Pack Kaclbling. Learning to achieve goals. In Ruzena Bajesy, editor, Proceedings of the I 13th International Joint Conference on Artificial Intelligence. Chambéry, France, August 28 September 3, 1993, pages 1094-1099, Morgan Kaufmann, 1993.
[24] Leslie Pack Kaelbling et al. An architecture for intelligent reactive systems. Reasoning about actions and plans, pages 395 410, 1987.
[25] Leslie Pack Kaelbling, Michael I. Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
[26] Khimya Khetarpal, 7afarali Ahmed, Gheorghe Comanici, David Abel, and Doina Precup- What can i do here? a theory of affordances in reinforcement learning. In International Con- ference on Machine Learning, pages 5243-5253. PMLR, 2020.
[27] Tejas Kulkarni, Karthik Narasimhan, Ardavan Sacedi, and Josh Tenenbaum. Hicrarchical deep reinforeement learning: Integrating temporal abstraction and intrinsic motivation. Ad- vances in neural information processing systems, 29, 2016.
[28] Alex Lamb, Riashat Islam, Yonathan Efroni, Aniket Didolkar, Dipendra Misra, Dylan Foster, Lekan Molu, Rajan Chari, Akshay Krishnamurthy, and John Langford. Guaranteed discovery of controllable latent states with multi-step inverse models. arXiv preprint arXiv:2207.08229, 2022.
[29] Romain Laroche and Merwan Barlier. Transfer reinforcement learning with shared dynamics. In Thirty-First AAAI Conference Artificial Intelligence, 2017.
[30] Romain Laroche and Remi Tachet des Combes. Dr jekyll & mr hyde: the strange case of off- policy policy updates. Advances in Neural Information Processing Systems, 34:24442-24454, 2021.
[31] Michael Laskin, Aravind Srinivas, and Picter Abbeel. Curl: Contrastive unsupervised rep- resentations for reinforcement learning. In International Conference Machine Learning, pages 5639-5650. PMLR, 2020.
[32] Andrew Levy, George Dimitri Konidaris, Robert Platt Jr., and Kate Saenko. Learning multi- level hierarchies with hindsight. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
12

---

[33] Dianbo Liu, Alex M Lamb, Kenji Kawaguchi, Anirudh Goyal, Chen Sun, Michael C Mozer, and Yoshua Bengio. Discrete-valued neural communication. Advances in Neural Information Processing Systems, 34, 2021.
[34] Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. Advances in Neural Information Processing Systems, 33:3686-366 3698, 2020.
[35] Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state abstraction and provably efficient rich uic-obbervattin reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020. 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 6961-6971. PMLR, 2020.
[36] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1312v:1312.5602, 2013.
[37] Andrew W Moore and Christopher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning, 13(1):103-130, 1993.
[38] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical rein- forcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Desa-Bianchii, and Roman Garnett, editors, Advances in Neural Information Process- ing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 3307-3317, 2018.
[39] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprini urXiv:1810.01255: 2018.
[40] Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? CoRR, abs/1909.10618, 2019.
[41] Suraj Nair and Chelsea Finn. Hicrarchical foresight: Self-supervised learning of fong-horizon tasks via visual subgoal generation. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[42] Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal- conditioned policies. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence Buché-BBuc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 148 14814-1498) 14825, 2019.
[43] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint 1rXiv:1882.004644 2018.
[44] Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep for model-based control, arXiv preprint 1XX2::880.00900 2018.
[45] Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep RL for model-based control. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[46] Roberta Raileanu and Tim Rocktäschel. RIDE: rewarding impact-driven exploration for procedurally--eenraae environments. In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[47] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
13

---

[48] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approx- imators. In International conference on machine learning, pages 1312-1320. PMLR, 2015.
[49]
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approx- imators. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICMI. 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1312-1320. Jog.. 2015.
[50] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889- 1897. PMLR, 2015.
[51] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[52] Wolfram Schultz. Predictive reward signal of dopamine neurons. Journal of neurophysiology, 80(1):1-27, 1998.
[53] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.055929 2020.
[54] Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[55] Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining representations for data-efficient reinforcement learning. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Informa- tion Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 12686-12699, 2021.
[56) David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence, 299: 10535,, 2021.
[57] Satinder P Singh and Richard Sutton. Reinforcement learning with replacing eligibility traces. Machine learning. 22(1): 1996.
[58] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pages 9870-9879. PMLR, 2021.
[59] Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam, and Rob Fergus. Learning goal embed- dings via self-play for hierarchical reinforcement learning. CoRR, abs/18 1 109083, 2018.
[60] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[61] Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Au- tonomous Agents and Multiagent Systems- 2, pages 761-768, 2011.
[62] Richard Stuart Sutton. Temporal credit assignment in reinforcement learning. PhD thesis, University of Massachusetts Amherst, 1984.
[63] Gerald Tesauro. Practical issues in temporal difference learning. Advances in neural informa- tion processing systems, 4, 1991.
[64] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
14

---

[65] Aad W. van der Vaart and Jon A. Wellner Weak Convergence and Empirical Processes. Springer New York, 1996.
[66] Vivek Veeriah, Junhyuk Oh, and Satinder Singh. Many-goals reinforcement learning. CoRR, abs/ abs/1806.09605, 2018.
[67] Nathaniel Weir, Xingdi Yuan, Marc-Alexandre Côté, Matthew Hausknecht, Romain Laroche, Ida Momennejad, Harm Van Seijen, and Benjamin Van Durme. One-shot learning from a demonstration with hierarchical latent language, 2022.
[68] Marco A. Wiering and Jürgen Schmidhuber. Hq-learning. Adapt. Behav., 6(2):219-246, 1997.
[69] Mengjiao Yang and Ofrr Nachum. Representation matters: Offline pretraining for sequential decision making. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th Interna- tional Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 11 11784-11799 PMLR, 2021.
[70] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 Proceedings of Machine Learning Research, pages (1920-19911 PMLR, 2021.
[71] Lunjun Zhang, Ge Yang, and Bradly C. Stadie. World model as a graph: Learning latent landmarks for planning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12611-12620. PMLR, 2021.
[72] Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating udjacency-constrriind subgoals in hierarchical reinforcement learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Informa- fion Processing Systems 2020, NewrIPS 2020, December 6- 12, 2020, virtual, 2020.
15


Reviews:
[{"review_id": "g8xn0HXEKIv", "reviewer": "Reviewer_oX1u", "summary": "The paper proposes a learning framework for compositional representations of goals for goal-conditioned RL, and proposes to obtain a coarse specification of goals using discretization. The goals reside in a low-dimensional representation space that is obtained from high-dimensional sensory data, ", "questions": "1. How are the training goals and new goals for evaluation in the MazeWorld environment being selected?\n2. Does the method always require a dataset of goals collected prior to training, in order to train the goal representations? Do the curves reflect the sampling cost of these pre-collected data? \n3. How does the method compared to end-to-end training, where goals and observations are encoded into a latent space and this encoder is learnt jointly with the policy?\n", "limitations": "The limitations and societal impact are not discussed. One limitation is that while a discretized goal representation enables composition, the composition is not necessarily interpretable, which is claimed in the introduction (line 78). ", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper shows favorable performance over several baselines on multiple environments. \n\nWeaknesses\n1. Why is it necessary to use two different autoencoder architecture for environments with different difficulty, if the author claims that representation learning method for observations can be used as described in Section 3.1? \n2. The method assumes that the observations and goals come from the same space, restricting the method from being used when goals are specified in a different modality. This assumption should be discussed in the limitations.\n3. The paper claims that the compositional representation helps encode goals into a semantically meaningful latent space (line 77), for example based on the example in Figure 1, I would expect some segments to correspond to goal colors and some segments to correspond to fruit types. However, HRL experiments in Section 5 do not provide evidence that the learnt goal representations are semantically interpretable. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BB291porV_2", "reviewer": "Reviewer_h8LS", "summary": "The paper presents an approach to improve the performance of learned goal-conditioned policies, by discretizing the goals using Vector Quantization. They call their framework Discrete Goal-Conditioned Reinforcement Learning (DGRL), and it consists of 3 stages.\n1) learn representations of the visual input using self-supervised learning, using either an autoencoder or an InfoMax objective.\n2) Process the continuous latent representations via a discrete codebook (generalisation of VQ-VAE to multiple codes) to obtain a discretized code.\n3) Use the pre-trained model on RL tasks, either by giving discrete goals and maximizing an intrinsic reward which is a function of the similarity between the given goal and the discretized embedding of the current observation, or in a HRL setting, where a high level controller is tasked with outputting goals, which are then discretized and used to condition the low level policy.\n\nThe authors provide a theorem that the goal discretization improves the lower bound of the expected sum of rewards of unseen goals. They then provide a series of results comparing their method to a number of different baselines and different task domains. The task domains they chose were a number of navigation tasks in a gridworld environment with pixel observations, a set of exploration tasks in the MiniGrid environment, a maze navigation task in 2D, ant manipulation control and navigation domains, and the Sawyer Robotic Manipulation Task. In the majority of the presented tasks, variants of DGRL have shown to provide some benefit in either sample efficiency or higher final performance.\n", "questions": "In many places in the manuscript, the discrete representations learned by the codebook are characterized as compositional. I would argue that compositionality in the representations learned using neural networks, though desirable, is something we have not yet achieved as a community. In my opinion, the reason for this is that compositionality requires some kind of syntax in the representation, which needs to be interpreted by something else in a very precise and consistent way. To claim that a representation is compositional, one would have to show that consistent replacements of parts of the representations lead to consistent results in behaviour. For example, exchanging the $i$th discrete component with a specific token from the cookbook would cause the conditioned policy to always go to red objects, whereas a different component could correspond to the type of object (ball, cube etc) and yet another could correspond to a particular area of the maze. Moreover, a truly compositional representation would require that any given combination of these tokens would lead to the expected behaviour, and perhaps even require evidence of nested syntax. Due to the lack of such evidence, I would encourage the authors to remove the term “compositional” and instead use, perhaps, “factored” or “structured”. I would also be interested to hear the authors’ thoughts about my concerns. ", "limitations": "I haven’t seen any critical assessment of limitations of the method, and would be very interested to hear the authors’ views on this. For example, how many different goals can we express in this method? How easily can this scale? How different are the resulting discrete codes between them?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The method is simple to implement in existing architectures and uses established approaches that have been developed for some time now.\n2. The authors did a comprehensive empirical study in many different domains and tasks. It is very useful to see how the method performs in all these domains.\n3. The paper is overall very well written, and the ideas and results are communicated clearly.\n\n### Weaknesses\n\nThe method does not provide a significant improvement in most environments, nor I consider to be necessarily original (see [1,2] for examples using a very similar idea). It is of course interesting to know the benefits of this addition, but as the results show, the results are comparable for the majority of tasks, except in some cases of gridworld or ant maze navigation and the sawyer arm tasks, where the DGRL seems to help with sample efficiency. For example, in Table 1, it is clear that some choices of G can even considerably harm performance compared to the baseline (see G=8). To me, this is not necessarily a reason to reject, as this idea is something that many others will think to try at some point. Publishing genuine empirical results showing the realistic improvements that should be expected by a simple addition to the method, such as the one presented in this work, should be encouraged in my opinion.\nIn terms of presentation, I found it a bit odd that the related work was in the appendix. Perhaps a bit part of the introduction could have been removed or delegated to the appendix, as it refers to high level ideas about goal grounding and goal specification which are not directly addressed by the paper.\n I have a comment about the use of the term “compositionality” in the paper used to refer to the discrete representation, which I elaborate more on in the “Questions” section below. I would urge the authors to remove the term compositional from the title as well. Please see the questions section below.\n\n[1] Ozair, S., Li, Y., Razavi, A., Antonoglou, I., Van Den Oord, A. and Vinyals, O., 2021, July. Vector quantized models for planning. In International Conference on Machine Learning (pp. 8302-8313). PMLR.\n\n[2] Fernández, F., Borrajo, D. (2000). VQQL. Applying Vector Quantization to Reinforcement Learning. In: Veloso, M., Pagello, E., Kitano, H. (eds) RoboCup-99: Robot Soccer World Cup III. RoboCup 1999. Lecture Notes in Computer Science(), vol 1856. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45327-X_24\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7PCRrGfxf7C", "reviewer": "Reviewer_PnN7", "summary": "This paper proposes to use discretized self-supervised representations for goal encoding in goal-conditioned RL. Specifically, the paper proposes using a codebook of L discrete codes to encode each of G chunks of a state/goal representation vector. The paper evaluates this method on a variety of environments for goal-conditioned and hierarchical RL, and demonstrates improved learning efficiency consistently, and improved generalization in several cases.\n", "questions": "See above, but briefly:\n* Would this method perform as well if the space of possible goals were itself high-dimensional, rather than effectively 2D?\n* How does the discretization change the training distribution, and can this be disentangled from the representation benefits?\n* What are the benefits of choices like updating the codewords over training with EMA, and could these be justified with ablation studies?\n* What are the codebook sizes?", "limitations": "See above, but briefly:\n* It's unclear whether discretized methods would perform well in more complex settings; this should be explored and/or discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The challenge of representing goals for RL is important, and the paper does a good job of motivating the problem and the proposed solution.\n* The solution is relatively simple to implement, and the breadth of established evaluation environments and algorithms suggests that this approach will be generalizable to other similar settings\n* The baseline comparisons seem generally strong, as they are taken from established works in the area.\n\nWeaknesses:\n* Discretized representations—even compositional ones—do not always scale well to high-dimensional, complex spaces. While the paper emphasizes the fact that the goals and states come from noisy, high-dimensional sensory inputs, the underlying goal spaces in each case are effectively 2D (goal positions to navigate to, to push a puck to, etc.), and thus it is unsurprising that a discrete representation would be effective. The strength of the paper would therefore be improved by also evaluating performance on higher dimensional goal spaces; for example, positioning 5 or 10 different objects in distinct positions with the robot rather than just 1, or stacking objects in particular places and orders, making sandwiches with different ingredients, etc. These experiments might be unreasonable to demand in the present submission—as the authors note, they evaluate on many environments from prior related works—but they would make the paper much stronger. Otherwise, it would be worth discussing this as a possible limitation explicitly.\n* One aspect that seemed unclear in the current presentation is whether the goal reaching policy is trained according to the continuous goals produced by the higher-level policy or the discretized versions of them. For example, if the goal reaching policy is rewarded for getting within a Euclidean distance of 0.5 of the goal, is that based on the continuous or discrete representation? I assume the latter, but in that case the discretization process changes not only the way that the goal is represented to the agent, but the distribution of goals the agent is trained on. In particular, it seems like it might result in the goal reaching policy having a more consistent set of goals, which might explain some of the improved learning efficiency. These two factors could be disentangled in some ways, for example by discretizing the goal when inputting it to the agent, but still rewarding the agent based on the continuous representation. Exploring these issues would help to clarify the benefits of the method.\n* Some more ablations could be useful. For example, what if the L latent vectors are fixed to reasonable values (e.g. the final codeword values of a prior training run) rather than updated with an exponential moving average? This might relate to the prior point, and would help identify whether something about the updating process is playing a role in the benefits, or whether it’s just the discretization. \n* As far as I could find, the paper does not specify the codebook size L anywhere in the main text or the supplement. This detail is relevant to interpreting how discrete the encoding actually is, as well as for future researchers who want to use the method.\n* To my understanding, text that is reproduced from a prior paper should be clearly marked as such; a non-trivial amount of section 3.2 is copied verbatim from one of the references without clearly communicating that fact (although the reference is cited as a source for the method). The paper should be transparent about details that are reproduced from prior work. \n* A minor point, but the paper says “[26], which generalizes the vector quantization used in VQ-VAE [53] to a list of discretized codes instead of a single discretized code.”–the VQ-VAE paper also used arrays of codes rather than a single code in all applications; for example a grid of codes for images or an array of codes for sequences. It might be good to clarify this point.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept