Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a learning framework for compositional representations of goals for goal-conditioned RL, and proposes to obtain a coarse specification of goals using discretization. The goals reside in a low-dimensional representation space that is obtained from high-dimensional sensory data, 
strengths_and_weaknesses: Strengths
1. The paper shows favorable performance over several baselines on multiple environments. 

Weaknesses
1. Why is it necessary to use two different autoencoder architecture for environments with different difficulty, if the author claims that representation learning method for observations can be used as described in Section 3.1? 
2. The method assumes that the observations and goals come from the same space, restricting the method from being used when goals are specified in a different modality. This assumption should be discussed in the limitations.
3. The paper claims that the compositional representation helps encode goals into a semantically meaningful latent space (line 77), for example based on the example in Figure 1, I would expect some segments to correspond to goal colors and some segments to correspond to fruit types. However, HRL experiments in Section 5 do not provide evidence that the learnt goal representations are semantically interpretable. 
questions: 1. How are the training goals and new goals for evaluation in the MazeWorld environment being selected?
2. Does the method always require a dataset of goals collected prior to training, in order to train the goal representations? Do the curves reflect the sampling cost of these pre-collected data? 
3. How does the method compared to end-to-end training, where goals and observations are encoded into a latent space and this encoder is learnt jointly with the policy?

limitations: The limitations and societal impact are not discussed. One limitation is that while a discretized goal representation enables composition, the composition is not necessarily interpretable, which is claimed in the introduction (line 78).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to use discretized self-supervised representations for goal encoding in goal-conditioned RL. Specifically, the paper proposes using a codebook of L discrete codes to encode each of G chunks of a state/goal representation vector. The paper evaluates this method on a variety of environments for goal-conditioned and hierarchical RL, and demonstrates improved learning efficiency consistently, and improved generalization in several cases.

strengths_and_weaknesses: Strengths:
* The challenge of representing goals for RL is important, and the paper does a good job of motivating the problem and the proposed solution.
* The solution is relatively simple to implement, and the breadth of established evaluation environments and algorithms suggests that this approach will be generalizable to other similar settings
* The baseline comparisons seem generally strong, as they are taken from established works in the area.

Weaknesses:
* Discretized representations—even compositional ones—do not always scale well to high-dimensional, complex spaces. While the paper emphasizes the fact that the goals and states come from noisy, high-dimensional sensory inputs, the underlying goal spaces in each case are effectively 2D (goal positions to navigate to, to push a puck to, etc.), and thus it is unsurprising that a discrete representation would be effective. The strength of the paper would therefore be improved by also evaluating performance on higher dimensional goal spaces; for example, positioning 5 or 10 different objects in distinct positions with the robot rather than just 1, or stacking objects in particular places and orders, making sandwiches with different ingredients, etc. These experiments might be unreasonable to demand in the present submission—as the authors note, they evaluate on many environments from prior related works—but they would make the paper much stronger. Otherwise, it would be worth discussing this as a possible limitation explicitly.
* One aspect that seemed unclear in the current presentation is whether the goal reaching policy is trained according to the continuous goals produced by the higher-level policy or the discretized versions of them. For example, if the goal reaching policy is rewarded for getting within a Euclidean distance of 0.5 of the goal, is that based on the continuous or discrete representation? I assume the latter, but in that case the discretization process changes not only the way that the goal is represented to the agent, but the distribution of goals the agent is trained on. In particular, it seems like it might result in the goal reaching policy having a more consistent set of goals, which might explain some of the improved learning efficiency. These two factors could be disentangled in some ways, for example by discretizing the goal when inputting it to the agent, but still rewarding the agent based on the continuous representation. Exploring these issues would help to clarify the benefits of the method.
* Some more ablations could be useful. For example, what if the L latent vectors are fixed to reasonable values (e.g. the final codeword values of a prior training run) rather than updated with an exponential moving average? This might relate to the prior point, and would help identify whether something about the updating process is playing a role in the benefits, or whether it’s just the discretization. 
* As far as I could find, the paper does not specify the codebook size L anywhere in the main text or the supplement. This detail is relevant to interpreting how discrete the encoding actually is, as well as for future researchers who want to use the method.
* To my understanding, text that is reproduced from a prior paper should be clearly marked as such; a non-trivial amount of section 3.2 is copied verbatim from one of the references without clearly communicating that fact (although the reference is cited as a source for the method). The paper should be transparent about details that are reproduced from prior work. 
* A minor point, but the paper says “[26], which generalizes the vector quantization used in VQ-VAE [53] to a list of discretized codes instead of a single discretized code.”–the VQ-VAE paper also used arrays of codes rather than a single code in all applications; for example a grid of codes for images or an array of codes for sequences. It might be good to clarify this point.


questions: See above, but briefly:
* Would this method perform as well if the space of possible goals were itself high-dimensional, rather than effectively 2D?
* How does the discretization change the training distribution, and can this be disentangled from the representation benefits?
* What are the benefits of choices like updating the codewords over training with EMA, and could these be justified with ablation studies?
* What are the codebook sizes?
limitations: See above, but briefly:
* It's unclear whether discretized methods would perform well in more complex settings; this should be explored and/or discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents an approach to improve the performance of learned goal-conditioned policies, by discretizing the goals using Vector Quantization. They call their framework Discrete Goal-Conditioned Reinforcement Learning (DGRL), and it consists of 3 stages.
1) learn representations of the visual input using self-supervised learning, using either an autoencoder or an InfoMax objective.
2) Process the continuous latent representations via a discrete codebook (generalisation of VQ-VAE to multiple codes) to obtain a discretized code.
3) Use the pre-trained model on RL tasks, either by giving discrete goals and maximizing an intrinsic reward which is a function of the similarity between the given goal and the discretized embedding of the current observation, or in a HRL setting, where a high level controller is tasked with outputting goals, which are then discretized and used to condition the low level policy.

The authors provide a theorem that the goal discretization improves the lower bound of the expected sum of rewards of unseen goals. They then provide a series of results comparing their method to a number of different baselines and different task domains. The task domains they chose were a number of navigation tasks in a gridworld environment with pixel observations, a set of exploration tasks in the MiniGrid environment, a maze navigation task in 2D, ant manipulation control and navigation domains, and the Sawyer Robotic Manipulation Task. In the majority of the presented tasks, variants of DGRL have shown to provide some benefit in either sample efficiency or higher final performance.

strengths_and_weaknesses: ### Strengths

1. The method is simple to implement in existing architectures and uses established approaches that have been developed for some time now.
2. The authors did a comprehensive empirical study in many different domains and tasks. It is very useful to see how the method performs in all these domains.
3. The paper is overall very well written, and the ideas and results are communicated clearly.

### Weaknesses

The method does not provide a significant improvement in most environments, nor I consider to be necessarily original (see [1,2] for examples using a very similar idea). It is of course interesting to know the benefits of this addition, but as the results show, the results are comparable for the majority of tasks, except in some cases of gridworld or ant maze navigation and the sawyer arm tasks, where the DGRL seems to help with sample efficiency. For example, in Table 1, it is clear that some choices of G can even considerably harm performance compared to the baseline (see G=8). To me, this is not necessarily a reason to reject, as this idea is something that many others will think to try at some point. Publishing genuine empirical results showing the realistic improvements that should be expected by a simple addition to the method, such as the one presented in this work, should be encouraged in my opinion.
In terms of presentation, I found it a bit odd that the related work was in the appendix. Perhaps a bit part of the introduction could have been removed or delegated to the appendix, as it refers to high level ideas about goal grounding and goal specification which are not directly addressed by the paper.
 I have a comment about the use of the term “compositionality” in the paper used to refer to the discrete representation, which I elaborate more on in the “Questions” section below. I would urge the authors to remove the term compositional from the title as well. Please see the questions section below.

[1] Ozair, S., Li, Y., Razavi, A., Antonoglou, I., Van Den Oord, A. and Vinyals, O., 2021, July. Vector quantized models for planning. In International Conference on Machine Learning (pp. 8302-8313). PMLR.

[2] Fernández, F., Borrajo, D. (2000). VQQL. Applying Vector Quantization to Reinforcement Learning. In: Veloso, M., Pagello, E., Kitano, H. (eds) RoboCup-99: Robot Soccer World Cup III. RoboCup 1999. Lecture Notes in Computer Science(), vol 1856. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45327-X_24

questions: In many places in the manuscript, the discrete representations learned by the codebook are characterized as compositional. I would argue that compositionality in the representations learned using neural networks, though desirable, is something we have not yet achieved as a community. In my opinion, the reason for this is that compositionality requires some kind of syntax in the representation, which needs to be interpreted by something else in a very precise and consistent way. To claim that a representation is compositional, one would have to show that consistent replacements of parts of the representations lead to consistent results in behaviour. For example, exchanging the $i$th discrete component with a specific token from the cookbook would cause the conditioned policy to always go to red objects, whereas a different component could correspond to the type of object (ball, cube etc) and yet another could correspond to a particular area of the maze. Moreover, a truly compositional representation would require that any given combination of these tokens would lead to the expected behaviour, and perhaps even require evidence of nested syntax. Due to the lack of such evidence, I would encourage the authors to remove the term “compositional” and instead use, perhaps, “factored” or “structured”. I would also be interested to hear the authors’ thoughts about my concerns. 
limitations: I haven’t seen any critical assessment of limitations of the method, and would be very interested to hear the authors’ views on this. For example, how many different goals can we express in this method? How easily can this scale? How different are the resulting discrete codes between them?
review_title: null

Meta-review: metareview: This paper proposes a discrete and compositional representation of goal states for goal-conditioned RL. The idea is to learn a goal representation via self-supervised learning and discretize the learned representation via VQ-VAE, and finally use the learned goal representation for goal-conditioned RL. The proposed method improves performance on several goal-conditioned RL benchmarks.

All of the reviewers found the idea simple and reasonable, and the results on a variety of benchmarks are quite comprehensive and strong. Although there were concerns around why the proposed discretized representation forms a semantically meaningful latent space and where the improvement comes from, the authors addressed them during the rebuttal period with updated results. All of the reviewers became in favor of the paper as a result. Thus, I recommend accepting this paper.
confidence: Certain
Decision: Accept