Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces DECSTR, which is an agent having a high-level representation of spatial relations between objects. DECSTR is a learning architecture that discovers and masters all reachable configurations from a set of relational spatial primitives. They demonstrated the characteristics in a proof-of-concept setup.

In the introduction, the inspiration obtained from developmental psychology is described. Motivation and background are broadly introduced. A wide range of related works are introduced in section 2.
The motivation and target of this paper are ambitious and important.

However, from the "methods" part, i.e., section 3, this paper is hard to follow. 
The supplementary material helps to understand. However, I believe some of the informative and detailed information in the supplementary material should come to the main manuscript.

The proposed method, i.e., DECSTR, is comprised of many components. Therefore, the main contribution is also not clear. # What is the main argument of the paper?
Experimental conditions are also hard to follow.

In evaluation, Figure 1 shows ablation studies alone, i.e., comparison with the variants of DECSTR.
Therefore, the contribution of the paper is hard to grasp.

We can understand what kind of task is achieved in this paper.
Currently, the paper somehow seems to be a demonstration of DECSTR. 
In this sense, if the authors state research questions, challenges, and contributions of this paper more clearly, that will make this paper more impactful.
review_title: The research question and the main contributions are not clear.

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**
This paper proposes DECSTR, a goal-driven RL framework where the goal is represented as a binary vector that encodes the semantic relationships between objects. The state is assumed to contain disentangled features for each of the objects (and other features relating to the agent’s end-effectors). The architecture is based on Deep Sets (Zaher et al., 2017), which allows the pairs of the objects to be encoded with a shared network. The paper also introduces a curriculum learning strategy similar to CURIOUS (Colas et al., 2019), which relies on metrics such as competence and learning progress (LP) in order to select goals to pursue during an episode. One key difference is that unlike CURIOUS which uses expert-defined “goal buckets”, DECSTR groups the goals based on recency of discovery. Once trained to be able to behave with respect to these semantic relationship goals, the second phase is language grounding. They learn a module (implemented as C-VAE) that converts from natural language text to the semantic configuration goal space. Experiments were conducted in the Fetch Manipulate robotic arm environment and compared with ablations of DECSTR without some of its components, demonstrating strong performance and generalization to various types of language instructions. 

**Pros**:
- The paper is well-motivated, citing literature from several fields.
- The sum is greater than its parts: many components in DECSTR are based on existing works (e.g. Deep Sets, C-VAE, using LP for intrinsically motivated goals, etc.), but empirically they have shown through ablations that all of their components were necessary for the agent to solve the Fetch Manipulation task successfully. 
- The experiment sections are fairly thorough, with ablations on the components of their methods (as said above), and various kinds of language command generalization evaluations (in a similar style to IMAGINE (Colas et al., 2020). 
- The interpretability of the semantic goal space aspect is interesting. And being able to have the agent explicitly maps from the natural language text to the semantic goal space also helps us debug/understand what the agent is thinking at inference time

**Cons**:
- Part of the thesis is that decoupling of sensorimotor learning from language acquisition is advantageous to an end-to-end language to sensorimotor learning. I have concerns/clarification about some of the baselines, which might not have been a fair comparison with DECSTR (see question 1 & 2 below)
- Some parts of the method are unclear/vague without reading the appendix section to get the full detail. I understand that is due to the space limitation issue and because there are so many components to DECSTR. (see question 3)

**Recommendation**:
 
Overall, I vote for marginally below acceptance threshold in the current form. As mentioned in the strengths section, I do like the motivation of the paper and the strong performance of the method. But I am also suspicious of the poor performance of the baselines (e.g. Figure 1c), which may be due to not having HER, instead of their proposed contributions. It would be good if the authors can clarify that concern. 

**Question**:
1. In Figure 1c, for the Language Goals baseline, was HER applied to the Language Goals in this case (i.e. similar to ACTRCE (Chan et al., 2019), IMAGINE (Colas et al., 2020)? Similarly, was HER applied to the Position-Goals baseline? If not, then it is possible the difference in performance between DECSTR and these baselines may be due more to HER than due to the difference in goal representation. 
2. Would it be possible to train Phase 1 and Phase 2 together or in an end-to-end fashion? This would provide a ‘coupled’ version that is different from any of the baselines studied in the paper because it still uses the semantic configuration as the intermediate goal representation while having joint training of the language representation and the sensorimotor. If this baseline struggles to learn (possibly due to difficult optimization/local minimas), then this will help further strengthen the thesis of the importance of decoupling the learning process into two distinct phases. 
3. Section 3.2: the main text and appendix C.2 was not very clear about the second inductive bias for the symmetry of the behavior required to achieve $above(o_i, o_j)$ and $above(o_j, o_i)$. Are you saying, for example, if we are trying to have object 1 above object 2, then we specify the goal in the form $g_1$, while if we want object 2 above object 1, then we specify the goal in the form $g_2$?  

**Minor comments**:
* When using double quotes in latex, use backticks for the opening quote.

**After rebuttal responses**: 

I have read the authors’ updated draft and response to my concerns, as well as the other reviews. The updated paper provides a clearer framing and some missing baselines have also been included. I raised my evaluation to a weak acceptance for the paper. 

review_title: Well motivated, sum greater than its parts, but some concerns with baselines

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work proposed DECSTR, a procedure for encouraging intrinsic motivation via an intermediate semantic state-space representation. The authors propose an intermediate semantic state space that the intrinsically motivated agent learns to explore. For the environment provided (a 3-block system), the agent fully explores the symbolic state space, reaching all feasible symbolic states. In the second part of the work, the authors train a language model capable of proposing symbolic goals (in the form of symbolic states) from natural language input and shows that the previously-intrinsically-motivated agent can now be made to reach these goals, demonstrating that the symbolic-goal-conditioned policy is sufficient for instruction following in their 3-block domain.

The work is generally interesting, and seems to address a simple version of a broader class of problems that embodied agents typically struggle with, particularly in the absence of clear goals. However, the approach presented in the behavior (in particular the form of the semantic representation that is claimed as one of the primary contributions of the work) is very specific to the single problem used for demonstrations in the paper, limiting the potential impact of the work.

First---and I think the most significant issue with the submission---is that many critical experimental details are included only in the lengthy appendix. Much of this information, including the information provided to the learning algorithm at every step and how that information is encoded such that it allows for a relatively object-agnostic representation, is only available in sufficient detail in the appendix. Relatedly, visualizations of the approach and experimental setup also only appear in the appendix, yet are extremely helpful (if not essential) for understanding. Detail critical to understanding the approach should be included in the body of the text.

Second, it is unclear exactly what problem is being solved in this work or what its primary contribution is. A clearer statement of its motivations will be necessary before publication. /What problem is the robot or system designers trying to overcome?/ Right now, the paper seems to come up with three potential answers to this question, none of which necessarily rises above the others. Here are what I think the main contributions of the work could be:

1. *The proposed semantic representation* The semantic goal representation used to define the space of intrinsic motivation seems to be a novel contribution. However, if the paper were to focus on this aspect of the contribution, it would need to do a better job understating why this representation were useful beyond a relatively small manipulation task. Critically: using only one problem setting with only three blocks is insufficient to convince the reader that this representation is useful more generally (as might be suggested by much of the talk about Inductive Bias).
2. *State of the art state-space exploration in intrinsic motivation.* This might be true, though I find such a thing hard to measure. In addition, it seems that many if not all of the tools used in the learning process are not novel. (Perhaps a combination of this and point 1. is the primary contribution.)
3. *State of the art performance on language-driven block manipulation tasks.* This might be true as well, but the results are so-far unconvincing. All baselines are varied forms of the proposed agent, which makes it difficult to compare against other approaches (e.g. something like Li et al 2019).

The paper currently seems to claim that the combination of progress in these three areas is a novel contribution; I am sympathetic to this idea (as I do not believe that every paper needs to be "state of the art" in one single thing), though it is sufficiently unclear at the moment what the takeaway message of the paper is that I cannot recommend it be published in its current state. In particular, the authors need to work on honing the message of the paper. It is also not unlikely that one or two more experiments will need to be added to support the focused narrative.

Smaller comments
- The name of algorithm should appear in the body of the text, not a footnote. Relatedly, it is unclear how the proposed approach uses the "Deep Sets" work in such a way that it justifies inclusion in the name of the proposed technique.
- The paper/Introduction would benefit from a summary of contributions: even after reading, it may not be clear to a reader which contributions are from this paper versus other work.
- Relatedly, much of the discussion of Inductive Biases that appear throughout the paper is of mixed relevance for this work. On the one hand, it is clear how the idea of an object-centric inductive bias helped to inform how the input to the neural network was encoded in a way that might allow the agent to apply its knowledge learned between two of the objects to a policy that allows it to manipulate all three. However, the goal condition is necessarily specific when it comes to representing which objects to which each element it refers. The structure of the goal and the semantic relations it encodes are quite specific to the particular problem at hand, and it is
- The reward for the "Position only" baseline seems artificially constructed: a non-binary reward function would likely allow the system to learn more easily. As of now I am unconvinced that the authors have worked hard enough to make a fair baseline for comparison. This is particularly problematic since this baseline is a key motivator for the existence of the proposed semantic goal representation.
- The paper overall is quite well written, despite relegating too much information to the abstracts.

review_title: An interesting approach, limited by domain-specificity and poor paper organization

final_score: rating: 7: Good paper, accept
review_content: 
The DECSTR system's intrinsic motivations may be applicable to other application domains, depending on how objects and relations are enumerated. This potential is not explored beyond the toy environment presented. The learning methods (especially inductive biases) are hand-crafted based on human-level knowledge about semantic predicates, but only two ("above" and "close") are demonstrated. Without demonstrating the system on any other configuration or world, it's difficult to tell whether it's able to solve only the problem it's been crafted to solve in this specific environment.

Questions:

3.1 "in principle ... could use any other combination of binary predicates and could be extended to use n-ary predicates" this claim is not demonstrated in the paper, and in 3.2 the inductive biases seem bespoke crafted for binary predicate 'above' which has particular symmetry. Would similar careful design of inductive biases be necessary and possible for n-ary predicates that do not demonstrate these as easily (e.g., "topmost")? What about predicates that involve an unspecified number of discrete arguments, like "base" -> holding up an indefinite N of other objects/structures in "use the green block as the base".

3.4 "or is union" this doesn't generally hold for natural language. A statement like "put the red block or the green block above the yellow block" does not mean to put both red and green (union of goals) above yellow. Typically langauge "or" is "xor"; is the notion of "or" here not given in language or not meant to represent human language?


Areas for Improvement:

5 "a learning architecture that discovers and masters all reachable configurations from a set of relational primitives" this is literally true but only demonstrated on 'a' single set of relational primitives, so it feels like overclaiming.


Nits:
- double citation for Mandler, 2012 in intro in adjacent sentences can be condensed to once
- footnotes on other side of period
- "Besides" in "Blocks Manipulation" seems a bit off-sounding; maybe "In addition,"?
- typo section 3 "based o abstract"
- Typo section 5 backwards quotes "overlapping waves" LHS.
- "Caregiver" in section 5 is an unintroduced role. The rest of the paper does not frame DECSTR or the oracle generator this way.
- Ending the paper with "etc." feels weird/informal.
review_title: The paper presents a method for two-stage self-supervised RL, where an agent first acquires semantic concepts and second grounds language tokens to these concepts (object relationships, primarily). While motivated by language grounding, the training and evaluation paradigms do not include any natural language input from human users, and object relationships are limited to two binary predicates between three objects.

Meta-review: This paper presents a new approach to grounding language-based RL tasks via an intermediate semantic representation, in an architecture called language-goal-behavior (LGB).  The architecture permits learning a mapping from internal goals to behavior  (GB) separately from learning a mapping from language to internal goals (LG), and prior to flexibly combining all three (LGB).  The architecture is studied in a specific implementation called DECSTR.  The architecture has multiple desired attributes including support for intrinsic motivation, decoupling skill acquisition from language grounding, and strategy switching.  The experiments demonstrate the utility of different components in the architecture with a variety of ablation results.

The reviews initially found the paper to be poorly organized with required content described only in the appendix (R1, R2, R4), with unclear main contributions (R1, R2, R4), and with results restricted to demonstrations (R3).  Despite these reservations, the reviewers found the content to be potentially relevant though narrow in scope.

The authors substantially revised the paper. They improved its organization, clarified contributions, separated the architecture from the specific examples, and improved the experimental baselines.  After reading the revised paper, the reviewers agreed that the paper's organization and insights were improved, making the new paper's contribution and insight clear.  The experimental baselines were also improved, providing more support for the potential utility of the proposed method.

Three reviewers indicate to accept this paper for its contribution of a novel approach to grounding language and behavior with an intermediate semantic representation. No substantial concerns were raised on the content of the revised paper. The paper is therefore accepted.
Decision: Accept (Poster)