Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: **Summary**

This paper describes the definition of an intrinsic reward designed to encourage an agent to behave in such a way that they can distinguish between MDPs that have different values of some hidden parameter that affects the way the MDP behaves. Proof-of-concept experiments are included to allow discussion of how providing this classification of MDPs to an agent that needs to learn how to perform some externally-designed task allows it to learn to complete the task faster and discussion of how the agent's observations affect its learning and behaviour.

**Strengths and Weaknesses**

The primary problem with this paper is that the description of the proposed method is too imprecise to allow me to review the quality of the research completed. I feel that the language and structure are not at an acceptable standard for ICLR. Some acute problems are that the protocol for training as well as some key algorithmic choices are never specified (i.e., the clustering algorithm and the planning algorithm for selecting action trajectories) and the notation and language around causal factors are not clear. 

When I say that the protocol for training is not specified, I am referring to many missing elements, like how the training dataset is composed and what information the agent is given about each environment it is placed in (e.g., is the agent able to distinguish one environment from another?) The explanation of how the clustering tree is formed lacks substance: Do the human experimenters change the composition of the training data once one causal factor has been learned sufficiently well (and how?), or is the agent able to distinguish which node of the tree a given environment belongs to? For example, what does it mean to "perform an intervention on the embedding of the causal factor isolated by the initial optimization of causal curiosity"? This is a very important aspect of the project, so it is critical to explain in plain language what you did.

For the sake of reproducibility and understanding, it is necessary to know what algorithm was used for clustering and what algorithm was used to adapt the agent's behaviour to increase the amount of intrinsic reward received.

One strength of the paper is how the authors have included intuitive examples throughout to help explain the intentions behind their design (e.g., "For example, if a body in an environment loses contact with the ground …"), though in some cases the language applied to the examples makes it unclear how they should be understood. For example, when you say "examples of the parameter $H$ include gravity, coefficients of friction…," it sounds like each $H$ could be any one of those examples, but from Definition 1, I suspect $H$ must be a set whose elements could include all or some of the examples. This should be clarified.

**Recommendation**

I am recommending that this paper be rejected because it is not written with sufficient clarity to understand the research project being reported.

**Specific Examples of Lack of Clarity**

Definition 1 seems critical to the paper, and as such, I think it needs to be written very carefully and clearly. The relationship between H and its associated sequences of actions is very unclear to me. Does each $h_i$ require a different sequence of actions from the sequence associated with $h_j \neq h_i$? (The definition doesn't seem to have that requirement.) An explanation of what it means for a sequence of actions to cluster state trajectories also seems to be needed. Later in the paper, Definition 1 is described using the phrase "when intervened on over a set of values" and I'm really not sure what this means practically.

As a reader, I need help with understanding your choice of notation. For example, I intuitively suspect that you as authors consider $H$ (pp. 1-2) and $H^{(i)}$ (p. 3) to be related by virtue of their very similar notation, but I don't know how I am supposed to connect them in my head, other than I know that both are made up of causal factors but one is a set and the other is a vector.

The connection to Rousseeuw's (1987) work is confusing. Is the reward to the agent actually given by Equation (3), which appears to be a variation of silhouette width as defined by Rousseeuw? The presentation makes it sound like Equation (3) and Silhouette Score are essentially unrelated. 

I suggest rephrasing the introduction of notation on page 1 to reduce ambiguity. I think that $z$ is a latent representation of the environment (my guess based on notation conventions), but I initially assumed that $z$ symbolically represented an environment based on the text.

It is unclear whether you are using the term "causal factor" in the same way as you have noted that previous studies do (p. 1), or whether it has some other meaning in this paper.

Based on notation, I think that a hyper-parameter $H \in \mathcal H$ (also referred to as simply a parameter) is equivalent to a hidden parameter (or causal factor) in the noted literature. This use of the term hyper-parameter is confusing and if I understand correctly, sticking with the term hidden parameter would be clearer (unless there is a reason I am missing for using hyper-parameter). 

**Additional Feedback (Here to help, not necessarily part of decision assessment)**

It was unclear what the set of trajectories during a rollout would look like. In other papers, trajectory sometimes refers to any observed sequence of state-action-rewards, while in others it might refer specifically to such a sequence that begins and ends at the same time as an episode. In definition 1, the phrase "Let T be the length of the trajetcories during each rollout and $s_{0:T} \in S^T$ denotes a trajectory" (p. 2) makes me think there might be multiple trajectories per rollout, so I would like to be clear on how each is defined.

Should $r_{0:T}$ be defined explicitly on page 1 too?
I encourage you to be careful with your language when you define mathematical objects to share the name of a naive concept like causal factor. The way the definition is presented already suggests that this definition doesn't capture everything you would like it to capture because of simplifying to binary clusters. While tying it to the naive concept can help provide some intuition about your intentions, I could see this name causing some confusion when this idea develops further.

"trajetcories" (p. 2)

"simplicitly" (p. 2)

I was surprised by the curly braces used for the observations (p. 3), which I think are ordered sequences rather than sets? (Typo?) 

There seems to be a typo in Equation (3): I think there should be "max" on the second and third terms.

If you need more space to provide a summary of the algorithm followed by your system, I don't think that the connection to model selection on page 4 is clearly necessary to the paper.

You'll probably want to go through the bibliography to check for capitalization errors like "Soft-dtw," "beta-vae," etc.  

---------------------------------------------------------------

**UPDATE after author rebuttal**

The authors have made substantial improvements to the paper over the rebuttal period, but there is still work to be done to make this paper communicate as clearly as it should for publication. I share the concerns of Reviewer 1 and feel that this paper would benefit from more thought into how to present the ideas and another round of reviews.

**Additional Feedback for future revision**

The preamble to Section 2 is now quite helpful, but it would be even more helpful if placed prior to Definition 1! Helping the reader to think about the elements of $H$ as random variables and to think of causal factors as determining which environment you end up in would aid understanding of Definition 1.

Depending on who you want your audience to be, I suggest considering adding an explanation of the "do" notation. While the sub-community focused on causality may be aware of the notation, the community interested in intrinsic motivation more broadly would probably be interested in this paper but not know the notation.

I think that the following sentence could be quite helpful if written in a different way: " In our definition, $h_j$ are causal factors such friction with some particular coefficient of friction, or gravity with acceleration constant $g$ or other" (p. 14). It would be helpful to have the variable (e.g. amount of friction) separated from its value (which might represent the coefficient of friction). I'd like the concept that causal factors are variables that could take on any of a set of values made more explicit. 

"the outcome of running" (p. 14) is slightly ambiguous (at first I read it that running was the outcome, rather than the experiment. A phrase like "the outcome of the attempt to run" or the outcome of the running experiment" might be clearer.

Consider using the singular they in your human example (p. 14). A decent primer: https://apastyle.apa.org/style-grammar-guidelines/grammar/singular-they

Additional typos found:
Let $o_{0:T} \in \mathcal O^T$ denotes → denote (p. 2).
$k$ is both the length of the set $H$ and the comparator with $h_j$ (p. 2) Maybe don't use $k$ and $K$ either, since it still gives me the vague sense that they might be related, but I'm pretty sure they're not. 
Neither CEM nor MDL is written out in full (p. 4 is first use).
"as compared to the vanilla CEM planner (Figure??)." (p. 7)
"are causal factors such friction" → such as friction (p. 14)

review_title: Initial review of Submission 764

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper considers the problem of skill discovery in settings where the data appears to be a Markov Decision Process and part of the state is unobservable. The hidden state variables are interpreted as causal factors that control important aspects of the environment  dynamics. Under this interpretation, the paper advocates for the use of a reward that encourages learned skills that exercise individual components of the hidden state. These skills are learned with a model-based RL algorithm -- one skill per causal factor -- then transferred for use in a downstream control problem that uses a different learning algorithm. The paper claims the learned skills are qualitatively meaningful, and that they enable agents to solve downstream problems without any additional training. Data used for empirical evidence comes from a simulated manipulation robot. 

The paper presented several interesting ideas with the potential for high impact in robotics and reinforcement learning. I thought the paper was generally well written and the ideas well-positioned with respect to related work. I have concerns about the empirical evaluation, which I detail in several followup questions: 

1. Can you explain the difference(s) in how states and observations are treated in this paper?
2. Can you verify that the structured behaviors shown in your videos are a result of the MDL reward and not the changes imposed to the observation?   
3. Are the same behaviors realizable using the MDL reward as an additive bonus on the downstream problem?
4. How do the hidden causal factors numerically differ in each video? 
5. How does the MDL skill differ from a skill trained from an external reward that explicitly encodes the same behavior?
6. Can you explain the notation in Equation 3? What does the minus sign denote?
7. Why is the causality formalism useful and necessary? 
8. Does the MDL reward structure lead to better skills than alternatives? Examples include:  a negative linear threshold of the hidden state vector, a reward of -1 for each non-zero hidden factor, the negative magnitude of the hidden factor vector, and possible competing methods found in the literature.
9. What is the sample cost of learning an individual skill, and how does it compare to training on the downstream problem from scratch?
10. Why are the best rewards negative and the worst positive in Figure 3?
11. Over how many trials was data collected for the results in figures 3 and 5?

review_title: A compelling technique with a questionable evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary

The authors introduce *causal curiosity*, an intrinsic reward that allows an agent to discover causal factors in an environment. The authors run experiment on `Causal World Simulation`  , a 3-fingered robot manipulating a single 3D object. These experiment show that the system:
* Learns, in a self-supervised way, interpretable behaviors corresponding to "experiments"
* Helps with two downstream tasks (`lifting` and `travel`) 
* Learns an interpretable latent space that hierarchically partitions the environment in a way that isolate its causal processes
* Generalizes over unseen variations in the environment

## Analysis

I struggled to understand Definition 1. "The set $H = \\{h_1, h_2, . . . , h_k\\}$ is called a set of $\epsilon$−causal factors if for every $h_i \\in H$, there exists a sequence of actions that clusters the state trajectories into two sets $S$ and $S_0$ such that..." I can't figure out what the $h_i$ are and how "a sequence of actions" could exist for each of them.

I found interesting that the system self-discovered toss, lift-and-spin and roll behaviors, although I was not able to understand whether these behaviors allowed the agent to disentangle a specific aspect of the environment (mass, shape, size). For example, was lift-and-sping used to disentangle the mass? A better/clearer analysis of this would be helpful.

In Section 4.2, the two proposed downstream task (`lifting` and `travel`) seem very close to the behaviors learned during the self-supervised phase. As such, I'm not surprised about the fact that you're able to learn them really quickly. However, I do not find this very convincing. I'd be more impressed if the tasks were significantly different that the ones the system discovered in a self-supervised way.

In section 4.3, I struggle to understand how the hierarchical binary latent space shown in Figure 4 is discovered. Are you training first on `Mass`, then the result on `SizeMass` and then on `ShapeSizeMass`? Is Figure 4 only reporting the results after `ShapeSizeMass`? How do you build these hierarchical binary clusters? Figure 4 shows a 2D PCA with clear clusters, are you using a clustering algorithm on the 2D PCA projection of the trajectories? Did you arrive to this hierarchy through manual analysis?

Overall, this strikes me as an interesting direction to pursue, but the experiments have not fully convinced me that the proposed approach of "causal curiosity" (ie. finding a sequence of actions that maximizes the inter-cluster distance between trajectories and minimizes the intra-cluster distance) is the right approach. I believe experiments on environments that are not so clearly binary and hierarchical (shape / size / mass) would be needed.

That being said, I believe the paper is interesting enough to be accepted, especially if reviewers with more background on the topic judge it valuable.

## Typos

* I believe you are missing some $\\max\\{...\\}$ in equation 3.
* Note that *the* since each causal factor



review_title: Review of Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning

final_score: rating: 5: Marginally below acceptance threshold
review_content: #### **Summary**

This paper develops an intrinsic reward to help identify factors of variation within a family of MDPs. This intrinsic reward takes a form of curiosity and is used to develop initial behaviors to identify the causes of the latent variation in the environment dynamics. The experiments are used to validate the proposed intrinsic reward across several analyses used to identify its utility and effectiveness.

#### **Assessment**

I found the conceptual basis and framing of the paper to be very strong. The authors have highlighted shortcomings in prior methods that learn contextual parameters over the variation in the observed dynamics of environments and proposed an improvement in a simple combinatorial setting. I do however have strong reservations about how the paper is presented and as such cannot recommend it for publication as currently written. There is very little technical content that describe the methods and how they are used to learn effective behaviors. I also do not feel that the paper properly outlines the constraining assumptions and limitations of the proposed approach (see "Weaknesses" for more detail). The authors make claims in the abstract (e.g. "learnt... with approximately 2.5 times less data...") that aren't addressed or discussed in the main paper. In some ways this paper, as currently written, doesn't feel complete.

#### **Strengths**
- The paper appropriately criticizes prior approaches for not learning a disentangled embedding over the contextual parameters of variation within an environment. While a counterargument could be made that these approaches are looking for a low-dimensional, high information content parameterization of the factors of variation, the authors correctly point out that this limits the behaviors that could be used to efficiently identify what the causal factors are. 
- The proposed solution to this shortcoming is a simple enumeration of factors of variation used to cluster trajectories with divergent behavior along the specified factor.
- This clustering of the trajectories by action sequences is used to define an intrinsic reward that encourages a clear separation between observed behaviors.
- The experiments identify salient factors of variation, allowing the learned policies to leverage the context in meaningful ways when operating in a variety of settings.

#### **Weaknesses**
I will highlight various weaknesses that I found while reading this paper below. Before I get to these things, I wanted to first discuss the major weakness of the paper that I touched on in my assessment of the paper. As currently written, there is surprisingly little technical information about how each of the components of Figure 1 are implemented. The most concerning omissions are about how the intrinsic reward is computed and how the interventions are chosen/estimated. While there are some limited details provided in the paragraphs following Equation 3, it is unclear how the clusters $S$, $S'$, $S''$ and $S'''$ are determined and how they are related. There is also no justification for the construction of Equation 3. Why 3 clustering terms? What would happen if you compared $S''$ and $S'''$? Is there any semantic meaning behind these set clusters? What is the significance of maximizing the inter-cluster distance while minimizing the intra-cluster distance? How does the objective 3 assist in computing the factors of variation $z_{(j)}^{(i)}$? What range of values are the $z_{(j)}^{(i)}$ constrained to be? How is the embedding vector provided to the policy network? Is it used to contextualize the observed state through concatenation or is it used to condition the intermediate feature representations internal to the network? What does the training and inference procedure look like? Is there psuedocode that can be incorporated into the main body of the text? The omission of these various technical details has severely weakened what appears to be a very good paper with informative results.

A second major weakness of this paper is in how the causal factors are represented. The construction of the vector $H$ assumes some *a priori* knowledge about the known causal factors of variation. While these may be possible to enumerate and abstract in simple simulated experiments such as the robotics setting demonstrated in this paper, this will not scale to larger dimension scenarios where the number of known and unknown factors of variation will potentially be very large. Without any understanding of how the individual $z_{(j)}^{(i)}$ are computed, it appears (based on the initial paragraphs of Section 2.1) that there needs to be several independent experiments + interventions for each of the K elements in the vector $z^{(i)}$. This is a significant limitation of the proposed approach in comparison with the BAMDP and HiP-MDP approaches that attempt to generalize over a dense continuous parameterization of the factors of variation. As proposed in Perez, et al [AAAI; 2020] this continuous parameterization can be factored to account for different forms of variation while also inferring a dense embedding to contextualize the dynamics and reward functions. It could be suggested that the core contribution of this paper under review is a more concrete or principled form of disentangling the causal factors of variation, but without a more formal definition it is difficult to fully ascribe where the true contributions of this paper lie. The proposed future work of determining a dense embedding seems to be overlapping with the cited work of Locatello, et al as well as the BAMDP and HiP-MDP approaches named here. Further clarification about the assumptions and limitations of the proposed approach would greatly improve the paper. As currently written, there is very little technical context by which to evaluate the core ideas and methods proposed. From a high-level view, this paper has proposed a small contribution in a highly specific sub-category of the more generalized methods is aims to improve on.

*Minor weaknesses:*
- It's not clear that the interventions will always be maximally informative. In general, what happens if more than one causal factor of variation are tied to the action. Say, for example mass and texture (eg. heavy and smooth) may jointly make it difficult for an agent to lift the object because of the confounding factor of not being able to get a secure hold on the object? 
- The choice of horizon for the experiment planner (third sentence following Equation 3) seems oddly specific for the experimental environment and also seems out of place at that juncture of the paper.
- How are the "2 main thrusts" listed at the beginning of Section 4 directly linked to the proposed contributions listed at the end of Section 1?
- The statement made at the end of the first paragraph of Section 4 is possibly only true in the isolated environment that was used for the experiments. Being more forthcoming with this context and constraint is most accurate and intellectually honest.
- There are no specific details provided for the tasks used in the experiments. What are the range of shapes and sizes used? How similar are the various tasks? When evaluating for strong generalization, how far OOD are the test tasks? 
- Results are said to be provided over 10 random seeds but there are no error bars or forms of variance provided in the figures.
- The only comparisons used are ablations of the proposed approach. Why not use direct baselines with the BAMDP or HiP-MDP methods? Without such comparisons, it is difficult to believe the stated conclusions that the proposed approach has fully made a contribution. 
- The tables included in figures 3 and 5 are not explained nor are they very clear. What are the major takeaways that one gets from these?
- It would be nice to see where within the PCA over the trajectories, the hierarchies break down within the separate clusters. 

#### **Additional Comments**
I want to close my review by commending the authors. This is good work and presents an interesting initial improvement on prior methods for generalization among related MDPs. While much of the writing and analysis of the proposed approach is qualitative in nature I feel that, if properly presented, it could be a good paper that would be worthy of acceptance to this conference. There are however significant revisions needed in order for the paper to be ready for publication. The core concepts are good and the framing is correct. But the significance and applicability of the method is hard to place due to the lack of technical detail as well as the omission of any discussion about the core assumptions and limitations of the proposed approach. Having said this, I would be inclined to raise my score if the authors were able to sufficiently address the two major weaknesses that I pointed out above and provide adequate clarification over areas of the paper that I may have misunderstood and not fairly evaluated. 

- - -

### After rebuttal and discussion period

I want to state once again that it was a pleasure to iterate on this paper throughout the rebuttal period. It was gratifying to see it improve significantly. The reviewers all agree that this is a very valuable research direction and the authors have begun an extremely interesting line of inquiry. However, there is still a good deal left to be completed prior to this paper being fully suitable for publication. There are some critical areas of improvement still needed in improving the overall clarity of the proposed approach.

One piece that stood out to me when re-reading the final submission and the other discussions with the reviewers, it's left unstated with any formal language how the causal interventions are chosen. My concern along these lines are a dressing up of monte-carlo approaches of varying the separate causal factors in more sophistication than is actually present in order to make the paper seem more concrete. Not saying that this is how I read the paper, but there is some space to wonder and be concerned by this. Along these lines, the distributions $q(z|S)$ are never concretely formalized nor is the inference process fully detailed surrounding how one may infer the $z_j^{(i)}$. I could maybe guess that the $z_j^{(i)}$ could perhaps be the cluster assignment but this shouldn't be something left to speculation and guessing... A simple statement of how these distributions are parametrized or even approximated would go a long way.

As a final note, I would suggest that the authors revise their conclusion and discussion sections to incorporate the limitations inserted into the Appendix. This follows from the discussion I had with the authors about the general scoping and applicability of the proposed causal curiosity mechanism. It is implicit that the authors have robotics applications in mind yet have written in a more general purpose manner. I believe that adjusting their focus and naming the robotics-directed focus explicitly will help immensely.

In the end, I unfortunately cannot recommend this paper for publication in its current "final" form for this conference. I do however really look forward to the complete and fully published form after another series of revisions and refinements by the authors. Best wishes.
review_title: Great concept and appropriate framing, inadequate presentation of the technical content, missing overall set of assumptions and limitations.

Meta-review: This paper discusses how one can equip reinforcement learning agents with an intrinsic reward function that helps identifying factors of variation within a family of MDPs, effectively allowing agents to do experiments in the environment. This is interpreted as causal factors that control important aspects of the environment dynamics.

Although this is a very relevant topic and there was extensive discussion during the discussion phase, with reviewers acknowledging that the final version of the submitted manuscript substantially improved over the original submission, most reviewers still recommend the rejection of the paper. This is mainly due to the assessment that there are still several unclear technical aspects related to the paper. Shortly, the reviewers felt that the paper had important clarity issues, that the claims being made were imprecise, and that there was a dearth of details about the empirical results, making them not fully convincing.

I strongly recommend the authors to take the reviewers suggestions into consideration to have a much stronger submission to future venues.

Decision: Reject