Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: SCALE is a representation learning algorithm for skills which are (1) generalizable, (2) account for multi-modality of robot skills. To achieve (1), they use causal feature selection to extract out a relevant subset of state variables, used for that skill policy's input. To achieve (2), they define a mode to be a set of action parameters which share the same relevant state variables from causal discovery. SCALE proposed to learn skill policies by partitioning a collected dataset by action relevant variable set and data generating region relevant variable set. The final system is a tri-level policy:
 - In the first level, a learned pre-condition classifier is used to pick one of regional compressed options.
 - In the second level, the chosen regional compressed option's upper level policy infers the low-level (third-level) policy's parameters given the compressed context.
review: ## Strengths

 - Video is informative, and was crucial for me to fully understand the approach. However it could still be improved by defining terms from CREST before using them.


## Weakness
 
### Accessibility to the Robot Learning Community

In its current state, the paper is not educational to those who are not fully familiar with the causal learning literature (the majority of robot learning researchers).
This is because many causal learning technical terms are used without informative examples, and assumptions deviating from the typical robot learning paper are not correctly spelled out.
Because this makes it challenging for many to understand the approach, appreciate its contributions, and grasp its limitations, people will be less likely to enter the space, extend the current frameworks, and relax existing assumptions in causal robotics.
This will create a greater divide between causality robotic researchers and the rest of the robotics learning community.
This reason is the key reason for my strong rejection.

To address this, I recommend re-writing with the goal of educating the average robot learning researcher.
I think if the paper do this in a self-contained manner and the figures are clear, many more can learn from and be excited by its contributions.
For instance, I find Figure 1 and its caption confusing because it contains symbols and uncommon words like data-generating regions and instance solvers without definitions.
Further, without watching the supplementary video, I would not have understood that the colored rectangles were blocks, not bar charts.

I understand that causality & robotics is still nascent, therefore many assumptions needs to be relaxed before it can be widely applied to robotic systems.
The limitations has listed all of these major challenges of integrating causality into robotics.
However, before causality & robotics become as mainstream as control & planning or neural networks, I think interleving these limitations in the paper writing will help the average reader more.


### Skill Definitions Depends on States

SCALE's formulation of clustering action parameters into different skills by the set of causally-relevant state variables is a interesting attempt at defining what a skill is. It is a useful and interesting application of CREST. 
However, this approach relies on an interprettable, controllable state representation (an requirement inheritted from CREST's context interventions).

The current approach can't be applied to general robotic systems (e.g, deformable/fluid manipulation), and causal representation learning's goal of learning disentangled features do not imply they will be interven-able.
This means SCALE's skill learning formulation will either remain unapplicable to domains for which state-estimation is difficult or require an even stronger notion than causal disentangle feature to be achieved.

### Minor Suggestion

Many notation is introduced in 3.1 that are never used again. I recommend removing them for clarity sake. In other cases, I would recommend making sure all definitions are in 3.1. For instance, pre-conditions is used throughout the paper, yet only defined in L135.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper, in its current state, is inaccessible to people outside of causality and robotics. I think with some rewriting, many more could learn from this paper's findings.

**Update**

My main concern was the original writing and presentation was poor and inaccessible to the wider CoRL community beyond causality robotics researchers. The authors have significantly improved the writing, sufficiently addressed all my questions, and explained clearly some of the limitations/scope of their approach. I think their attempt at defining skills is interesting and insightful, although clearly limited by its dependency on an interven-able state representation. My final recommendation is a `weak accept`.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes a method for causal learning in simulation by performing sample-based causal interventions (randomizing the environment variables) measuring task success. The learned causal relationship is captured by a dimensionality reduction matrix that filters out unimportant environment variables. Experiments validate that the method is able to extract a very compact skill library from large a number of environments.
review: 
Pro:
1. Causal learning in the face of distractor features is an important and often understudied task in robotics. While most existing methods rely on hand-engineering of causal features, this paper provides a feasible way of learning such relationships automatically in simulation. 
2. The method offers more interpretability to robot learning, as it is able to categorize learned policies into different skills, and pinpoint the relevant features.


Cons:
1. Clarity on main skill formulation. The main method sections (section 4 and 5) are very heavy on notations. Yet, many of the notations are only loosely defined, without a concrete example to ground them. More specifically, how are D_{A_k} and D_{D_k} computed and what are their differences? While I was able to infer them from the tables in the experiment section, I was left confused after reading section 4.1 and 4.2. 
2. Clarity on figures. Would be great if the authors could elaborate on the causal feature selection section in Figure 1. More specifically, why in certain cases, the red block is kept in the final policy-relevant parameters and in other cases removed? Why is the final example failed? What is the implication of the final example? While some of these are illustrated in appendix E, it would be good to provide explanations in the main text.
3. Sample Complexity. The causal structure is learned through preforming random scene generation (section 5.1) and policy learning in all generated scenes. However, for more complicated causal structures — for instance, multi-step block stacking with complex rules — this would likely result in very  large number of scenes in order for the causal structure to be extracted. I wonder if the authors could shed any light on the sample complexity of batch data generation.
4. Visualization of the learned skills. While the experiments on learning causal policies in block stacking and peg insertion are very essential parts of explaining the main approach, the paper glosses over the resultant skills (k1-kn) in the main text, leaving it very difficult to interpret table 1 and 2 without going back and forth between the main text and the appendix.  The authors could consider moving the explanatory toy example in the main paper, and additionally, offer more insight into the individual skills in each experiment.


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Valid approach for learning causal features and improves policy interpretability. However, readability is a weakness and the paper could benefit from a rewrite. I will re-assess if the authors could address the issues in the review.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: In this paper, the authors introduce a novel method for skill discovery, which explicitly manipulates task parameters, referred to as the 'context.' This method employs simulators to observe the resultant effects of such manipulation and identify skills by the relevant subspace of context.
The authors evaluate the method's efficacy using various manipulation tasks, where it demonstrates an improved success rate in comparison to other ablation methods.
Furthermore, the authors showcase the applicability of the framework with a real-world robot.
review: **Strengths**

1. **Interpretable Skills Discovery**: A major strength of the paper is its capability to discover interpretable skills.
2. **Context Space Subspace Discovery**: Another commendable feature is the ability to discover a subspace in the context space, making the framework more transferable to real-world applications.

**Weaknesses**

1. **Dependency on Prior Work**: The manuscript relies heavily on the understanding of CREST [33], making it difficult for readers not familiar with this work to grasp the concepts and significance fully. Reducing this dependency would make the paper more accessible.
2. **Lack of Comparative Discussion**: The paper falls short in discussing how the proposed method stacks up against other skill discovery methods.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The manuscript at hand builds upon previous context compression methods to extract skills in an option-oriented manner. The proposed framework, validated through multiple robotics tasks, certainly sparks interest. However, the paper falls short in providing a comprehensive discussion and comparison with other skill discovery methods.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work takes the options framework and provides an option discovery and skill-learning framework. It uses factor-specific interventions to determine which causal factors are relevant for reward in a given state. It then fits a one-class algorithm to the dimension-reduced factor-involving data to get the initiation set, regresses a policy and uses evaluation data to determine the preconditions.
review: While the overall intuition is relatively understandable, the details of the algorithm itself are sparse enough to the overall algorithm is difficult to follow. For example, It is not clear how the initiation set is differentiated from the initiation set. It is not clear how the policies are cleared using regression, nor what reward function is being used to train them. It is also not clear why a matrix is used to represent feature selection, instead of x_i notation, because it is unclear if a true dimensionality reduction method would be applicable to this case.
Figure 1 should help provide some of the details, but it does not really provide much beyond the same high-level overview.

The experiments provide some ambiguous results. In particular, in the block stacking task it is not clear why none of the obstacle blocks are used in any of the skills. If they are completely irrelevant to the task, then it seems to limit ---wouldn't the context specificity be useful for detecting the particular times when the distractor blocks actually inhibit motion? If not, then it is not clear why a policy composed of skills that are agnostic to these objects performs better than baselines. 

By using final reward for the interventions, it seems like this could be a significant limiting factor. If a causal chain is required to achieve high reward (ex: An agent has to grab a stick to push a block to get reward), then interventions on one factor alone would give no information about the final reward. 

This work could benefit from being more upfront about assumptions. Not only is the decoupled simulator assumption quite strong, but also the one-class initiation set, the regression-trained policies and the ability to find clear separations in the dataset, and the assumption f a Factorized dataset in the first place. 

Some discussion of the causal relationships and the nature of the interventions would provide valuable context that this work currently lacks.

The work could benefit from additional ablatives to determine the effectiveness of some of the components: The policy regression instead of RL, the SVM one-class model, and the counterfactual testing region. 

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The work introduces some interesting ideas, but the presentation is somewhat terse and the experiments do not provide sufficient evidence that this method will work generally, though the algorithm is described as being general.
review_title: null

Meta-review: The general consensus is that this is a good step towards an important topic (causality in robotics) with a reasonable set of experiments. However, the reviewers are generally confused about when the proposed methodology can be applied. The updated paper improved the understanding and all reviewers granted weak acceptance. 

I believe the proposed method makes an explicit assumption that the world can be modeled as a *factored* MDP where a discrete set of variables (ex. gravity) is used to define the model and the intervention is done using these variables. Perhaps stating this clearly and giving examples of interventions would mitigate the issue regarding the applicability of the method.
Decision: Accept (Poster)