Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Summary. In reinforcement learning problems, many entities often act independently and only have influence on a small set of other entities. The paper aims to use causality to discover which components of state space are affected by agent’s action and leverage this fact for a more efficient reinforcement learning. To detect cause influence between state component and action, the paper uses mutual information between that component and action (conditioned on current state). Finally, it leverages the detected causal influence to (i) help with exploration by using it as exploration bonus, (ii) take actions with maximum causal influence and (iii) prioritizing experiences with causal influence for training policy. Overall, the paper shows improvement over baseline algorithms in state based robotic tasks.
 
My main concern is the applicability of the method to image-based settings. I do think latent variable models will cause issues and I am not sure about the extent of the problem. To be fair, authors mention this issue as well in the discussion section. It would be interesting to see how this method performs if state variables are replaced by latent variables (coming from some pretrained generative model; authors can use some expert data for pretraining the generative model for a start to get good features)  

main_review: Originality: The paper presents a novel way to incorporate detection of causal influence in state-based RL framework.

Quality: The claims made in the paper are backed up by proper experiments.

Clarity: The paper is clearly written.

Significance: The paper does a good job of showing importance of leveraging causality for sample efficient RL. However, it’s hard to know the long term significance without knowing the applicability of the method to image based setting.

limitations_and_societal_impact: It is not addressed in the paper
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a state-dependent measure of “causal action influence” (CAI) to detect to what extent the agent can affect the next state. The high-level conceptual contribution (as stated by the authors at the beginning of Section 2) is that the measure should in fact be state-dependent. After introducing the measure and a tractable approximation for it, the authors demonstrate various ways in which it can be used to improve the efficiency of learning:
1. Guiding exploration by rewarding the agent for visiting states where it is determined to have more influence
2. Guiding exploration by selecting actions which are determined to have the most influence (among those generated by the policy)
3. A prioritized experience replay sampling scheme which upweights trajectories where the agent visited more states where it has high influence
These approaches are also complementary, and combining the three leads to better results than any alone.
main_review: I like the approach of estimating causal influence in the environment. It does seem intuitive that most “real-world” RL tasks would involve causal relationships due to their sequential nature, and uncovering those relationships should lead to more efficient learning.

The CAI estimator presented in the paper has several complementary uses in efficient exploration and learning, so the contribution of the work is clear. The primary downside of the approach is that it requires some domain knowledge, such as a known factorization of the state space and full observability, as well as an idea about which coordinate(s) are likely to contain causal relationships which are useful for exploration. However, in the cases where such information is known, there is a clear benefit to using it.

One suggestion to make the paper even stronger would be to evaluate against another baseline or two for the exploration experiments, e.g. a model-based surprise or random network distillation. The paper mentions a variety of approaches in the related work, but only compares against VIME.

The paper is clear and well-written. I have no significant issues with the writing or organization. The appendix contains useful information such as hyperparameter values and additional plots. I particularly like Figure S4, which plots the CAI over time along with corresponding images of the robot.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors study how to determine what factors in an environment can be controlled by the agent using a causal setup. One of the contributions is a perspective that causal influence is dependent on situation. They propose to use the conditional mutual information to measure if a future state variable is independent of action conditioned on the current state. Of course, estimating this value is difficult. There are some simplifying assumptions made: that the per state variable transition distribution is Gaussian and various quantities can be predicted by neural networks. After an empirical experiment validating that their Causal Influence Detection approach outperforms other attention-based methods, the authors propose 3 ways in which this detector can be used to improve sample efficiency in RL: 1) better state exploration, 2) causal action exploration and 3) prioritized replay with causal influence.
main_review: originality: Causality and RL are a combination that has received a lot of recent interest, but I found the connection to be well-used in this work to justify the objective. While the objective itself is not particularly novel, the authors derive a way to estimate the quantity in a way that seems to work well empirically. 
Some of the three ways in which this quantity is used have certainly been tried before but, again, the combination of the novel measure and the empirical results are compelling.

quality: This paper was a pleasure to read. The experiments are thoughtfully designed to break down the different contributions and show sensitivity to introduced hyper parameters. Baselines are also thoughtfully chosen and explained for most of the experiments. This leads me to some questions and suggestions:

For Figure 4, why was no other exploration method compared to? There are similar intrinsic reward-based exploration methods I would assume to perform similarly, like a count-based method or one based on “surprise,” as measured by epistemic uncertainty via ensembles, or even just simple entropy regularization.

clarity: Same as above. I found the background thoughtfully presented, derivation of the method well explained, and experiments well-designed.

significance: The contributions of this work are substantial. An effective method for computing mutual information quantities in robotics environments will be of great use to RL methods for those applications. The authors have also presented three ways to use that quantity to improve sample efficiency over SOTA in reasonable environments significantly.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work is built upon the idea of independent causal mechanisms in reinforcement learning environments where an agent only has limited inference over other entities in a given situation. Based on this premise, the authors introduce a measure called situation-dependent causal influence (CAI), which is conditional mutual information. The authors show how to estimate CAI and how to use it in exploration and RL policy training. 

main_review: I like the idea behind the work: because an RL agent has only a limited invervetional range, exploration and experience replay in RL could be more focused to be effective. At the same time, there are a couple of issues that need to be addressed/clarified. 

First, it seems that simple heuristics could achieve the same goal. Specifically, the work seems to be targeting rotis environments (line 192). In this case, my feeling is that CAI identification be rather straightforward; similar to the heuristics used in the simulation (line 216) - the agent can only affect the states of the possible movement range of the robotic arm in one step. Clearly, exploration should be more in these areas. Would that suffice? When would we need the more sophisticated algorithm proposed in Sec. 4?  

Second, in the current evaluation, the authors evaluate how effective their algorithms can estimate CAI in Sec. 5. Clearly, their algorithm outperforms others significantly. There are two questions. First, how about a simple heuristic, like the one mentioned above? Second, just estimating CAI itself is not the ultimate goal. The ultimate goal is training an RL agent effectively. In Figure 5, no other exploration method is compared except fixed \epsilon-greedy. In Figure 6, only VIME and its variants are compared. In the literature of effective exploration, many algorithms have been proposed.   A more thorough evaluation is needed to really demonstrate the benefit of the proposed algorithm. As shown in Figure 7, CAI has similar performance with EBP in two cases and outperforms EBP in one. 

There are a few strong assumptions needed in the paper. On one hand, I understand the necessity of starting with something simple. On the other hand, I wonder how much it would be possible to relax some of these assumptions. First, the model requires a known space factorization. How critical is this assumption? In robotics environments, could this be done relatively easily using some heuristics? Second, the paper is based on the causality framework, which I greatly appreciate. However, causality has its strong assumptions. I was wondering if the work really depends on the causality framework. It seems to me that the key component is to evaluate the conditional mutual information, presented in Eq. 2. Could we start from that? That would make the work much more general. 

limitations_and_societal_impact: The discussion is satisfactory.
review_title: null

Meta-review: I would like to congratulate the authors on an excellent investigation. This paper makes very nice progress in the combination of RL and causal inference, and all the reviewers and myself found the work exciting and important. 

Please make sure to address the reviewers' feedback in the final version of the paper.
Decision: Accept (Poster)