Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors proposed to learn causal relations to accelerate the
learning process in deep reinforcement learning. The proposed approach
first learns a graph to represent the causal structure of the
environment calculating the Average Causal Effect (ACE) between different
categories. It uses an intrinsic reward to encourage interacting with
the most relevant entities of the causal graph which accelerates the
learning process.

The proposed approach first need to create a factored state space,
given by the user. It is not clear how intuitive is this process and
if it can be performed automatically.

They first learn a model of the environment, predicting the next state of
each category of entities. For that they obtain data of the agent
interacting with the environment and use an encoder-decoder approach
to obtain k categories. Then predict the next state based on the
current state and an action, where the decoder predicts the next state
of each category. 

It is not clear how to determine the number of categories? (how many
decoders?) If there is a single agent interacting with different
objects, a new category needs to be defined for each object?

It then evaluates the ACE between every pair of categories. Following
Chattopadhyay et al., they used the causal attribution method to infer
the causal relations between entities. The proposed approach then
constructs a DAG, where an edge between category "i" and "j" is placed
if ACE i->j > ACE j->i. The DAG represents a causal ordering where
categories with higher ACE value are placed before. Each category is
given an intrinsic reward, in addition to the extrinsic reward, based on
its order.

The approach was tested on two domains and compared against a flat DRL
(A2C) and Relational DRL with clearly better results.

The results should also include the time used to obtain the causal graphs.

Typos:
... analysis et, al (Ren et al. ...
... deconvlutional ...
... cause that the ewe ... => ... cause the ewe ...
... the the ...

Pros:
- The use of causal relations to accelerate the learning process
Cons:
- The approach requires manual segmentation of the state space into
categories 
- Tested on two new domains and compared against a baseline an another
algorithm, so it is not completely clear the generality of the approach.

review_title: A novel approach to learn causal relations and used them to accelerate the learning process of deep reinforcement learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors propose a new deep reinforcement learning algorithm that learns the causal graph representation of the environment and leverages it to assist policy learning.

The authors should define a critical entity since they mentioned it many times in the article. Since the learned graph considers categories as nodes, I did not understand
the reference of critical entities as top-ranked nodes in the graph (Sec. Calculating ACE).

I would like also to see a proper definition of the entities and how the learned causal graph of entities impacts the performance of the proposed method.

There is confusion about the nodes in the causal graph, sometimes the authors mention them as entities:

Abstract: “critical entities of the causal graph”
Introduction:: ”The proposed algorithm learns a graph to encode the relation information between entities”.

And they change to consider nodes as categories of entities:

Method: “we calculate ACE values between categories of entities, which are used for constructing a causal graph G”
Sec.3.1: “we focus on the causal relation among categories of entities.”

For my review, I am considering nodes as categories of entities, given that the authors discuss a restriction on learning the category causal graph
rather than the entity causal graph to make their approach scalable. But this inconsistency and lack of definition lead to confusion.

My main concern about this paper is about the experimentation, specifically: baselines. 

I consider that the requirement of no prior knowledge is an interesting and valuable contribution of this paper. However, in scenarios in which we have such information, I do not consider that we should ignore it. Thus, I consider it is an open research question the application of the proposed method in these scenarios.
In this sense, I consider that the approaches proposed in [1,2] should be considered as baselines in an experiment where CARE receives the causal graph and does not
“update" it during the sequential decision-making process. The categories of the given entity causal graph could be inferred to build the causal relation among categories
of entities as assumed as input to CARE. 

I did not understand why CARE is not appropriate to Text-Adventure Game [3].

[1] Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
[2] Varun Kumar Vijay, Abhinav Ganesh, Hanlin Tang, and Arjun Bansal. Generalization to novel objects using prior relational knowledge. arXiv preprint arXiv:1906.11315, 2019.
[3] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2018.

References should point to the published work rather than the arxiv entries, when the former is available (e..g. Neural Network Attributions: A Causal Perspective, ICML’19).

Typo:

 - Sec.3.2: “of several deconvlutional …” -> “of several deconvolutional …”
review_title: Interesting method for deep reinforcement learning that learns causal structure of the environment and leverages it to assist policy learning.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: 
The paper proposes a deep reinforcement learning algorithm of advantage actor-critic (A2C), which ﬁrstly learns the causal structure of the environment and then leverages the learned causal information to assist policy learning. The causal structure is computed by calculating Average Causal Effect (ACE) between different categories of entities, and the authors assume that an intrinsic reward is given to encourage the agent to interact more with critical entities of the causal graph. Two experiments were conducted on simulation environments (Shepherd and Ant tasks) and demonstrated the effectiveness in obtaining the rewards and interpretable and accurate detection of the true graph.

Reasons for score: 
The motivation and solutions were simple and clear, but the related work about deep reinforcement learning leveraging causal relation was not properly introduced. Furthermore,  there were unclear points in the implementation (without sharing codes) and the results. I think the idea is interesting and contributed to this community, but for the above reasons, it is difficult to provide a higher rating. 


Pros:
1. The authors designed causality-based intrinsic reward to an agent, to encourage it to interact with critical entities for accomplishing the task.
2. Two experiments were conducted on simulation environments (Shepherd and Ant tasks with heterogeneous agents) and demonstrated the effectiveness in obtaining the rewards and interpretable and accurate detection of the true graph.

Cons:
1. There were also other researches in deep reinforcement learning leveraging causal relation (e.g., [i] and [ii] below). The title and the first contribution of this paper should be more specific and the related work should be included in the main text.
2. There were unclear points in the implementation (without sharing codes) and the results (below). 

[i] W. Shi et al.  Self-Supervised Discovering of Causal Features: Towards Interpretable Reinforcement Learning, arXiv, 2020 
[ii] W. Agnew and P. Domingos. Relevance-Guided Modeling of Object Dynamics for Reinforcement Learning, arXiv, 2020 

Other comments:

There is little information about the algorithm of Relational DRL (Zambaldi et al., 2018) as one of the baseline. Relational DRL seems to be an attention-based A2C algorithm, but is the difference only related to the attention-based approach? At least, whether it is an A2C algorithm or not should be clarified. 

In both experiments, is the number of categories K the number of agents? This was not specified.

I did not understand Figure 5 right and the sentence (”The value of coordinate (x, y) in the heatmap corresponds to the effect of the agent on the ewe when it is on position (x, y) and taking action drive”). The correspondence of xy coordinates with Figure 5 left might help my understanding. 
review_title: Marginally below acceptance threshold

Meta-review: The work proposed to learn causal structure of the environment and use the average causal effect of different categories of the environment, between the current and next state after performing an action as intrinsic reward to assist policy learning. While the reviewers find the ideas presented in the paper interesting and of potential, there are some concerns regarding proper introduction and comparison to related works, and clarity of the algorithm itself. While the two experimental results presented in the paper do show the potential of the work, it is missing an important baseline to disentangle the effectiveness of introducing the causal structure alone vs intrinsic reward. For example, how would A2C with curiosity or surprised based intrinsic reward, which also introduce the surprisingness of the next state as a result of performing an action as additional reward perform on these tasks? 

Decision: Reject