Did I do that? Blame as a means to identify controlled effects in reinforcement learning
Authors: Oriol Corcoll
Youssef Sherif Mansour Mohamed
Raul Vicente
Venue: Accepted by TMLR ()

Abstract: Affordance learning is a crucial ability of intelligent agents. This ability relies on understanding the different ways the environment can be controlled. Approaches encouraging RL agents to model controllable aspects of their environment have repeatedly achieved state-of-the-art results. Despite their success, these approaches have only been studied using generic tasks as a proxy but have not yet been evaluated in isolation. In this work, we study the problem of identifying controlled effects from a causal perspective. Humans compare counterfactual outcomes to assign a degree of blame to their actions. Following this idea, we propose Controlled Effect Network (CEN), a self-supervised method based on the causal concept of blame. CEN is evaluated in a wide range of environments against two state-of-the-art models, showing that it precisely identifies controlled effects.

Published in Transactions on Machine Learning Research (07/2022)
Did do that? Blame as a means to identify controlled effects in reinforcement learning
Oriol Corcoll
wriol.corcoll.ondreede
Institute of Computer Science University of Tortu
Youssef Mohamed
yousee/.mohmmeddhu...c
Instibute of Computer Science University of Tortu
Raul Vicente
raul. www.ntt..a.fwwuu...l.......
Institute of Computer Science University of Tortu
Reviewed on OpenReview: https: openreview. net/ net/feruue forum? NL2L3X/VFe
Abstract
Affordance learning is crucial ability of intelligent agents. This ability relies understand- ing the different ways the environment can be controlled. Approaches encouraging RL agents to model controllable aspects their environment have repeatedly achieved state-of-the-art results. Despite their success, these approaches have only been studied using generic tasks as a proxy but have not yet been evaluated in isolation. In this work, we study the problem of identifying controlled effects from a causal perspective. Humans compare counterfactual outcomes to assign degree of blame to their actions. Following this idea, we propose Controlled Effect Network (CEN), self-supervised method based on the causal concept of blame. CEN is evaluated in wide range of environments against two state-of-the-art models, showing that it precisely identifies controlled effects.
1 Introduction
The recent succoss of reinforcement learning (RL) mothods in complex environments such as Hide & Seek (Baker et al., 2019), StarCraft II (Vinyals al., 2019), or Dota2 (OpenAT al., 2019) has shown the polential of RL to learn complex behavior. Unfortunately, these methods also show RL's inefficiency learn (Espeholt et al., 2018; Kapturowski et al., 2019; Gulcchre al., 2020), requiring vast amount of interactions with the environment before meaningful learning occurs. Consequently, environments with sparse rewards are known to be extremely difficult, making imperative a good exploration strategy. A popular approach to exploration is to introduce behavioral biases in the form of intrinsic motivators (Chentanez et al., 2005; Mohamed & Rezende, 2015). This technique aims to facilitate the learning of task-agnostic behavior by producing dense rewards, driving the agent to discover novel states and by doing so increase the chance of discovering the environment's reward.
Numerous motivators have been developed taking inspiration from humans, e.g. curiosity control (Bellemare et al. 2012b; Pathak et al., 2017; Burda al,, 2018; Choi et al., 2019; Badia et al., 2020b). Recent work (Choi et al., 2019; Song et al., 2019; Badia et al., 2020a;b) has achieved State-of-the-Art on the Atari benchmark (Bellemare et al., 2012a) by rewarding agents for the discovery of novel ways to control their environment. A typical design principle among these methods using an inverse dynamics model to predict the chosen action from two consocutive observations with the hope that the latent representation learned encloses aspocts of the environment controlled by the agent. This approach assumes that the action can be recovered from consecutive observations, which not always the case.

---

Published in Transactions on Machine Learning Research (07 (0222)
c
Health
8
uncontrolled
controlled
Figure 1: Left) Blame compares imagined normative world to what actually happened to attribute the movement of the agent and the box to the action. Middle) Using a do-nothing action as normative world is not enough since do-nothing has an effect. Right) Controlled effects are the effects of an action on the environment.
A more causal approach to compare counterfactual worlds (Pearl, 2009), i.c., an effect is controllable if the effect would have been different had the agent taken another action. caveat of this approach is that things become trivially controllable. Fig. (left) shows an scenario where an agent moves box by moving left. Here the box becomes controllable even when the agent performs the action 'do-nothing' since there is action, "move-left", that would move the box. Contrarily, it is believed that humans identily controlled effects by assigning degree of blame to their actions. In particular, humans compare what happened to what normally would happen (Halpern, 2016; Morris et al., 2018; Langenhoff et al., 2019; Grinfeld et al., 2020). If what happened is normal, humans would not assign blame to their actions, e.g. when performing "do-nothing", the box's effect would not be controlled since normally the box would not move. However, it. would be considered controlled when moving left since its normative state is to not move.
This work proposes Controlled Effects Network (CEN), completely unsupervised approach to identify controlled aspects of the environment based on the human notion of blame. CEN can be incorporated to any forward model and works by dividing its latent representation into two branches, normal and a roltrd view of the world. Our experiments show that CEN can disentangle effects precisely, outperforming state-of-the-ate approaches to detect controlled effects.
2 Identifying controlled effects using blame
Our goal is to identify changes in the enviromment that were controlled by the agent. This section introduces Individual Causal Effect (ICE), a fundamental measure in causal literature, and frames in the context of RL. We show how this mcasure can be used to identify controllable effects but argue that these are not suitable for RL. In contrast, the human perception of causality associated with the concept of blame (Gerstenberg & Lagnado, 2014). For example, if lightning hits forest tree and starts fire, humans would point to the lightning the cause of the fire, not the oxygen wood since they normally present in the forest. Consequently, we expand the idca of blame to identify controlled effects by using measures of normality and counterfactuals.
2.1 Controllable effects
What does mean to cause something? Pearl et al. (2016) provide an intuitive definition of cause-effect relations: "A variable is a cause variable Y if Y, in any way, relies on X for its value". This kind of formulation trics to answer questions like *docs smoking causes cancer?". Actual causality, proposed in Halpern (2016), studies causal relations between individual events of X and Y. It aims to answer questions like, "did smoking for 30 years caused David's cancer?". The individual nature of Actual causality makes this framework especially suited to RL problems. In the following, we introduce the concept of causal effect in the context of RL.
The individual causal effect (TCE) of an event = x on a variable Y; can be measured by comparing counterfactual worlds
ICE, = =Y Y
(1)
2

---

Published in Transactions on Machine Learning Research (07/2022)
where Y," reads as "what would the value of an individual Y; be if X is forced to be x². Similarly, Y," describes the value of Y; when X is forced to not be x. Note that the sub-index i refers to an individual, and hence in the following, we use Y and y interchangeably.
The fundamental problem of causal inference states that we can only observe one of these counterfactual worlds and the other needs to be imagined. Intuitively, Eq. 1 compares the world where the event x happened to an alternative world where event r had not happened. Consequently, we say that æ has a causal effect on Y if there is an è X that makes Eq. 1 nonzero. In the context of RL, X and Y take the form of actions, states and/or observations. Fig. 1 (right) illustrates the causal relations present in typical RL setting, where state s has an effect on both the next state s' and the produced observation o which, in turn, has an effect on the agent's 's choice of action a E A. Similarly, an action has an effect on the next state. Since states are typically not accessible by the agent, we do not use states as variables; nevertheless the same principle can be applied. We define the perccived effect eg as the difference between consecutive observations when taking action a, i.c. eg = o' o. As in Eq. 1, we say that perceived effect is controllable by the agent's action when
E (c- #0.
Since we want to know what elements of the perceived effect are controllable, the difference is an element-wise operation. It is important to notice that Eq. 2 has far-reaching consequences, for example, an agent next to a box would have a causal effect on it even when doing nothing since there is a counterfactual world where that box would have moved. If we use Eq. 2 as reward, the agent would be rewarded for almost every action at every state! Note that taking ā as a special "do-nothing" action would not work since even doing nothing does something, e.g., Fig. 1 (middle) shows scenario where doing nothing has an effect on the agent's 's health. Taking do-nothing á would not attribute the effect to the agent. Instead, we would want more human-like definition of what is controlled where agent controls box if moved or its life if a bomb could have been easily avoided.
2.2 Blame
It has been shown that the human notion of causality is affected by what is normal (Kahneman & Miller, 1986; Cushman et al., 2008; Knobe & Fraser, 2008; Hitcheock & Knobe, 2009). Here, we resort to concepts of normality from actual causality to find if the agent's 's action is to blame for what happened. Halpern & Hitchcock (2014) propose to compare what actually happened with what normally would happen. Following this idea we build a normative world to replace y in Eq.
ICEY = Y By
(3)
where By is the value Y would normally take. Such a value is of course contingent to the notion of normality used, which is for us to define. We can reformulate Eq. 3 to compute the controlled effect of an action eg as
c = -ICE, = Ex. Be,
(4)
c
E a.
c
where Be, represents what normally would have happened to the environment. This quantity not only depends on the environment's 's dynamics but also on the agent's 's typical behavior.
In this work, we compute Be, as the expectation over all possible futures of of each action in A. Note that stochastic environments may have be multiple next observations for each action. To simplify notation, the following sections use normal effect as en Ea,o' [e]] Intuitively, Eq. 4 builds a normal world by observing every alternative ep produced by cach action creating an average perceived effect. Consider the example in Fig. (middle), moving left or doing nothing would make the agent's 's health decrease. Eq. 4 would indicate that what is normal is for health to not change since in average health stays the same; thus, the loss of health when moving left or staying would be attributod to the agent. On the other hand, moving right would only position as controlled. Note that the explosion would never be credited to the agent since it would have happened no matter what action is taken.
attribute the change in the agent's 's
3

---

Published in Transactions on Machine Learning Research (07 (2222)
a ho
Controlled
he
Dec.
Encoder
0
ep
Normal
hn
Dec.
En
Figure 2: CEN divides the latent space of a forward model into controlled and normal branches. Each branch disentangles controlled and normal effects and decodes each into pixel space independently.
Special care needs to be taken when constructing the normal world Se, for continuous or large action spaces. Computing counterfactuals on an infinite number of possibilities cannot be done and some approximation needs be implemented. Although our experiments use discrete actions, the proposed method in the following soction equipped to handle continuous action spaces since it does not compute counterfactuals for each possible action but approxinates the normal world directly. It. is also important to notice that the controlled effects that Eq. 4 can identify in partially observable setting (o s) are constrained to those observed by the agent. Nevertheless, humans cannot perceive every change in state but can identify relevant controlled effects for their survival and joy.
3 Unsupervised learning of controlled effects
In practice, we do not have access every world and cannot compute Eq. 4 directly. We propose Controlled Effects Network (CEN), unsupervised method depicted in Fig. 2 that disentangles controlled and normal effects only using perceived effects as self-supervised training signal. CEN is modeled as neural network and it is based on a forward model, where observation and action are used to predict the outcome of performing such action on the environment. In contrast to conventional forward models, CEN divides its latent space into controlled and normal representations; similarly to Dueling Networks (Wang et al., 2015). These two representations approximate the controlled and normal effects in latent space. A decoder converts these latent representations into pixel space allowing estimate e + en = as in Eq. 4.
The controlled branch has privileged access to the action, consequently, having only this branch would make CEN regular forward model, i.e., the controlled branch alone can predict the perceived effect resulting from the action. Then, why do we need the normal branch? The role of the normal branch is to force the controlled branch to predict only what is not predictable from the observation alone and hence, modeling what is controlled by the agent. In way, the normal branch acts as a distillation mechanism where only what can be controlled will be represented by the controlled branch. To promote the controlled branch model only controlled effects, we use the following objective
MSE (ee + En, + MSF (G))
where the first term is the reconstruction loss in which the predicted target is compared to the perceived effects provided by the environment. The second part of the loss enforces the network to use the normal branch to model the world as much as possible. Since this branch cannot predict everything without the action the model will converge to the expected effect due to the MSE loss. Additionally, a hyperparameter a regulates how much the normal branch should model the environment. In practice, we found that this hyperparameter creates agreement between branches uncertain futures which seemed to be critical in environments with stochastic entities. Note that we combine normal and controlled effects in pixel space, this is because we would like to know the pixels that are controlled by the agent. Nevertheless, if the downstream task CEN is part of only needs the latent representation, the combination of controlled and normal representations could be done in latent space ie. he + Ktn.
Let us look again the example in Fig. (middle) and assume the agent picks the do-nothing action. The normal branch is encouraged to model the bomb since it does not depend on the action. Furthermore, it.
4

---

Published in Transactions on Machine Learning Research (07 (07/202) /022)
Figure 3:
Suite of environments used for the experiments. From left feit/rigtt top/bottom: Clusters, Lights, Spiders and Montezuma's 's Revenge (MZR).
should not predict any change health since what is normal is for health to not change. Thus, the controlled branch must model the change health.
4 Experiments
This section evaluates CEN- on the following main questions: 1) Can CEN identify controlled effects pixel-level? ie. it produce accurate segmentation mask? 2) Some applications may not require pixel-level precision; we asses CEN on predicting controlled effects at attribute-level from both pixel masks and latent representations. Every experiment reports 95% confidence interval? using three different seeds.
Environments: we use multiple environments (Fig. 3) to answer the above questions, each showcasing a different aspect of what can be controlled. These environments are based on Griddly (Bamford et al., 2020) and Atari ALE (Bellemare et al., 2012a), both using the Gym interface (Brockman al., 2016). More details of these environments are given in each experiment and appendix.
Baselines: We use Attentive Dynamics Model (ADM) (Choi et al., 2019) in experiments 1) and 2). ADM is an action-prediction model based on an spatial attention mechanism. It works by predicting the action performed on individual image patches. Then, a spatial attention mechanism selects a sparse set of patches to use when making a final prediction of the action. The masks produced by the attention mechanism are considered controlled aspects of the environment. Although ADM does not produce pixel level masks, only patch level, to our knowledge ADM the most competitive method to provide pixel-level information about controlled aspects. For 2) we rely on the inverse modeled proposed in Never Give Up (NGU) (Badia et al., 2020b), the current SOTA for exploration in RL.
Implementation: CEN is implemented as an encoder-decoder architecture with 2D convolutional layers and ReLU activation functions; the normal and controlled branches are implemented with lincar layers. Additionally decoder weights are shared. Throughout the experiments we use the same neural networks and hyperparameters unless specified otherwise. Our implementation of ADM uses the same architecture and hyperparameters proposed in Choi et al. (2019). See appendix for more details on the architecture and hyperparameters.
4.1 Controlled effects pixel-Ievel
This set of experiments explores CEN's ability to identify pixels corresponding to controlled entities. Although CEN computes the magnitude and direction the effects, we create a binary mask by setting a threshold for the predicted controlled effects (see exact details in appendix D). We report pixel F1 scores between ground truth and prodicted binary masks. We explain how ground truth masks are computed below, but should be clear that CEN is fully unsupervised method that does not use this ground truth in any way, ground truth is only used to evaluate the produced masks. The network is trained to minimize Eq. 5 using the ADAM optimizer (Kingma & Ba, 2015) and 500K samples of the form (o, eg) collected using random policy.
Networks, training and evaluation have been implemented using PyTorch (Paszko al., 2019), NumPy (Harris al., 2020) and PFRL (Fujita al., 2021); our experiments managed using W&B (Biewald, 2020). 2CT computed using bootstrap resampling per the Seaborn (Waskom, 2021) package
5

---

Published in Transactions on Machine Learning Research (07 /022)
Spiders F1 avatar
1.0
CEN (ours)
ADM
1.0
0.8
0.6
02
0.0
F1 avatar
F1 box
1.0
0.8
0.2
0.0
3
5
5
2
3
5
(ours)
ADM Training steps (1es)
(b)
(a)
Figure 4: a) CEN can correctly disentangle the agent from the randomly moving objects. b) Clusters environment where CEN is able to model not just the agent but also the movement of boxes.
Ground truth masks: Griddly provides access to each entity's state (x, y, light is on/off, etc); a binary mask is produced for each controlled entity with any of its attributes changed when transitioning between two time steps. x and y coordinates are projected into pixel space and a bounding box is generated using the size that entity. The resulting masks for each entity are combined into a single mask m by taking the maximum value among them. Since we want to know what pixels were actually controlled, the final ground truth mask is produced as: m e.
In case of MZR, Atari ALE allows us to actually compute counterfactual worlds by saving and loading the state of the game (RAM) multiple times when taking different actions. For this, directly compute Eq- 4 using ALE's special calls cloneSyslemSSlale and restoreSyslen,Slale More precisely, we compute every possible perceived effect reachable from the current state and build normal effect using the mode over all possible effects. Then, we compare the percoived effect for the agent's chosen action against the normal effect. The ground truth mask will have is where these two effects are different.
CEN and ADM masks: CEN's controlled masks are generated using the encoder, controlled branch and decoder. The predicted controlled effect is binarized using threshold T as (-T < ( > T). In the case of ADM, its atlention mask is thresholded in the same way and the mask is resized to the size of the effect.
4.1.1 Controlled vs uncontrolled effects
Here we use the Spiders environment to evaluate CEN's ability to disentangle controlled from uncontrolled effects. This environment has two main entities, the agent and a spider. The controlled masks must only focus on the agent and ignore the spider.
Fig. 4a shows the pixel Fl score for our model and the baseline. CEN is able to correctly disentangle controlled effects and can produce accurate masks. Although our implementation of ADM can predict the agent's action with 88% accuracy, is not capable of modeling the agent's controlled pixels. We conjecture that this is due to ADM's sparse softmax mechanism; nonetheless this behavior persisted when increasing its entropy weight which should produce more dense masks.
4.1.2 Nearby controlled effects
Models based on action prediction are expected to work well aspects related to the agent. For example, if agent moves a box due to moving right; the box's movement is also controlled. It is unclear why these models would pay atlention to the box since just knowing where the agent is, suffices to predict the chosen action. CEN's controlled branch, on the other hand, is motivated to model the bax's effect since the normal branch would predict that the box stays where is. We call "nearby" controlled effect to an effect that happens adjacent the agent, like the box's movement. To evaluate CEN on this kind of effects we use the Clusters environment where an agent needs to move colored boxes to their corresponding fixed colored blocks.

---

Published in Transactions on Machine Learning Research (07 (0222)
MZR F1
CEN tours)
ADM
0.8
0.2
0.0
avatar
light
0.4
0.8
0.6
3
(a)
(ours)
Training steps (1e5)
(b)
Figure 5: a) F1 score on the Lights, and b) MZR environments. CEN outperforms the baseline even in an environment with more complex dynamics and features.
Fig. 4b shows that CEN can precisely model effects on the agent and boxes. We breakdown individual effects to account for the class imbalance between the agent's movement and the boxcs. CEN seems to make more mistakes with boxes than the agent but nonetheless, it can consistently model both.
4.1.3 Faraway controlled effects
In contrast to the previous experiment, we want to evaluate CEN can model distant effects, i.c. effects that are reasonably far away from the agent's location. In this case, we use the Lights environment. Here the environment presents two buttons of different color that, when pressed, turn on their corresponding light. Lights are relatively far away from their corresponding buttons thus making it difficult to model them. As show in Fig. 5a, CEN is able to model this kind of effects.
4.1.4 Controlled effects in Montezuma's Revenge
This last pixel-level experiment evaluates CEN on Montezuma's Revenge (MZR) environment. Although agents in Atari environments have limited control over the environment, the relatively complex dynamics of Montezuma's Revenge makes it challenging test-bed. Results shown in Fig. 5b indicate that CEN can also model controlled effects.
Unlike previous environments, CEN's F1 score behaves differently due two main reasons: 1) F1 decreases over time: the Fl score extremely sensitive for empty ground truth, single pixel marked as controlled takes the score from to O. At the beginning of training, CEN produces empty masks correctly but performs poorly at controlled effects, which makes the F1 score artificially high. Over time it improves at controlled effects but starts marking very few pixels as controlled for effects where nothing is controlled. 2) F1 is lower: the sensitivily of the F1 score also affects CEN's performance on coltedl effects since pixels outside the ground truth are penalized the same no matter how close to the ground truth they are. Appendix B.3 includes additional masks showing that CEN fails by pixels close the controlled object instcad of identifying the wrong object controlled.
4.2 Controlled effects at attribute-level
In some cases requiring pixel-level precision be excessive. The following experiments analyze how different representations can predict effects on attributes from the environment's state, c.g. changes on the agent's 's (x, y) location or if a light turned on. To this end, we use a probing technique (Alain & Bengio, 2016) similar to the one described in Anand et al. (2019). This approach trains a classifier per each attribute of interest. in the environment's 's state using frozen versions of trained networks to produce the classifiers inputs. More specifically we use two different sources for the probing classifiers, pixel-level masks (as in the previous section) the model's latent representation.

---

Published in Transactions on Machine Learning Research (07/2022)
F1 PIXELS
FI LATENT
ENVIRONMENT ATTRIBUTE
CEN (OURS)
ADM
CEN (OURS)
INVERSE
AGENT
1.0+0.00 0.35+10.03
0.47+0.23
0.97440001
0.67+00005
SPIDERS
SPIDER
0.25+0.03
0.41#0.001
0.44:0.02
AGENT
0.76+0.41
0.28+0.08
0.97+10.01
0.56++0.09
CLUSTERS
Box
0.78:0.37
0.3240.19
0.95+10.02
0.77+10.00
AGENT
0.97
0.33
1.010.01
0.84110.08
LIGHTS
BUTTON
0.93.10.05
0.33100.01
0.9910.0
0.99110.0
LIGHT
0.93100004
0.4110.114
1.0.0.00
0.9911.00
AGENT SKULL
0.66110.08 0.1910.03
0,4210.25 23 0.2010.08
0.91 0.61#0.03
0.88110.02 0.61+0.04
MZR
Table 1: F1 score when predicting attributes from pixel or latent space. Lower is better for Skull and Spider.
For the first case, we produce a binary mask, as in the previous experiments, to occlude perceived effects and use these to train each probing classifier. Classifiers have to predict there was a positive, negative or none effect. Note that the classifiers need to predict any effect, not just controlled. Thus, the classifier should only be able to predict accurately controlled attributes such as the agent's 's position but should fail at predicting uncontrollable effects like the location of the spider or skull. We 35K samples of the form (m Ep. y) where is the mask produced by the model and y is the ground truth class. Each dataset is split into a typical 70/20/ 10, allowing 20% class imbalance, We report Fl score of each attribute on the test set.
random policy to collect dataset of
4.2.1 Attribute probing from pixels
The results in Table 1ffrrst block) indicate that CEN consistently outperforms the baseline when predicting controlled effects for state's attributes, and thus modeling controlled effects accurately. Furthermore, for both Spiders and Montezuma's Revenge environments the model cannot predict the uncontrolled effects, as expected. Even though ADM's action prediction accuracy was high (~ 88%) on every environment, it is not able to consistently predict controlled effects at attribute-level.
4.2.2 Attribute probing from latent representations
In this case, we train classifiers using a latent representation instead of pixels. We use the latent representation from CEN's controlled branch (ha). unclear how create latent representation from ADM, so we use an inverse model. The features of current and next observations are concatenated create latent
GT
Controlled (ours)
Normal (ours)
ADM
F1: 0.60
F1: 0.34
Figure 6: Example masks for Montezuma's Revenge. Additional masks included in appendix B.3.
8

---

Published in Transactions on Machine Learning Research (07 (07/2022)
0.0
2.0
emppty
empty
Na 11
0.4
02
0.0
(a)
(b)
Figure 7: Ablation study on the effect of a from Eq. 5 on the learned representations the normal branch. X axis of a) indicate training steps (le5)
representation of what is controlled. As before, we train linear probe to predict controlled attributes from the latent space of these models. The probe predicts changes on the x, and direction (if applies) for agent, spider, skull and boxes; and on/off for lights and buttons.
Table (second block) show that CEN improves on the baseline's performance. Although the inverse model is closer to CEN's performance than ADM, it still has difficulties predicting the agent and box changes in location. The reason of having high score in Skull is that the Skull is still only when the agent dies, making it easy to predict from the agent's s position. Removing this event leads to score of ~ 0.35 for both models.
4.3 How does the alpha hyperparameter affects CEN?
Here study how affects the loss in Eq. 5. This experiment uses Montezuna's Revenge and the same setup as in Experiment 4.1. We analyse the effect of alpha on CEN using values ranging from 0.01 to 20.
As can be seen in Fig. 7a, the different alphas do not impact the performance when the ground truth or controlled effects are not empty. On the other hand, the performance degrades the higher the alpha. We hypothesize that this behavior happens when the normal branch is forced to model controlled effects too strongly and the controlled branch needs counter those bad predictions. In this case, the controlled branch will produce wrong masks, especially for empty ground truth. Fig. 7b shows that the higher the the more the controlled branch needs to remove pixels modeled by the normal branch.
5 Related work
Intrinsic motivators: A popular way of introducing behavioral biases in RL agents is the use of intrinsic motivators (Singh et al., 2005; Mohamed & Rezende, 2015). These motivators can promote different types of exploration, from observational surprise (Burda et al., 2018) to control seeking agents (Pathak et al., 2017; Choi et al., 2019). Mothods in the latter category have shown extremely good results achieving SOTA in important benchmarks. Choi et al. (2019) proposed Attentive Dynamics Model (ADM), an attention based method that discovers controlled elements in the environment and rewards the agent for discovering them. This method and its extension (Song et al., 2019) showed SOTA in Montezuma's 's Revenge. Badia al. (2020b) combined control and observational surprise to promote exploration. Their method uses episodic memory with an inverse model to promote the discovery of controlled effects and Random Network Distillation (Burda et al., 2018) to promote long term progress; again achieving SOTA in Atari's hard exploration environments. These methods show the importance of identifying what an agent can control.
Causality in deep reinforcement learning: Causality is central to humans; we think in terms of cause- effect. similar method Blame was proposed in Chattopadhyay al. (2019), where they use causal attribution methods to analyze the effect of inputs neural network's outputs. Recent work has introduced
9

---

Published in Transactions on Machine Learning Research (07/2022)
causality into deep reinforcement learning (Schölkopf et al., 2021; Ke et al., 2021; Foerster et al., 2018; Buesing et al., 2018; Jaques et al., 2018; Dasgupta et al., 2019; Goyal et al., 2019; Nair et al., 2019; Madumal et al., 2020) showing that this is a promising avenue for the training of agents. Corcoll & Vicente (2020) proposed an attribution method to learn temporal abstractions for object-centric hierarchical RL. Bellemare et al. (2012b) compule controllable aspects of the environment by generating a mask with all possible rollable areas of an image and uses as part of the policy's input. In this work, we identify the controlled effects of individual actions using causal concepts of normality and blame. A similar concept to normality has been explored in control theory by Todorov (2009) called "passive dynamics". Passive dynamics compute how dynamical system would evolve without interventions.
6 Conclusions and limitations
This work provides a cansal perspective to the problem of identifying controlled aspects of the environment and proposes Controlled Effect Network (CEN), a fully unsupervised approach to this problem. CEN creates a normative world using counterfact and compares what actually happened with what normally would happen to attribute changes on the environment the agent. The presented experiments show that, despite being unsupervised, this method preciscly identifies controlled effects. In future work, we will explore CEN as intrinsic motivator way to discover skill in hierarchical reinforcement learning setting.
6.1 Limitations
Normalily: although we propose a measure of normality in Eq. 4, this is far from ideal. For example, the resulting normal world may not be real/possible under the environment dynamics, a more realistic solution would be to use the mode instead of expectation. This could be achieved by using critic, as in GANs (Goodfellow et al., 2014). We believe the way humans see normality is context dependent and should be learned instead of a fixed function. This is an active research area in the field of psychology where researchers look to underpin human causal judgment.
State instead of obscreations: our current formulation of ICE uses perceived effects as opposed to states. When CEN indicates that some change is controlled this is not (necessarily) equivalent to stating that the objects represented by those pixels are controlled. An exciting avenue to explore is combining CEN with a state representation model (e.g. LSTM or RSSM by Halner et al. (2019)).
Reliance on policy: since Eq. does not depend on the policy, CEN needs diverse data for each action to approximate it, idcally from random policy. This important problem is not specific to CEN, any forward model needs a policy that explores multiple actions to provide accurate predictions.
Multi-step controllability: CEN assumes that objects are controlled immediately after performing action. A valuable extension is to identify the consequences of an action multiple steps ahcad. Possibly by incorporating Blame to works like Mohamed & Rezende (2015); Gregor el al. (2016).
Acknowledgments
The authors would like to thank Jaan Aru and Daniel Majoral for insightful comments on the manuscript. This work was supported by the University of Tartu ASTRA Project PER ASPERA, financed by the European Regional Development Fund.
References
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2016. URL https://arxiv. https///axxiv.oo/abb/////// wrgg/bs////// https//11...1....0
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised State Representation Learning in Atari, 2019.
Adríà Puigdoménech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark, 2020a.
10

---

Published in Transactions on Machine Learning Research (07/2022)
Adrià Puigdoménech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies, 2020b.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula, 2019.
Chris Bamford, Shengyi Huang, and Simon Lucas. Griddly: A platform for ai research in games, 2020.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificiol Intelligence Reseorch, Vol. 47:253-279, 2012a. doi: 10.1613/jair.3912 URL http://axxvv..rg/abaaa www.:.rgg/aaaa..a. 1400b//...77788...... cite arxiv:1207.4708.
Mare G. Bellemare, J. Veness, and Michael Bowling. Investigating contingency awareness using atari 2600 games. In AAAI, 2012b.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://wwwwwadd wwndw/.wwwwwwn.coo........w..... com/ Software available from wandb.com.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016. URL
01540. cite arxiv:1606.0144:
Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean-Bapliste Lespiau, and Nicolas Heess. Woulda, coulda, shoulda: Counteffactuully--giieee policy search, 2018.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov Openai. Exploration by Random Network Distillation. 2018.
Aditya Chattopadhyay, Piyushi Manupriya, Anirban Sarkar, and Vinceth N Balasubramanian. Neural network attributions: A causal perspective. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learnáng, volume 97 of Proceedings of Machine Learning Research, pp. 981 990. PMLR, 09 15 Jun 2019. URL http://proceedings mml. mlr wwt.ppress/v/ Saattopaddyayy9a.ht...
Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. In L. Saul, Y. Weiss, and L. Bottou (eds.), Advances in Neural Information Processing Systems, vol- ume 17, pp. 1281-1288. MIT Press, 2005. URL https://proceedingss caarrppp.cc/paae/////////1/ .ddf.
Abbfafccaccat000266eeeeeeee..
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lec. Contingency-Awart Exploration in Reinforcement Learning. ICLR, pp. 1 20, 2019.
Oriol Corcoll and Raul Vicente. Disentangling cansal effects for hierarchical reinforcement learning, 2020.
Fiery Cushmann, Joshua Knobe, and Walter Simnott-Armstrong. Moral appraisals affect doing/allowing judgments. Cognition, 108()):281-225)) July 2008. doi: 10.1016/j.cognittonn200 200...00.00000.00.... URL https:
//doi. oooo..org//0..o 10t///00.001/1.... 1016/ wtw.:////.....iiooooonnoo. cognition. www.enttioon.oo.....mmmm 2008.02.005.
Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthow Botvinick, and Zeb Kurth-Nelson. Causal reasoning from meta-reinforcement learning, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Tain Dunning, Shane Tegg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, 2018.
Jakob Focrster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counter- factual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. URL https://oja aaai wet..orgggeeeee http:/inne.pppp.aaaaeaeea hhtp:///Al//aatrrcc/////e//////
11

---

Published in Transactions on Machine Learning Research (07/2022)
Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa. Chainerrl: A deep reinforcement learning library. Journal of Machine Learning Research, 22(77):1-14, 2021. URL http://jmlr org/ papers/ v22/20-376 10000-776htmm..
T. Gerstenberg and D. A. Lagnado. Attributing responsibility: Actual and counterfactual worlds. In Joshua Knobe, Tania Lombrozo, and Shaun Nichols (eds.), Oxford Studies in Experimental Philosophy, volume 1, pp. 91-130. Oxford University Press, 2014.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.). Advances in Neural Information Processing Systems, vol- ume 27. Curran Associates, Inc., 2014. URL https://proceedings meeeredddnnnnnnnnss...... caaeebh122611@f000044s/771fffff
neurips....... www.riss.ccpappr/201///////
pdf.
Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng, Sergey Levine, and Yoshua Bengio. Reinforcement Learning with Competitive Ensembles of Information-Constree Constrained Primitives. pp. 1-21, 2019.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wicrstra. Variational intrinsic control, 2016. URL
https://arxie org/abs/ ///11/1111000010
Guy Grinfeld, David Lagnado, Tobias Gerstenberg, James F. Woodward, and Marius Usher Causal responsibility and robust causation. Fronliers in Psychology, 11:1069, 2020. ISSN 1664-1078. doi: 1020.00069. www.eersiii.rrrrrrieeee///// wtt.://icc.ee.....///sss///// URL https://www. https://www.frotiierriiit
Caglar Gulcehre, Tom Le Paine, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, Gabriel Barth-Maron, Ziyu Wang, Nando de Freitas, and Worlds Team. Making efficient use of demonstrations to solve hard exploration problems. In International Conference on Learning Representations, 2020. URL
Danijar Hafner, Timothy Lillierap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels, 2019.
Joseph Y. Halpern. Aclual Causalily. The MIT Press, 2016. ISBN 0262035022.
Joseph Y. Halpern and Christopher Hitchcock. Graded Causation and Defaults. The British Journal for the Philosophy of Science, 66(2):413-44)) 447, 04 2014. ISSN 0007-0882. doi: 10.1093 /bjps/jps///o bobps//xx000 /xx0000.
Charles R. Harris, K. Jarrod Millman, St'cfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Malti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fern andee del R'io, Mark Wiebe, Pearu Peterson, Pierre G'erard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):355 362, September 2020. doi: 10.1038/s41558 2649-2. URL httpp://daii http:://dooooooo//// 241586-020-2649-
Christopher Hitcheock and Joshna Knobe. Cause and norm. Journal of Philosophy, 106(11):587 612, 2009. doi: 10.5540//phh122001011222.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A. Ortega, DJ Strouse, Joel Z. Leibo, and Nando de Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning, 2018.
D. Kahneman and D. T. Miller. Norm theory comparing reality to its alternatives. Psychol Rev Psychol Rev, 93(2):136 153, 1986.
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent experience replay in distributed reinforcement learning. In International Conference on Leorning Representations, 2019. URL. htttp:///pperreviee. http:///gppneeeeeeeeeeeerrrraaaa net////orrmm..drr111114
12

---

Published in Transactions on Machine Learning Research (07/2022)
Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery in visual model based reinforcement learning, 2021. URL
00848.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (ods.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URI. http:///xxwvvv......ee
6980.
Joshua Knobe and Benjamin Fraser. Causal judgment and moral judgment: Two experiments, pp- 441 447. The MIT Press, United States of America, 2008. ISBN 9780262195690.
Antonia Langenhoff, Alex Wiegmann, Joseph Y Halpern, Joshua Tenenbaum, and Tobias Gerstenberg. Predicting responsibility judgments from dispositional inferences and causal attributions, Sep 2019. URL psyarxiv. com/63zvv,
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. Proceedings of the AAAI Conference on Artificial Intelligence, 34(03):24993 2500, Apr. 2020. doi: 10.1609/aaaa /tt.aaaaai.vi....ee. v341.3.5631. URL https://oja aaaai.or//ine http://ndee..p///M//raeeeee http:p//AA//aattilll/////e/////s
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically
motivated reinforcement learning, 2015.
Adam Morris, Jonathan S Phillips, Thomas Icard, Joshua Knobe, Tobias Gerstenberg, and Fiery A Cushman. Causal judgments approximate the effectiveness of future interventions, Apr 2018. URL psyarxiv.com/ nq53z.
Suraj Nair, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Causal induction from visual observations for goal directed tasks, 2019.
OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca. Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by sele-supervised prediction, 2017.
J Pearl, M Glymour, and N P Jewell. Causal Inference in Statistics: A Primer. Wiley, 2016. ISBN 9781119186847.
Judca Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning, 2021. URL /ttps://arxiv..orgg targ/abbs2100 11107.
Satinder Singh, Andrew G Barto, and Nuttapong Chentancz. Intrinsically Motivated Reinforcement Learning. IEEE Transactions on Autonomous Mental Development, 2(2):70-82, 2005. ISSN 19430604. doi: 10.1109/ TAMD.2010.20510311
13

---

Published in Transactions on Machine Learning Research (07/2022)
Yuhang Song, Jianyi Wang, Thomas Lukasiewicz, Zhenghua Xu, Shangtong Zhang, Andrzej Wojeicki, and Mai Xu. Mega-reward: Achieving human-level play without extrinsic rewards, 2019.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning An Introduction. 2018. ISBN
9780262039246.
Richard S. Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M. Pilarski, Adam White, and Doina Precup. Horde: a scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In In Proceedings of 10th International Conference on Autonomous Agents and Mulliagend Systems (AAMAS), May 2- 6, 2011.
Emanuel Todorov. Efficient computation of optimal actions. Proceedings of the National Academy of Sciences, 106(28):11478- -1488, 2009. doi: 10.1073 (0003//pas.000 ppas.07107431000 URL https: / /wwww pnas org dos//////bb/.... 1078/ppas.07107481065 0710743106.
Oriol Vinyals, Igor Babuschkin, Wojciech Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575, 2019. doi:
10.1038/<41156-011 -12------
Ziyu Wang, Nando de Freitas, and Mare Lanctot. Dueling network architectures for deep reinforcement learning. CoRR, abs/1511.06581, 2015.
Michael L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021, 2021. doi: 10.21105, 10.21105/Joss.00000 03021. URL attps://doi http:://doo.ogg/....ee 03021.
14

---

Published in Transactions on Machine Learning Research (07 (07/202) 2022)
Clusters (ours)
Inverse
5
CEN cours)
3-
Reward
2-
Inverse
2.0395) 3,dted
(0 scle
1
2
3
(a)
Environment Steps (1e6)
(b)
Figure 8: a) State visitation maps at different points of training of the Empty environment. CEN values different locations similarly, and consequently, the agent learns to explore states more uniformly. The inverse model encourages going walls where predicting the action is hard. b) CEN promotes the movement of boxes and consequently faster learning
A
Appendix
Additional experiments
B.1 CEN as intrinsic motivator
We have shown that CEN can learn controlled effects in an unsupervised manner. Here we showcase the use of this ability as an intrinsic motivator of a reinforcement learning agent. We consider two tasks, an empty environment without any extrinsic reward where the agent can only control itself and Clusters. The RL agent is implemented using PPO (vanilla). Additionally, PPO is augmented with the exploration bonus proposed in Never Give Up (NGU Inverse) (Badia et al., 2020b). NGU is composed of two modules for computing episodic and life-long rewards. For simplicity the following experiments only the episodic module which in NGU consists of a count-based method using an episodic memory to approximate the number of times agent visited each state, and an inverse model to identify controlled states. We replace the inverse model with CEN (NGU- + CEN) and use the latent representation of the controlled branch to compute NGU's episodic reward. In this experiments, CEN is trained along the PPO policy using experiences collected with the same policy.
Empty environment: the goal of this environment to showcase how each NGU variant rewards controlled events. In this environment, only the agent's movement is controlled; thus, rewarding for controlling the agent's 's location should promote uniform exploration of the enviromment since once location is visited, it should not be rewarded as much as visiting new location. We hypothesize that an inverse model will create similar representations for the same action disregarding the location where it was taken. This should impair exploration since the reward will be similar regardless of where the action is performed.
Fig. 8a shows that agent trained with the inverse model hogs walls and corners. This bocause is hard to predict the action near unmovable objects, leading higher value. Contrarily, CEN promotes different. locations more uniformly and the agent learns to explore better the environment.
Clusters environment: A more challenging environment is Clusters, where the agent needs to move colored boxes to their respective colored blocks. This environment provides a reward at the end of the episode corresponding to the total mumber of boxes correctly placed. Results are provided in Fig. 8b. Due to the sparsity of the reward PPO does learn a correct bchavior the given time. Similarly, NGU + Inverse learns to place one box but fails to learn general behavior to solve the task. Conversely, NGU CEN quickly learns to move boxes leading to high extrinsic reward.
15

---

Published in Transactions on Machine Learning Research (07 (07/2022)
B.2 Additional baselines for attribute prediction
The following table an extension of Tab. with additional simpler baselines. The No mask baseline does not filter any pixel from the image, Le., the mask is all ones. We would expect close to perfect prediction of the attributes since all the information is in the input of the probing network, our results confirm that this is the case.
The CEN (rand.) and Normal (rand.) baselines are untrained, randomly initialized, CEN networks where one provides the controlled hidden representation and the other the normal hidden representation to the probing network. This experiment aims isolate the benefits of the architectural biases introduced by CEN.
Interestingly, when there is no uncontrolled effects (Clusters and Lights), the controlled branch provides good representation. Even with random weights the network is able to encode the action in the representation, making it casier for the linear probe to predict the agent attribute. Action and agent are correlated making the task of the linear probe easier. If the relation between object and action is more complex (e.g. boxes, lights or buttons), the linear probe has more difficulties. This indicates that the architectural biases are enough for simple relations but the more complex the relation the more learning is needed. When there are uncontrolled effects (Spiders and MZR) the representation is entangled and the performance of the lincar probe is low.
F1 PIXELS
F1 LATENT
ENVIRONMENT
ATTRIBUTE
CEN (OURS)
No MASK
CEN (OURS)
CEN (RAND.)
NORMAL (RAND.)
AGENT
1.040.00 0.35:10.03
1.0:0.000
0.9710.01
0.56+00.06
0.16=0.01
SPIDERS
SPIDER
1.0:0.00
0.4110.01
0.2911.01
0.17-0.01
AGENT
0.7640.41
1.0+0.00
0.97+0.01
0.95+0.01
0.17=0.01
CLUSTERS
Box
0.7810.37
1.0.0.000
0.9510.02
0.8410..4
0.20
AGENT
0.97+0.01
1.0+0.00
1.0+0.011
0.95+0.01
0.18=0.01
LIGHTS
BUTTON
0.93+0.05
1.0+0.00
0.99+00.0
0.59+0.04
0.49=0.14
LIGHT
0.93+0.04
1.010.00
1.040.0
0.5610.06
0.47=0.05
ACENT SKULL J
0.66+00.88
0.97+0000 1.0+0.01
0.91 0.002
0.62++0.02 0.48+
0.17=0.01 0.36=0.00
MZR
0.1940.03
0.61#0.03
Table 2: F1 score for CEN and additional baselines when predicting attributes from pixel latent space.
B.3 Masks
Here we provide masks for both CEN and ADM. Mask are based on effects and extend over two frames, for visualization purposes here we only show the next observation with the full mask. can be seen that CEN fails for few pixels in MZR, which may explain the low F1 score for empty GT.
16

---

Published in Transactions on Machine Learning Research (07 (0222)
GT
Controlled (ours)
Normal (ours)
ADM
F1: 0.60
F1: 0.34
GT
Controlled (ours)
Normal (ours)
ADM
F1: 0.49
F1: 0.35
J A.S.R.
J.S.A.
GT
Controlled (ours)
Normal (ours)
ADM
0.00
GT
Controlled (ours)
Normal (ours)
ADM
F1: 1.00
F1: 0.00
Figure 9: Examples of success and failure cases for CEN and ADM MZR.
17

---

Published in Transactions on Machine Learning Research (07 (022)
Controlled (ours)
Normal (ours)
ADM
F1:0.99
0.30
GT
Controlled (ours)
Normal (ours)
ADM
F1: 0.88
F1: 0.25
Figure 10: Examples of masks for Clusters.
18

---

Published in Transactions on Machine Learning Research (07 (022))
Controlled (ours)
Normal (ours)
ADM
GT
Controlled (ours)
Normal (ours)
ADM
GT
Controlled (ours)
Normal (ours)
ADM
Figure 11: Examples of masks for Spiders.
19

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Controlled (ours)
Normal (ours)
ADM
F1: 0.69
0.26
GT
Controlled (ours)
Normal (ours)
ADM
Figure 12: Examples of masks for Lights.
C Relation of Blame to the advantage function
A typical use of the formulation introduced in section 2 is compute the causal effect of action on the return G relative to policy
(s) G (s) Bc(s) ((s - V(s)
(6)
A(s,a).
(s(s) is the return the agent would get if action a were to be taken at state s and is typically estimated using a state-action value function Q(s, a). The choice of normality for fa(s) is to estimate the expected return with the state-value function V(s), giving us the advantage function A(s, a). Inspired by Generalized Value Functions (Sutton et al., 2011; Sutton Barto, 2018) which aim to integrate general knowledge of the world; Blame takes more general approach than the advantage function by leaving return as special case.
Environments
D.1 Griddly
Griddly (Bamford et al., 2020) is a highly optimized grid-world based suite of environments, Enviromments used this work based on Griddly generate 64 x 64 pixel observations, although the size of the grid-world may vary. Griddly supports multiple rendering formats, this work the 2D rendering of sprites.
Spiders: is 6x6 arena where Gnome (the agent) has to grab Gem without being killed by Spider. The agent dies if it collides with the spider. In this environment, the agent can move left, right, up, down or slay. The spider takes an action randomly from the following: rotate left, rotate right or move forward. This environment's controlled entities are: Agent.
Clusters: 13x10 arena where a Knight has move boxes of the same color their corre- sponding colored-block without touching the spikes. There two different colors, blue and red. The
20

---

Published in Transactions on Machine Learning Research (07 (07/2022)
agent is rewarded with +1 whenever a box is pushed towards similar colored block. The agent dies if it collides with spikes or if a box is destroyed by spikes. The agent can move lest, right, up, down stay. This environment's 's controlled entities are: Agent and Boxes.
For the RL experiments, we made the environment more sparse by removing all intermediate rewards and only rewarded the agent after all the boxes of the same color are pushed blocks. Since the agent can not get any reward whenever a box is stuck to wall. We removed boxes that touches the wall and punished the agent with -0.01. We then scaled the reward of solving color with the number of boxes pushed to the block. This modifications encourage the agent to solve the environment by pushing the maximum number of boxes into blocks while preventing it from getting deprived from reward by accidentally pushing boxes to walls.
Lights: is 11x8 arena where a Ghost (the agent) has to turn all the lights on by pressing each button. Buttons and lights are colored either blue or green. Pressing a button of one color turns the light of the same color on. The agent can move left, right, up, down or stay. This environment's controlled entities are: Agent, Buttons and Lights.
Empty: this environment is copy of the clusters environment where all the boxes, blocks, spikes and rewards were removed.
D.2 Atari Montezuma's Revenge
The ALE (Bellemare et al., 2012a) provides access to Atari 2600 games to learning methods like RL. As it has been popular choice by methods using inverse models, in this work we use the game Montezuma' 's Revenge to evaluate CEN. This environment provides uncontrolled as well as controlled effects with more complex entities. The environment typically generates observations of 210x160 pixels which we downscale to 64x64 pixels. Additionally the action space is of size 10.
E
Training
E.1
Architecture
Encoder: is composed of two 2D convolutional layers with 4x4 kernels, stride 2 and padding of 1. Additionally, we have residual blocks each with two 2D convolutional layers with stride 1 and padding of 1. The first layer has a kernel of 3x3 and the second layer of Ixl. ReLU is used as activation function throughout the network; BatchNorm is used between each layer; and 64 channels every convolutional layer. We project the resulting maps into flatten vector of size 32 using a linear layer with ReLU activation function.
Decoder: this module is composed of six 2D transposed convolutional layers all having 4x4 ker- nels, stride and padding of 1. Each layer uses ReLU as activation function but the output layer which uses Tanh activation. Every layer uses 64 channels with the exception of the last layer which outputs a channel prediction of the perceived effects. Parameters are shared among the controlled and normal branch decoders.
Controlled and normal modules: both modules composed of three linear layers with 32 hidden units, each with ReLU as activation function. The input to the controlled branch are the encoded observation and an embedding of size of the chosen action.
PPO Agent: uses an encoder consisting of 3 convolutional layers with (channels, padding, strides) equal (32, 8, 4), (64, 4, 2), (64, 3, 1) respectively. The encoder is followed by two lincar layers of sizes 512 and number of actions respectively, to transform the feature map to the environment's 's number of actions.
21

---

Published in Transactions on Machine Learning Research (07 (022))
E.2 Hyperparameters
Name
Value
Sweep 16, 32, 64, 128)
hidden size
32
latent size
128
[16, 32, 64, 128, 256]
channels
64
[16, 32, 64, 128)
learning rate
0.0001
[0.0001, 0.0005, 0.001, 0.005]
0.01
[0.001, 0.01, 0.1, 1, 10, 20, 30, 50]
T
0.01
Table 3: CEN hyperparameter sweeps and final values used.
Name
Value
Sweep [0.01, 0.05, 0.1, 0.5, 1, 5]
entropy
0.05
hidden size
64
[16, 32, 64, 128)
attention size
128
[32, 64, 128, 256]
learning rate
0.0001
[0.0001, 0.0005, 0.001, 0.005]
T
0.01
Table 4: ADM hyperparameter sweeps and final values used.
Name
Value
Sweep
encoder channels
32
[32, 64]
encoder hidden
[16, 32, 64, 128)
latent size
128
[64, 128, 256]
learning rate
0.0001
[0.0001, 0.0005, 0.001, 0.005]
Table 5: Inverse model hyperparameter sweeps and final values used.
Name
Value
Sweep
batch size
512
[64, 128, 512, 1024
latent size
32
[8, 16, 32]
CEN encoder output size
128
[16, 32, 64, 128)
learning rate
0.0005
[0.00005, 0.0001, 0.0002, 0.0005, 0.001] [0.0001, 0.001, 0.002, 0.005, 0.01, 0.1]
IR Beta
0.001
Rollout size
2048
[1024, 2048, 4096)
discount
0.95
epochs
10
Table 6: PPO hyperparamcter sweeps and final values used.
22


Reviews:
[{"review_id": "QNQIS4RPRA", "reviewer": "Reviewer_b3Ck", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nOverall, the work is very easy to follow. Particularly in the context of work at the intersection of psychology, causality, and reinforcement learning, which tends to yield either impenetrable jargon or hand-waving with respect to at least of one of the fields involved.\n\nThe method is simple, novel, well-motivated, and supported by a reasonable set of experimental results.\n\nI know the limitations and requested changes are longer than this strengths section, but I promise you that these strengths outweigh the rest :)\n\nLimitations:\nMost of the obvious limitations are well-addressed by the paper. Using a random policy for \"normality\" feels particularly limiting in the context of large/high-dimensional action spaces (e.g. humanoid or robotic hand), but this is fine as the first step in a novel direction of research.\n\nThat said, I was quite surprised the expectation in Eq 5 wasn't discussed more. This expectation over observations could easily backfire even in very simple environments. For example if action_0 shifts a certain observation dimension by -1 and action_1 shifts the same dimension by +1, then the \"normal\" stream will predict no change for that dimension, even though change always occurs. Dealing with the full distribution of observations under the \"normal\" policy seems more ideal. Something like the KL between e_p^a and E_\\hat{a} e_p^\\hat{a} or some other measure between distributions (e.g. Wasserstein). Obviously, this would be much harder implement, but talking about the ideal case and then showing that the expectation over observations is \"good enough\" would be sufficient -- this limitation should be mentioned explicitly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a novel representation learning scheme inspired by psychological theories of causal reasoning in humans. Specifically, they assess causal influence by comparing potential interventions to the \"normal\" dynamics. Dimensions of the observation space (pixels for all experiments) that differ are considered to be controlled by the agent.\n\nTo evaluate this notion of causal influence, a forward model is learned with a unique architecture that forces separate interventional and \"normal\" streams whose predictions are subsequently dimension-wise summed to produce the final step-observation prediction. An auxiliary loss is used to prevent the normal stream from degenerating: its log likelihood is maximized independent of the interventional streams contribution. This modeling approach is called the Controlled Effect Network (CEN) and its efficacy in determining which pixels are controlled is evaluated experimentally.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1) A good deal of the motivation for this work appears to stem from recent advances in exploration. While CEN does do similar things to methods that have been shown to aid in exploration, no direct evidence of CEN's exploratory utility is given in this paper. Either some experimental results along these lines should be attempted, or this connection should be limited and CEN's utility for exploration explicitly noted as an open question.\n\n2) The probe experiments shown in Table 1 are interesting, but two additional baselines would better help me contextualize CEN's performance. If I understand correctly, Probe accuracy from pixels should be highest (for both controlled and uncontrolled attributes) for unmasked images. Could you report these numbers? This would confirm the relative predictability of each attribute; I worry the uncontrolled attributes might just be harder to predict regardless of the mask. Similarly, probe accuracy from CEN's latent representation prior to training (i.e. random weights) would shed light on the contribution of inductive biases relative to objective functions.\n\n3) A few connections to related work are probably worth including (Specific papers are merely illustrative, feel free to cite different papers if you feel they make more sense in addressing the following points):\n* The notion of \"normality\" feels quite similar to that of \"passive dynamics\" from control theory. I couldn't track down the original citation for this, but \"Efficient computation of optimal actions\" by Emo Todorov is fairly representative of its usage in the field.\n* The discussion around the limitations of inverse models is solely focused on single-step inverse models. But many recent works explicitly deal with multi-step predictability in either the form of action sequences (e.g. Mohamed & Rezende, 2015 -- already cited for a different reason) or options (e.g. \"Variational Intrinsic Control\"). It's probably out-of-scope to compare to such methods since it's unclear how best to adapt them to produce segmentation masks, but their existence should mentioned and subsequently hedge some of your conclusions about the superiority of CEN to inverse modeling approaches.\n\n4) The relationship between the RL notion of Advantage and your notion of \"Blame\" is interesting if this hasn't been previously published elsewhere (its new to me at least), but it doesn't seem particularly relevant to the rest of the paper. Particularly since \"Blame\" is only calculated locally, rather than being the expected value of some future quantity, no reinforcement learning is going on, etc. Assuming this is novel, then I believe it'd be better served in the appendix (perhaps a \"relationship between blame and rl\" section?). All of this goes double for the GVF reference -- its not transparent how Eq 5 relates GVFs, but its unclear how relevant it'd be to the rest of the paper even if it were. So I'd suggest unpacking the relationship in the appendix. (Could also just cut out these bits altogether, but it is an interesting connection.)\n\n5) As mentioned in limitations, using an expectation over pixel values for the \"normal\" stream feels very limiting, please explicitly address or acknowledge this.", "broader_impact_concerns": "No broader impact statement given, but I don't think one is needed here, as all possible concerns would be shared with the whole subfield of Deep RL.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2BV2G6hj10", "reviewer": "Reviewer_4dHp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n### Strengths\n- Clear acknowledgment of limitations\n- Simple idea leading to empirical improvements in the results provided\n\n### Weaknesses\n- Limited set of baselines to accurately test the core hypotheses.\n- Unclear where this fits into an RL agent. The paper lightly suggests the disentangling these effects could be useful to RL agents, but conceptually where this sits in an agent isn't clear to me given the text of the paper.\n- There are parts of the paper that seem unnecessary given the core proposal/hypotheses (see below in organization).\n- Much of the motivation seems centered on the fact that this is \"more human like\". I find this kind of motivation a bit hard to digest given our current understanding in the cognitive sciences. I think the core concept of blame the paper proposes can stand independently from this line of motivation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a causal interpretation for learning affordances. Specifically, the paper proposes a notion of blame, in contrast to individual causal effects (ICE), which measures the difference between what the agent observes versus what the agent expects to happen when no intervention (or action) is taken. Individual causal effects would instead measure the counter-factual difference/effect of taking a different intervention, and really finding if there exists an intervention that would lead to a different outcome. \n\nThe paper then applies the definition of blame to a self-supervised network for predicting the effect of an action. The Controlled Effects Network (CEN) tries to disentangle the controlled effects and uncontrolled effects to predict the outcome of the action (i.e. intervention) made by the agent. This is done through the loss function in equation 6 and a two stream architecture where the controlled effects and uncontrolled effects are modeled through separate network streams with a shared base encoder.\n\nThe main questions the authors want to address are:\n- Can CEN identify controlled effects at the pixel-level?\n- Are there applications which don't require pixel-level precision and can CEN predict controlled effects at the attribute level.\n\nWhile not explicitly stated as a question for empirical study, another claim made in the methods section is: \"Since this branch [normal branch] cannot predict everything without the action the model will converge to the expected effect due to the MSE loss.\"\n\nThe paper uses Attentive Dynamics Model (ADM) [Choi et al, 2019] as a baseline for comparisons when answering these questions. Overall, the empirical results confirm that CEN is able to better predict the effects of an action on the environment. The paper also provides some evidence that the two streams are decoupling the controllable and uncontrollable effects.\n\n### Main Contributions\n- Connection between blame (as defined by [Halpern, 2016]) and affordances for use in a reinforcement learning/self-supervised learning setting.\n- Introduction of Controlled Effects Network (CEN) which is a two stream architecture to predict the effect of an action on the environment, and a novel objective inspired by the previous contribution.\n- Empirical experiments focusing on CEN's ability to predict effects in the environment in several domains compared with ADM.\n- A discussion about limitations of the approach leading to the potential for future work.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall, I don't think this paper is quite ready for publication. I'm going to break my suggestions into \n- Writing/Organization: different ways of organizing the paper to make the concepts clearer, suggestions on how to better present some of the background\n- Method: thinking about the core proposed architecture\n- Empirical: some suggestions on better ways to test the main hypotheses, and some suggestions on how to make this paper more relevant to the RL literature.\n\n\n## Writing/Organization\n\nThese are all suggestions aimed at making the paper clearer and focusing in on the key concepts presented.\n\nWO-1. The paper focuses a lot on the difference between ICE and blame. I'm wondering if section 2 focused more on blame and presented ICE as some auxiliary. I think it is reasonable to motivate moving away from ICE, but I think spending so much time on the idea is drawing focus and attention away from the actual proposal. \n\nWO-2. I'm confused why you define the advantage function in terms of blame as this isn't used anywhere else in the paper.\n\nWO-3. You should cite [Horde: A Scalable Real-time Architecture for Learning Knowledge from Unsupervised Sensorimotor Interaction][https://aamas.csc.liv.ac.uk/Proceedings/aamas2011/papers/A6_R70.pdf] when discussing GVFs, not the RL textbook.\n\nWO-4. I'm wondering what the authors desired core proposal is. The initial sections read more like a larger discussion on using blame as a core concept in reinforcement learning that can be used to learn affordances but also explain other RL concepts. The rest of the paper is an architecture proposal, and doesn't really have much connection to RL beyond needing agent's which take actions. I think if the authors can clarify this (i.e. focus on the architecture proposal) it would clean up a lot of the unneeded text and make the paper much easier to read/follow.\n\nWO-5. I think the idea of blame, or the difference between what we expect and what is observed, is a useful concept for RL, but I feel the paper doesn't do a good job bridging between the two concepts well. I think if the paper instead proposed ICE and blame using more RL notation/language the idea would be more digestible for an RL audience (which I interpret as the main audience of this paper).\n\nWO-6. The empirical section would be more complete if details on the ground truth used was included. You have space.\n\nWO-7. I would like more context about the experiment in the figure captions. Also I think the plots axes should be labeled (currently there are very few labels for any of the axes)\n\n## Questions about Methods/Concepts\nThis section is mostly questions/clarifications on some of the concepts.\n\nQ-1. What is the connection between equation 5 (i.e. blame) and general value functions? If there is a connection the paper should make this clearer.\n\nQ-2. Equation 3 suggests the notion of blame used in this paper is equivalent to ICE defined in equation 1. Is this actually the case? This might also be because the definition of ICE in equation 1 is not clear. My interpretation is that it returns a boolean on whether there exists an intervention $\\tilde{x}$ which changes the observed outcome for individual $Y_i$. This seems quite different from measuring the difference between what we expect to happen to individual $Y$ (i.e. normality through $\\beta_Y$) and the observed outcome over intervention $x$.\n\nQ-3. I'm not sure I understand the connection to the dueling networks architecture. While there are two stream in that architecture the goal of the two streams in Wang et al's architecture is to estimate the value function and the advantage function separately. Your architecture separates between controllable and uncontrollable dynamics in the environment. So maybe you meant just in the sense that there are \"two streams\" in the architecture?\n\nQ-4. How would this be applied in a reinforcement learning agent? Would you use the learned representations to learn policies, would you use the bit masks to drive attention in the agent? I think the paper leaves this unclear.\n\n## Empirical Section\n\n### E-1\nI think the core questions addressed by the authors in their empirical exploration is mismatched from the claims made throughout the paper about their approach. While I think there are several questions that need addressed in a better way, the main missing claim the authors seem to miss in their testing is the claim made in section 3: \n\n    Since the this branch [normal branch] cannon predict everything without the action the model will converge to the expected effect due to the MSE loss.\n    \nI think the best way to test this question is the inclusion of more baselines. I suggest two new architectures to explore this claim:\n1. Same objective function, but combine the streams into a single stream with only two linear heads on the output to learn $e^a_c$ and $e^n$.\n2. Same objective function and architecture, except include the action into the normal stream as well as the controlled stream.\n\nThese two architectures should address the thrust of this claim adequately. The first gives direct insight into whether the separation of streams improves performance and the second tries to get at the responsibility of the action on the possible parts of the dynamics modeled by each stream. \n\nIn the end, it would be even better to directly measure what parts of the dynamics are being modeled by the two streams. I would make a suggestion here, but I think the current measurement is too unclear for me to make a reasonable suggestion in the current framing of the paper. You begin to evaluate this on Montezuma's Revenge by showing the masks for the different streams, but I would like to see this go further.\n\nI also think the mixture enabled by the hyperparameter $\\alpha$ should be added to the main work. Specifically, the values between [0, 1]. Another interesting direction here is to have the two parts of the objective be a true mixture (i.e. $L = (1 - \\alpha) MSE(e_c^a + e_n, e^a_p) + \\alpha MSE(e_n, e_p^a)$). This would enable a better understanding of what each part contributes to the overall performance of the proposed architecture.\n\n### E-2\n\nI don't quite understand how the bitmasks are constructed for both CEN and ADM. I think this procedure and how the threshold is chosen should be expanded in the main paper. I also found the explanations in the appendix to be unsatisfactory.\n\n\n### E-3\n\nWhile I'm not sure it is required for publication (I think the above issues are already quite a bit), applying your approach in some of the same experiments proposed by ADM would make the comparison more interesting generally. It would also help narrow in on the implied hypothesis of this paper (more explicitly stated in the ADM paper): \"better separation of controllable and uncontrollable effects in the agent's dynamics learning will lead to better behavior\". This paper and the ADM paper share this hypothesis and if you can better show your method does indeed separate the effects and if the final agent performs better on the given tasks this can be evidence in favor of the question. Again, I'm not sure it is needed for publication at TMLR, but would make the paper significantly stronger.\n\n\n## Missing citations/connections\n\nWhile I don't give a full account of missing works, I'll give some lines of research I think is relevant to the work presented in this paper.\n\n- Affordances: https://proceedings.mlr.press/v119/khetarpal20a.html, https://journals.sagepub.com/doi/abs/10.1177/1059712321999421?casa_token=fFj0p4elcUQAAAAA:nyXqZgmomqT68jPdlauSIKgOkJyqBKZytGwFIf0Tkxw_BaRxgCu3q70MLuhc6Xp7MJJOeoV1ot8V3g\n- Model-based reinforcement learning: Generally, I think this idea is related to model based reinforcement learning (as you are trying to model the effect of an action on the environment). While this may be applied in other aspects of the agent, generally this kind of modeling is discussed quite a bit in MBRL. https://proceedings.neurips.cc/paper/2015/hash/6ba3af5d7b2790e73f0de32e5c8c1798-Abstract.html, https://arxiv.org/abs/2006.16712, etc...\n- General Value Functions (if applicable): https://aamas.csc.liv.ac.uk/Proceedings/aamas2011/papers/A6_R70.pdf, https://journals.sagepub.com/doi/abs/10.1177/1059712313511648?casa_token=ttGynWjW6OwAAAAA:t1XDAZQv9K2Lm7UPrAaJW5rYK3SgLMvJEXMM0x71DRvHL5KZLmQdKunSfO83x0j049V9hXsj5ix8qQ, etc...\n\n## Minor Suggestions/edits:\n- Section 2.2: \"actual causality\" -> You should cite where you are getting the definition of normality from.\n- What is the p in e^a_p in equation 2? If it stands for \"perceived\" is it really needed in this context as we aren't discussing the actual underlying effect.\n- You mention three main questions at the beginning of section 4, but only list one question and then conjecture about what might be needed in terms of precision.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fbVfVDCckH", "reviewer": "Reviewer_utPk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ This paper contributes to a very interesting research question which remains fairly open, introducing a novel way to be more explicit about causal effects in deep RL architectures.\n+ The authors draw inspiration from the cognitive science literature, which brings an apparently important point: humans attribute blame in a way that is not just due to the Pearl-style direct causal effect magnitude but also depends on (and contrasts with) what may be considered a normal outcome.\n+ The experiments seem to confirm that the proposed approach can do a good job of attributing effects, either in pixel space or in the space of hand-crafted attributes that are known to be relevant to the task, using specially designed learned probes.\n\nWeaknesses:\n\n- The notion of normality is too vague, both in its textual explanations and in the mathematical formalization: I did not understand Eq. 5 clearly enough (what are the different symbols and letters and the expectation standing for exactly?)\n- The notion of normality seems to depend heavily on the current policy (which is ok, but should be stated more clearly)\n- The statement \"Eq. 5 would indicate that no changes to the health bar are normal\" on p.3 needs some clarification. The reason (I think) for no change in health being the \"normal\" outcome is not that one normally does not die from bombs, but rather that the current policy understands the effect of bombs and how to avoid them or not, and thus the learned model should know that the policy will escape the bomb (since it is \"easy\" to do) and thus the expected effect would be no change in health.\n- Adding the normal and controlled branch effects in pixel space does not make sense in general: the addition should be in some appropriate abstract space. Why not add them up in h-space and then apply the decoder?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors take a causal and counterfactual view on attributing effects to an action for an RL agent.\n\nThey introduce a specific neural architecture (the Controlled Effect Networks) for predicting effects (as in model-based RL) with one branch that has access to the action and one that doesn't, as well as an objective function such that the uninformed branch (which they call the normal branch) tries to explain as much as it can, in order to leave the informed branch (which they call the controlled branch) focus on the action-specific changes in observations.\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\nI believe all of these points can and should be addressed, after which I would probably be satisfied with seeing this paper accepted.\nRegarding the last of the above weak points, the issue should at least be discussed, but the paper would be greatly improved if the authors actually ran the modified architecture I am suggesting (which may not make any difference in simple worlds where the effects can be attributed to individual pixels).\n\n- address all the \"-\" weak points in the above list\n- the idea of using linear probes to analyze the internal behavior of deep nets was proposed by Guillaume Alain (2016) \"Understanding intermediate layers using linear classifier probes\", which should be cited.\n- in the list of deep RL + causality papers, I would add Scholkopf et al 2021 \"Towards causal representation learning\" as well as Ke et al 2021 \"Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning\"\n- typo: is to us to define --> is for us to define\n- typo: to analyzed --> to analyze\n", "broader_impact_concerns": "None I can foresee besides the general danger of generic advances in AI.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)