Did I do that? Blame as a means to identify controlled effects in reinforcement learning
Authors: Oriol Corcoll Andreu
Youssef Sherif Mansour Mohamed
Raul Vicente
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
unsupervised reinforcement learning

Abstract: Identifying controllable aspects of the environment has proven to be an extraordinary intrinsic motivator to reinforcement learning agents. Despite repeatedly achieving State-of-the-Art results, this approach has only been studied as a proxy to a reward-based task and has not yet been evaluated on its own. We show that solutions relying on action-prediction fail to model critical controlled events. Humans, on the other hand, assign blame to their actions to decide what they controlled. This work proposes Controlled Effect Network (CEN), an unsupervised method based on counterfactual measures of blame to identify effects on the environment controlled by the agent. CEN is evaluated in a wide range of environments showing that it can accurately identify controlled effects. Moreover, we demonstrate CEN's capabilities as intrinsic motivator by integrating it in the state-of-the-art exploration method, achieving substantially better performance than action-prediction models.

Under review as a conference paper at ICLR 2022
DID I DO THAT? BLAME AS A MEANS TO IDENTIFY CONTROLLED EFFECTS IN REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Identifying controllable aspects of the environment has proven to be extraor- dinary intrinsic motivator to reinforcement learning agents. Despite repeatedly achieving State-of-the-Art results, this approach has only been studied as a proxy to a reward-basse task and has not yet been evaluated on its own. Current methods are based on action-pre prtioonprediction. Humans. on the other hand, assign blame to their actions to decide what they controlled. This work proposes Controlled Effect Net- work (CEN), an unsupervised method based on counterfactual measures of blame to identify effects on the environment controlled by the agent. CEN is evaluated in a wide range of environments showing that it can accurately identifyy controlled effects. Moreover, we demonstrate CEN's capabilities as intrinsic motivator by integrating it in the state-of -fee--ff-heee-ee the-art exploration method, achieving substantially better performance than action- cittoonpreeittion models.
INTRODUCTION
The recent success of reinforcement learning (RI.) methods in difficult environments such as Hide & Seek (Baker et al., 2019). StarCraft II (Vinyals al., 2019), or Dota2 (OpenAI et al., 2019) has shown the potential of RL to learn complex behavior. Unfortunately, these methods also show RL's inefficiency to learn (Espeholt al., 2018; Kapturowski et al., 2019; Gulcehre al., 2020), requiring a vast amount of interactions with the environment before meaningful learning occurs. Consequently, environments with sparse rewards are known to be extremely difficult making imperative a good exploration strategy. A popular approach to exploration is to introduce behavioral biases in the form of intrinsic motivators (Chentanez et al., 2005; Mohamed Rezende, 2015). This technique aims to facilitate the learning of task-agnostic behavior by producing dense rewards, driving the agent to discover novel states and by doing so increase the chance of discovering the environment's reward.
Numerous motivators have been developed taking inspiration from humans, e.g. curiosity control (Bellemare et al., 2012b; Pathak et al., 2017; Burda et al., 2018; Choi et al., 2019; Badia et al., 2020b). Recent work (Choi et al., 2019; Song et al., 2019; Badia et al., 2020a:b) has achieved State-of-the-AAe on the Atari benchmark (Bellemare et al., 201 2a) by rewarding agents for the discovery of novel ways to control their environment. A common design principle among these methods is the use of an inverse model to predict the chosen action from two consecutive observations. The hope is that the latent representation learned encloses aspects of the environment controlled by the agent.
A more causal approach is to compare counterfactual worlds (Pearl, 2009), i.e., an effect is controllable if the effect would have been different had the agent taken another action. A caveat of this approach is that things become trivially controllable. Fig. 1 (left) shows an scenario where an agent moves a box by moving left. Here box becomes controllable even when the agent performs the action "do-nothing" since there is an action, "move-left", that would move the box. Contrarily, it is believed that humans identify controlled effects by assigning degree of blame to their actions. particular, humans compare what happened to normative world (Halpern, 2016; Morris al., 2018; Langenhoff' et al., 2019; Grinfeld et al., 2020). If what happened normal, humans would not assign blame to their actions, e.g. when performing "do-nothing", the box's 's effect would not be controlled since normally the box would not move. However, it would be considered controlled when performing "move-left" since its normative state is to not move.
This work proposes Controlled Effects Network (CEN), a completely unsupervised approach to identify controlled aspects of the environment based the human notion of blame. In contrast to

---

Under review as a conference paper at ICLR 2022
3 a
Health
2
uncontrolled
controlled
Left) Blame compares an imagined normative world against reality to attribute the movement of the agent and the box to the action. Middle) Using do-nothing action as normative world is not enough since do-nothing has an effect. Right) Causal graph of a typical RL setting.
Figure 1:
models based on action prediction, CEN is composed of a two-branch forward model that creates a normal and controlled view of the world. Our experiments show that CEN can disentangle effects precisely, outperforming state-of-the-art approaches to detect controlled effects. Additionally, we evaluate CEN as an intrinsic motivator by replacing the action-prediction model in Never Give Up's (Badia et al., 2020b) episodic reward with CEN, leading to substantial gain in performance.
IDENTIFYING CONTROLLED EFFECTS USING BLAME
Our goal is to identify changes in the environment that were controlled by the agent. This section introduces Individual Causal Effecc (ICE), fundamental measure in causal literature, and frames it in the context of RL. We show how this measure can be used to identify controllable effects but argue that these not suitable for RL. In contrast, the human perception of causality is associated with the concept of blame (Gerstenberg & Lagnado, 2014). For example, if lightning hits forest tree and starts a fire, humans would point to the lightning the cause of the fire, not the oxygen wood since they are normally present in the forest. Consequently, we expand the idea of blame to identify controlled effects by using measures of normality and counterfactuals.
2.1 CONTROLLABLE EFFECTS
What does it mean to cause something? Pearl et al. (2016) provide an intuitive definition of cause- effect relations: "A variable X is cause of variable Y if Y, in any way, relies on X for its value". This kind of formulation tries to answer questions like "does smoking causes cancer?". Actual causality, proposed in Halpern (2016), studies causal relations between individual events of X and Y. It aims to answer questions like, "did smoking for 30 years caused David's cancer?". In the following, we introduce the concept of causal effect to then define controllable effect in the context of RL.
The individual causal effect (ICE) of an event X = x on a variable Y, can be measured by comparing counterfactual worlds
ICE; YY,
where Y: reads as "what would the value of an individual Y, be if X is forced to be x". Similarly, Y, describes the value of Y; when X is forced not be x. Note that the sub-index i refers to an individual, and hence in the following, we use Y and Y interchangeably.
The fundamental problem of causal inference states that we can only observe one of these counter- factual worlds and the other needs to be imagined. Intuitively, Eq. 1 compares the world where the event x happened to an alternative world where event x had not happened. Consequently, we say that & has a causal effect on Y if there is an ã E X that satisfies Eq. 1. In the context of RL, X and Y take the form of actions, states and observations. Fig. (right) illustrates the causal relations present in typical RL. setting, where a state s has an effect on both the next state s' and the produced observation o which, in turn, has an effect on the agent's 's choice of action a E A. Similarly, an action has an effect the next state. Since states are typically not accessible by the agent, do not states as variables; nevertheless the same principle can be applied if these are accessible. We define the perceived effect ef the difference between consecutive observations when taking action a, i.e. eg o' As in Eq. 1, say that perceived effect was controllable by the agent's action when
3ā A: cp cg
2

---

Under review as a conference paper at ICLR 2022
Since we want to know what elements of the perceived effect are controllable, the inequality is an element- operation. It is important to notice that Eq. 2 has far-reaching consequences, for example, an agent next a box would have a causal effect on it even when not moved since there is counterfactual world where that box would have moved. Using Eq. as reward, the agent would be rewarded for almost every action at every state! Note that taking would not work since even doing nothing does something, c.g., Fig. 1 (middle) shows a scenario where doing nothing has an effect on the agent's 's health. Taking do-nothing as would not attribute the effect to the agent. Instead, we would want a more human-likk definition of what is controlled where an agent controls box if moved or its life if a bomb could have been easily avoided.
special "do-nothing" action
2.2 BLAME
It has been shown that human notion of causality is affected by what is normal (Kahneman & Miller, 1986; Cushman et al., 2008; Knobe & Fraser, 2008; Hitchcock & Knobe, 2009). Here, we resort to concepts of normality from actual causality to find if the agent's action is to blame for what happened. Halpern & Hitchcock (2014) propose to compare what actually happened with what normally would happen. Following this idea we use normative world in replacement to y
ICE = - -
(3)
where By is the value Y would normally take. Such value is of course contingent to the notion of normality used, which is to us to define. Note that since we are interested in the magnitude and direction of the effect, Eq. 3 uses the difference rather than the less specific inequality used in Eq. 1.
Advantage function as Blame: A typical use of this formulation is to compute the causal effect of an action on the return relative to a policy π as
(s) = -s) (c(
Q(s,()))=))))) (s)
= A(s,a).
(s) is the return the agent would get if action a were to be taken at state s and is typically estimated using state-action value function (ss,a). The choice of normality for (a(s) is to estimate the expected return with the state-value function (s)., giving us the advantage function A(s, a).
As described in Sutton & Barto (2018). Generalized Value Functions aim to integrate general knowledge the world; leaving return as special case. Following the same idea, we can reformulate Eq. 3 to compute the controlled effect of an action
ICE Ep
ep
Bep
(5)
c
E 2.0
[
Note that in stochastic environments there may be multiple next observations o' for each action à E A. To simplify notation, the following sections use controlled effect e ICE and normal effect en = Bey. Intuitively, Eq. 5 builds normal world by observing every alternative ep produced by each action creating an average perceived effect. Consider the example in Fig. 1 (middle), moving left or doing nothing would make the agent's health decrease. Eq. 5 would indicate that changes to the health bar are normal; thus, the loss of health when moving left or staying would be attributed to the agent. On the other hand, moving right would only attribute the change the agent's position as controlled. Note that the explosion would never be credited to the agent.
Special care needs to be taken when constructing the normal world Se, for continuous action spaces. Computing counterfactuals on an infinite number of possibilities cannot be done and some approximation needs to be implemented. Although our experiments use discrete actions, the proposed method in the following section is equipped to handle continuous action spaces since it does not compute counterfactuals for each possible action but approximates the normal world directly. is also important to notice that the controlled effects Eq. 5 can identify in partially observable setting (o a s), are constrained to those observed by the agent. Nevertheless, humans cannot perceive every change in state but can identify relevant controlled effects for their survival and joy.
3

---

Under review as a conference paper at ICLR 2022
a
Controlled
ha
Dec
Ec
Encoder
ho
e
Normal
hn
Coc
én
Figure 2: CEN divides the latent space of a forward model into controlled and normal branches. Each branch disentangles controlled and normal effects and decodes each into pixel space independently.
UNSUPERVISED LEARNING OF CONTROLLED EFFECTS
In practice, we do not have access to every world and cannot compute Eq. directly. We propose an unsupervised method that disentangles controlled and normal effects only using perceived effects as a self-supervised training signal.
Here, we introduce Controlled Effects Network (CEN), depicted in Fig. 2. CEN is based on forward model, where observation and action are used to predict the outcome performing such action the environment. In contrast conventional forward models, CEN divides its latent space into controlled and normal representations; similarly to Dueling Networks (Wang et al., 2015). These two representations approximate the controlled and normal effects in latent space. A decoder converts these latent representations into pixel space allowing to estimate e + C, ep as in Eq. 5.
The controlled branch has privileged access to the action; having only this branch would make CEN a regular forward model, i.e., the controlled branch alone can predict the perceived effect resulting from the action. Then, why do need the normal branch? The role of the normal branch is to force the controlled branch to predict only what is not predictable from the observation alone and hence, modeling what is controlled by the agent. mechanism where only what can be controlled will be represented by the controlled branch. To promote the controlled branch to model only controlled effects, we use the following loss
way, the normal branch acts as distillation
C MSE (eg + Gn, pp)) a MSE (G,, E
where the first part is the reconstruction loss in which the predicted target ep = E is compared to the perceived effects provided by the environment. The second part of the loss enforces the network to use the normal branch as much as possible to model the world. Since this branch cannot predict everything without the action the model will converge to the expected effect due to the MSE loss. Additionally, hyperparameter a regulates how much normal branch should model the environment. In practice, we found that this hyperparameter creates an agreement between branches on uncertain futures which seemed to be critical in environments with stochastic entities.
Let look again at the example in Fig. 1 (middle) and assume the agent picks the do-nothing action. The normal branch is encouraged to model the bomb since it does not depend on the action. Furthermore, it should not predict any change in health since what is normal is for health to not change. Thus, the controlled branch must model the change in health.
EXPERIMENTS
This section evaluates CEN on three main questions: 1) Can CEN identify controlled effects at pixel-level? i.e. can it produce an accurate segmentation mask? 2) Some applications may not require pixel-level precision; we asses CEN on predicting controlled effects attribute-level from both pixel masks and latent representations. 3) Can RL. agents benefit from using CEN intrinsic motivator?
Environments: we use multiple environments (Fig. 3) to answer the above questions, each showeas- based on Griddly (Bamford
ing a different aspect of what can be controlled. These environments et al., 2020) and Atari ALE (Bellemare et al., 201 2a), both using the Gym interface (Brockman et al., 2016). More details of these environments are given in each experiment and appendix.
Networks, training and evaluation have been implemented using PyTorch (Paszke et al., 2019), NumPy (Harris al., 2020) and PFRL (Fujita al., 2021); our experiments are managed using W&B (Biewald, 2020).
4

---

Under review as a conference paper at ICLR 2022
Figure 3: Suite of environments used for the experiments. From left/right top/bottom: Clusters,
Spiders, Lights, and Montezuma's Revenge (MZR).
Baselines: We use Attentive Dynamics Model (ADM) (Choi al., 2019) in experiments 1) and 2). ADM is an oction-predictton model based on spatial attention mechanism. It works by predicting the action performed on individual image patches. Then, a spatial attention mechanism selects sparse set of patches to use when making final prediction of the action. The masks produced by the attention mechanism are considered controlled aspects of the environment. Although ADM does not produce pixel level masks, only patch level, our knowledge ADM is the most competitive method to provide pixel-level information about controlled aspects. For 2) and 3) we rely on Never Give Up (NGU) (Badia al., 2020b), the current SOTA for exploration, and its inverse model.
Implementation: CEN is implemented as an decoder architecture with 2D convolutional layers and ReLU activation functions; the normal and controlled branches implemented with linear layers. Additionally decoder weights are shared. Throughout the experiments we use the same neural networks and hyperparameters unless specified otherwise. Our implementation of ADM uses the same architecture and hyperparameters proposed in Choi et al. (2019), See appendix for more details on the architecture and hyperparameters.
4.1 CONTROLLED EFFECTS AT PIXEL-LEVEL
This set experiments explores CEN's ability to identify pixels corresponding to controlled entities. Although CEN computes the magnitude and direction of the effects, we create binary mask by setting a threshold for the predicted controlled effects (see exact details in appendix B). We report pixel Fl scores between ground truth and predicted binary masks. The network is trained minimize Eq. 6 using the ADAM optimizer (Kingma & Ba, 2015) and 500K samples of the form (o, eg) collected using random policy.
4.1.1 CONTROLLED UNCONTROLLED EFFECTS
Here we use the Spiders environment to evaluate CEN's ability to disentangle controlled from uncontrolled effects. This environment has two main entities, the agent and spider. The controlled masks must only focus on the agent and ignore the spider.
Fig. 4a shows the pixel Fl score for our model and the baseline. CEN is able to correctly disentangle controlled effects and can produce accurate masks. Although our implementation of ADM can predict the agent's 's action with 88% accuracy, is not capable of modeling the agent's controlled pixels. We conjecture that this is due to ADM's is sparse softmax mechanism; nonetheless this behavior persisted when increasing entropy weight which should produce more dense masks.
4.1.2 NEARRY CONTROL.EED EFFECTS
Models based on action prediction are expected to work well on aspects related to the agent. For example, if an agent moves box due to moving right; the box's movement is also controlled. is unclear why these models would pay attention to the box since just knowing where the agent is, suffices to predict the chosen action. CEN's controlled branch, on the other hand, is motivated to model the box's 's effect since the normal branch would predict that the box stays where it is. We call "nearby" controlled effect to an effect that happens adjacent the agent, like the box 's movement. To evaluate CEN this kind of effects colored boxes to their corresponding fixed colored blocks. Fig. 4b shows that CEN can precisely model effects the agent and boxes. We breakdown individual effects account for the class
the Clusters environment where an agent needs to move
5

---

Under review as a conference paper at ICLR 2022
Spiders Fl avalar
CEN (purs)
ADM
1.0
0.8
0.2
0.0
0.4
1.0
0.8
0.2
06
0.0
0.0
CEN (ours)
ADM Iraining (1e5)
(b)
Figure 4: a) CEN can correctly disentangle the agent from the randomly moving objects. b) Clusters environment where CEN able model not just the agent also the movement of boxes.
imbalance between the agent's 's movement and the boxes. CEN seems to make more mistakes with boxes than the agent but nonetheless, it can consistently model both.
4.1.3 FARAWAY CONTROLLED EFFECTS
In contrast to the previous experiment, we want to evaluate if CEN can model distant effects, i.e. effects that reasonably far away from the agent's location. In this case, we the Lights environment, Here the environment presents two buttons of different color that, when pressed, turn on their corresponding light. Lights are relatively far away from their corresponding buttons thus making it difficult to model them. As show in Fig. 5a, CEN is able to model this kind of effects.
4.1.4 CONTROLLED EFFECTS IN MONTEZUMA'S ' REVENGE
This last pixel-level experiment evaluates CEN on Montezuma's Revenge (MZR) environment. Although agents in Atari environments have limited control over the environment, the relatively complex dynamics of Montezuma's 's Revenge makes it a challenging test-bed. Results shown in Fig. 5b indicate that CEN also model controlled effects.
Unlike previous environments, CEN's Fl score behaves differently due to two main reasons: 1) Fl decreases over time; the Fl score is extremely sensitive for empty ground truth, single pixel marked as controlled takes the score from to 0. At the beginning of training, CEN produces empty masks correctly but performs poorly controlled effects, which makes the F1 score artificially high. Over time it improves at controlled effects but starts marking very few pixels as controlled for effects where nothing is controlled. 2) F/ is lower: the sensitivity of the F1 on controlled effects since pixels outside the ground truth are penalized the same matter how close the ground truth they are. Appendix A.2 includes additional masks showing that CEN fails by pixels close to the controlled object instead of identifying the wrong object as controlled.
also affects CEN's performance
4.2 CONTROLLED EFFECTS AT ATTRIBUTE-LEVEL
In some cases requiring pixel-level precision be excessive. The following experiments analyze how different representations can predict effects on attributes from the environment' 's state, e.g. changes on the agent's (z. y) location if a light turned on. To this end, we use a probing technique similar to the one described in Anand et al. (2019). This approach trains classifier per each attribute interest in the environment's state using frozen versions of trained networks to produce the classifiers inputs. More specifically pixel-level masks (as the previous section) or the model's latent representation.
two different sources for the probing classifiers,
For the first case, we produce binary mask, in the previous experiments, to occlude perceived effects and use these to train each probing classifier. Classifiers have to predict there was positive, negative none effect. Note that the classifiers need predict any effect, just controlled. Thus, the classifier should only be able to predict accurately controlled attributes such as the agent's position but should fail predicting uncontrollable effects like the location of the spider skull. We use

---

Under review as a conference paper at ICLR 2022
MZR F1
lourst
1.0
0.4
0.2
0.0
avator
light
0.4
0.6
0.4
0.2
0.0
0.0
(a)
CEN tours)
ADM Training steps (1e5)
(b)
Figure 5: Fl score on the Lights (a) and MZR (b) environments. CEN outperforms the baseline even in an environment with more complex dynamics and features.
a random policy to collect dataset of 35K samples of the form (m * ep.y) v) where m is the mask produced by the model and is the ground truth class. Each dataset is split into typical 70/20/10, allowing a 20% class imbalance. We report F1 score of each attribute on the test set.
4.2.1 LATENT PROBING FROM PIXELS
The results in Table [(first block) indicate that CEN significantly outperforms the baseline when predicting controlled effects for state's attributes, and thus modeling controlled effects accurately. Furthermore, for both Spiders and Montezuma's 's Revenge environments the model cannot predict the uncontrolled effects, expected. Even though ADM's action prediction accuracy high (~ 88%) on every environment, it is not able to consistently predict controlled effects at attribute-level.
4.2.2 LATENT PROBING FROM LATENT REPRESENTATIONS
In this case, we train classifiers using a latent representation instead of pixels. We use the latent repre- sentation from CEN's controlled branch (he). It is unclear how to create a latent representation from ADM, so we use an inverse model. The features of current and next observations are concatenated to create latent representation of what is controlled. As before, controlled attributes from the latent space of these models. The probe predicts changes on the x, and direction (if applies) for agent, spider, skull and boxes; and on/off for lights and buttons.
train linear probe to predict
Table (second block) show that CEN improves on the baseline's performance. Although the inverse model is closer CEN's performance than ADM, it still has difficulties predicting the agent and box changes in location. The reason of having high score in Skull is that the Skull is still only when the agent dies, making easy to predict from the agent's position. Removing this event leads a score of ~ 0.35 for both models.
GT
Controlled (ours)
Normal (ours)
ADM
0
Figure 6: Example masks for Montezuma's 's Revenge. Additional masks are included in appendix A.2.
7

---

Under review as a conference paper at ICLR 2022
FI PIXELS
F1 LATENT
ENVIRONMENT
ATTRIBUTE
CEN (OURS)
ADM
CEN (OURS)
INVERSE
AGENT
1.010.00
0.47
0.97-10.11
0.67
SPIDERS
SPIDER
0.35#00.3
0.25=0.03
0.41#0.01
0.44=0.02
AGENT 0.76
0.28
0.97-10.01
0.56:10.09
CLUSTERS
Box
0.78
0.32+10.19
0.95:10.02
0.77
AGENT 0.97:10.01
0.3
1.0+:0.01
0.84
LIGHTS
BUTTON
0.93+0005
0.33+
0.99++0.0
0.9
LIGHT
0.9310.04
0.41
1.0:10.0
0.99
AGENT 0.66++0.08
0.42+0023 0.20+10.08
0.91+0.02 0.61+0.03
0.88=0.02 0.61+0.04
MZR
SKULL
0.19+0.03
Table 1:
Fl score for the state attributes when predicted from pixel or latent space. Note that the lower the score for Skull and Spider the better.
4.3 CEN AS INTRINSIC MOTIVATOR
We have shown that CEN can learn controlled effects in an unsupervised manner. Here we showcase the use of this ability as an intrinsic motivator of reinforcement learning agent. We consider two tasks, an empty environment without any extrinsic reward where the agent can only control itself and Clusters. The RL agent is implemented using PPO (vanilla). Additionally, PPO is augmented with the exploration bonus proposed in Never Give Up (NGU + Inverse) (Badia et al., 2020b). NGU composed of two modules for computing episodic and life-long rewards. For simplicity the following experiments only use the episodic module which in NGU consists a count-based method using episodic memory to approximate the number of times an agent visited each state, and an inverse model to identify controlled states. We replace the inverse model with CEN (NGU + CEN) and use the latent representation of the controlled branch compute NGU's episodic reward. In this experiments, CEN is trained along the PPO policy using experiences collected with the same policy,
Empty environment: the goal of this environment is showcase how each NGU variant rewards controlled events. In this environment, only the agent's movement is controlled; thus, rewarding for controlling the agent's location should promote uniform exploration the environment since once a location is visited, it should not be rewarded as much as visiting a new location. We hypothesize that inverse model will create similar representations for the same action disregarding the location where it was taken. This should impair exploration since the reward will be similar regardless of where the action performed.
Fig. 7a shows that agent trained with the inverse model hogs walls and corners. This is because is hard to predict the action near unmovable objects, leading to higher value. Contrarily, CEN promotes different locations more uniformly and the agent learns to explore better the environment.
Clusters environment: more challenging environment Clusters, where the agent needs move colored boxes to their respective colored blocks. This environment provides a reward at the end of the episode corresponding to the total number boxes correctly placed. Results are provided in Fig. 7b. Due the sparsity of the reward PPO does not learn correct behavior in the given time. Similarly, NGU + Inverse learns to place one box but fails to learn a general behavior to solve the task. Conversely, NGU + CEN quickly leams move boxes leading to high extrinsic reward.
5 RELATED WORK
Intrinsic motivators: A popular way of introducing behavioral biases in RL agents is the use of intrinsic motivators (Singh et al., 2005; Mohamed & Rezende, 2015). These motivators can promote different types of exploration, from observational surprise (Burda al., 2018) to control seeking agents (Pathak et al., 2017; Choi et al., 2019). Methods in the latter category have shown extremely good results achieving SOTA in important benchmarks. Choi et al. (2019) proposed
8

---

Under review as a conference paper at ICLR 2022
CEN
Vanila
Clusters
CEN
Reward
Inverse
1.0090
2030 3jsten
20100051
Environment Steps (1e6)
(8)
(b)
Figure a) State visitation maps at different points of training of the Empty environment. CEN values different locations similarly, and consequently, the agent learns to explore states more uniformly. The inverse model encourages going to walls where predicting the action is hard. b) CEN promotes the movement of boxes and consequently faster learning
Attentive Dynamics Model (ADM), an attention based method that discovers controlled elements in the environment and rewards the agent for discovering them. This method and its extension (Song et al., 2019) showed SOTA in Montezuma's 's Revenge. Badia et al. (2020b) combined control and observational surprise to promote exploration. Their method uses an episodic memory with an inverse model to promote the discovery of controlled effects and Random Network Distillation (Burda et al., 2018) to promote long term progress; again achieving SOTA Atari's hard exploration environments. These methods show the importance of identifying what an agent control.
Causality in deep reinforcement learning: Causality is central to humans; we think in terms of cause-effect. A similar method to Blame was proposed in Chattopadhyay et al. (2019), where they use causal attribution methods to analyzed the effect of inputs on a neural network's s outputs. Recent work has introduced causality into deep reinforcement learning (Foerster et al., 2018; Buesing et al., 2018; Jaques et al., 2018; Dasgupta et al., 2019; Goyal et al., 2019; Nair et al., 2019; Madumal et al., 2020) showing that this is promising avenue for the training of agents. Corcoll & Vicente (2020) proposed an attribution method to learn temporal abstractions for object-centric hierarchical RI. Bellemare al. (2012b) compute controllable aspects of the environment by generating mask with all possible controllable areas of an image and uses it as part of the policy's input. In this work, we identify the controlled effects of individual actions using causal concepts normality and blame.
CONCLUSIONS AND LIMITATIONS
This work proposes a fully unsupervised approach this problem named Controlled Effect Network (CEN). CEN creates a normative world using counterfactuals and compares what actually happened with what normally would happen to attribute changes on the environment the agent. The presented experiments show that, despite being unsupervised, this method precisely identifies controlled effects. Furthermore, CEN is showcased intrinsic motivator for RI. agents where results suggest that more targeted exploration leads to substantially better policies.
Limitations
Normality: although we propose a measure of normality in Eq. 5, this is far from ideal. We believe the way humans see normality is context dependent and should learned instead of a fixed function. This is an active research area by psychologist looking to underpin human causal judgment.
State instead of observations: our current formulation of ICE uses perceived effects as opposed to states. When CEN indicates that some change is controlled this is not (necessarily) equivalent to stating that the objects represented by those pixels are controlled. An exciting avenue explore is combining CEN with state representation model (e.g. LSTM RSSM by Hafner et al. (2019)) where changes would happen at state level.
Reliance policy: since Eq. 5 does not depend the policy, CEN needs diverse data for each action to approximate it, ideally from a random policy. This important problem not specific to CEN, any forward model needs policy that explores multiple actions to provide accurate predictions.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY
To ensure reproducibility, we have included the CEN architecture in Fig.2 while discussing the details and intuition behind the design choices in section 3. We also included all the hyperparameters used in the appendix C.3, For the RL experiments, we based our implementation on PFRL (Fujita et al., 2021), we included changes in the default hyperparameters in appendix C.3. Finally, we used the NGU episodic memory described in Badia et al. (2020b). We used the default configuration of Griddly environments except for Clusters environment our RI. experiments, where intermediate rewards were removed to make the environment sparse, details are discussed in appendix B.I. Currently, our implementation depends on internal dependencies, making impractical share. We will decouple the implementation from internal dependencies and make it publicly accessible upon acceptance.
REFERENCES
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised State Representation Learning in Atari. 2019.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark, 2020a.
Adrià Puigdomênech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies, 2020b.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula, 2019.
Chris Bamford, Shengyi Huang, and Simon Lucas. Griddly: A platform for ai research in games, 2020.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, Vol. 47:253-277 2293--27,,22 201 2a. doi: 10.1613/jair.992.. http://////as org abtp////b//// /ttb////12222 4708. cite arxiv:1207.4700
Marc G. Bellemare, J. Veness, and Michael Bowling. Investigating contingency awareness using atari 2600 games. AAAI, 2012b.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL htttts://///tt wandb. com/. Software available from wandb.com.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
/0..//666.0000 01540. cite
arxiv: 606.01540.
Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean- -aan-Baptiste Lespiau, and Nicolas Heess. Woulda, coulda, shoulda: Counterfactualll-gguin policy search, 2018.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov Openai. Exploration by Random Network Distillation. 2018.
Aditya Chattopadhyay, Piyushi Manupriya, Anirban Sarkar, and Vineeth N Balasubramanian. Neural network attributions: A causal perspective. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 980- PMLR, 09-15 Jun 2019. URL http: //proceed ings. ngs. ml r. press 99 chatttoppdhyaaa 9a html.
Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learn- ing. In L. Saul, Y. Weiss, and L. Bottou (edsa)n in Neural Information Processing Systems, volume 17, pp. 1281-1288. MIT Press, 2005. URL https: / http://pooeeeennneeeeeepoe neurips. cc/ paper/2004/F5 fi e/ 5b http://eeee.eeaaaaaaaaaaaaaaaaa.
10

---

Under review as a conference paper at ICLR 2022
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and
Honglak Lee. Contngency-Aware Exploration in Reinforcement Learning. ICLR, pp. 1-20, 2019.
Oriol Corcoll and Raul Vicente. Disentangling causal effects for hierarchical reinforcement learning, 2020.
Fiery Cushman, Joshua Knobe, and Walter Sinnott-Armstronn Moral appraisals affect doing/allowing judgments. Cognition, 108(1):281-285 July 2008. doi: 10. 1oo.o//..ggniiionn220... 200.. 2008.02.005 URL attts:///oll .rgg 1 101 http///cooonoooooooooooo 2008 02 005.
Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson. Causal reasoning from meta-reinforceemnt learning. 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg. and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, 2018.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intel- ligence, 32(1), Apr. 2018. URL https: //tss///jssss s.j........... aaai org index..nn.. php AAAI AAAA//artica /article/ view/11 194.
Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa. Chainerrl: A deep reinforcement learning library, Journal of Machine Learning Research, 22(77):1-14, 2021. URL
http://jmle www.ooog/ppppeeeeo.. http//ppprs/////// /varts//////// / 376 html.
T. Gerstenberg and D. A. Lagnado. Attributing responsibility: Actual and counterfactual worlds. In Joshua Knobe, Tania Lombrozo, and Shaun Nichols (eds.), Oxford Studies Experimental Philosophy, volume 1, pp. 91-130. Oxford University Press, 2014.
Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng, Sergey Levine, and Yoshua Bengio. Reinforcement Learning with Competitive Ensembles of Information-Constranee Primitives. pp. 1-21, 2019.
Guy Grinfeld, David Lagnado, Tobias Gerstenberg, James F. Woodward, and Marius Usher. Causal responsibility and robust causation. Frontiers Psychology, 11:1069, 2020. ISSN 1664-1078. doi: 10.3389/fpsyg.2002222 20w.0y00.200000660 URL htt.s::///// /www. front ie iersin. 3389 / 2020. 01069.
Caglar Gulcehre, Tom Le Paine, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, Gabriel -Brrth-Maron, Ziyu Wang, Nando Freitas, and Worlds Team. Making efficient use of demonstrations to solve hard exploration problems. In International Conference Learning Representations, 2020. URL https: /ttts::/oppee /ttp////eeeeeeeeeee//eeeeeeeeeee net/ www.eeewwwwwweeettttmeiiieee Srrrummid-Syyyteee
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels, 2019.
Joseph Y. Halpern Actual Causality. The MIT Press, 2016. ISBN 0262035022.
Joseph Y. Halpern and Christopher Hitchcock. Graded Causation and Defaults. The British Journal for the Philosophy of Science, 66(2):413-457, (44 2014. ISSN 0007-0882. doi: 10. /xx///
Charles R. Harris, K. Jarrod Millman, St'efan van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk. Matthew Brett, Allan Haldane, Jaime Fern't andez del R'10, Mark Wiebe, Pearu Peterson. Pierre G' Grard-Marchhan. Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825))337-36666 September 2020. doi: 10. 10.1038/441586-000-2266 URL https: / http://ooooooooo///////// 038 / a41 441586 22002649--2
Christopher Hitchcock and Joshua Knobe. Cause and norm. Journal of Philosophy, (06(11):577-61, 2009. doi: 10.5840//phi12009900

---

Under review as a conference paper at ICLR 2022
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A. Ortega, DJ Strouse, Joel Z. Leibo, and Nando de Freitas. Social influence as intrinsic motivation for aulti-agent deep reinforcement learning, 2018.
D. Kahneman and D. T. Miller. Norm theory comparing reality to its alternatives. Psychol Rev Psychol Rev, 93(2):136 1986.
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent experience replay in distributed reinforcement learning. International Conference on Learning Representations, 2019, URL https: http///pppeeeeeeeeeeeeeeeeeeeeee net www.ee/oooormmidr.lly......e....
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arxiv. org/ 1112..111100 6980.
Joshua Knobe and Benjamin Fraser. Causal judgment and moral judgment: Two experiments, pp. 441 447, The MIT Press, United States of America, 2008. ISBN 9780262195690.
Antonia F Langenhoff, Alex Wiegmann, Joseph Y Halpern, Joshua Tenenbaum, and Tobias Gersten- berg. Predicting responsibility judgments from dispositional inferences and causal attributions, Sep 2019. URL psyarxiv. com/ 632vw.
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through causal lens. Proceedings of the AAAI Conference on Artificial Intelligence, 34(0)): 2493-2500, Apr. 2020. doi: (w.6////a...34433...... 5631. URL https://ojs.... awa.saaaaaioo.r............ index. paceex.pppaaeee www.pppppp/A///... www.t//tttttttteeeeeee....aeee.e
5631.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning, 2015.
Adam Morris, Jonathan S Phillips, Thomas Ieard, Joshua Knobe, Tobias Gerstenberg, and Fiery A Cushman. Causal judgments approximate the effectiveness of future interventions, Apr 2018. URL
www.xxxi..cm///3.... ng3z..
Suraj Nair, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Causal induction from visual observations for goal directed tasks, 2019.
OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction, 2017.
J Pearl, M Glymour, and Jewell. Causal Inference Statistics: Primer. Wiley, 2016. ISBN 9781 1 191866877
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X:
12

---

Under review as a conference paper at ICLR 2022
Satinder Singh, Andrew G Barto, and Nuttapong Chentanez. Intrinsically Motivated Reinforcement Learning. IEEE Transactions on Autonomous Mental Development, 2(2):70-88) 82, 2005. ISSN 19430604. doi: 10019//TAMD..2212255500000
Yuhang Song, Jianyi Wang, Thomas Lukasiewicz, Zhenghua Xu, Shangtong Zhang, Andrzej Wojeicki, and Mai Xu. Mega-reward: Achieving human-level play without extrinsic rewards, 2019.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning An Introduction. 2018. ISBN 9780262039246.
Oriol Vinyals, Igor Babuschkin, Wojciech Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg. and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575, 11 2019. doi: 10.
172-----
Ziyu Wang, Nando de Freitas, and Marc Lanctot. Ducling network architectures for deep reinforce- ment learning. CoRR, abs/1511 1.06581, 2015.
13

---

Under review as a conference paper at ICLR 2022
Appendices
A ADDITIONAL EXPERIMENTS
A.1 EFFECT OF ALPHA ON CEN
Here study how affects the loss in Eq. 6. This experiment uses Montezuma's Revenge and the same setup as in Experiment 4.1. We analyze effect of alpha on CEN using values ranging from 0.01 to 20.
F1
0.01
1.0
1.0
0.1
20.0
0.8
0.4
0.2
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
F1 not empty
F1 empty
1.0
0.6
0.4
0.2
0.0
0
1
1
Figure 8: Ablation study on the effect of in Eq. 6 on CEN in MZR.
As can be seen in Fig. 8, the different alphas do not impact the performance when ground truth or controlled effects are not empty. On the other hand, the performance degrades the higher the alpha. We hypothesize that this behavior happens when the normal branch is forced to model controlled effects too strongly and the controlled branch needs to counter those bad predictions. In this case, the controlled branch will produce wrong masks, especially for empty ground truth.
Controlled (0.01)
Controlled (0.1)
Controlled
Controlled (201
Normal [0.01]
Normal (D.1)
Normal
Normal (20)
Figure 9: Examples of masks for = 0.01 and a 20 MZR. Masks generated by the controlled and normal branches for each (in parenthesis) are highlighted (darker).
14

---

Under review as a conference paper at ICLR 2022
A.2 MASKS
Here we provide masks for both CEN and ADM. Mask are based on effects and extend over two frames, for visualization purposes here we only show the next observation with the full mask. It can be seen that CEN fails for few pixels in MZR, which may explain the low Fl score for empty GT.
Controlled (ours)
Normal (ours)
GT
Controlled (ours)
Normal (ours)
ADM
F1: 0.49
F1: 0.35
A.A.A.
J.A.A.L
J.AAA.
GT
Controlled (ours)
Normal (aurs)
ADM
F1: 0.00
F1: 0.00
Controlled (ours)
Normal (ours)
Figure 10: Examples of success and failure cases for CEN and ADM in MZR.
15

---

Under review as a conference paper at ICLR 2022
Controlled (ours)
Normal (ours)
ADM
0.30
GT
Controlled (ours)
Normal (ours)
ADM
F1: 0.86
F1: 0.25
Figure 11: Examples of masks for Clusters.
Controlled (ours)
Normal (ours)
Controlled (ours)
Normal (ours)
Controlled (ours)
Normal (ours)
ADM
Figure 12: Examples of masks for Spiders.
16

---

Under review as a conference paper at ICLR 2022
Controlled (ours)
Normal (ours)
ADM
F1: 0.69
F1: 0.26
GT
Controlled (ours)
Normal (ours)
ADM
Figure 13: Examples of masks for Lights.
B ENVIRONMENTS
B. GRIDDLY
Griddly (Bamford et al., 2020) is highly optimized grid-world based suite of environments. Environments used in this work based on Griddly generate 64 64 pixel observations, although size of the grid-worke may vary. Griddly supports multiple rendering formats, this work uses the 2D rendering of sprites.
Spiders: is a 6x6 arena where a Gnome (the agent) has to grab a Gem without being killed by Spider. The agent dies if it collides with the spider. In this environment, the agent move left, right, down or stay. The spider takes an action randomly from the following: rotate left, rotate right move forward. This environment's controlled entities are: Agent.
Clusters: a 13x10 arena where a Knight has to move boxes of the same color their corresponding colored-bloct without touching the spikes. There are two different colors, blue and red. The agent is rewarded with whenever box is pushed towards similar colored block. The agent dies if it collides with spikes or if box is destroyed by spikes. The agent can move left, right, up, down or stay. This environment's controlled entities are: Agent and Boxes. For RL experiments, only rewarded the agent after all the boxes of the same color are pushed to blocks. Since the agent not get any reward whenever box is stuck to wall. We removed boxes that touches the wall and punished the agent with -0.01. We then scaled the reward of solving color with the number of boxes pushed to the block. This modifications encourage the agent solve the environment by pushing the maximum number of boxes into blocks while preventing from getting deprived from reward by accidentally pushing boxes to walls.
made the environment more sparse by removing all intermediate reward and
Lights: 11x8 arena where Ghost (the agent) has to turn all the lights on by pressing each button. Buttons and lights are colored either blue or green. Pressing button of one color turns the light of the same color on. The agent can move left, right, up, down or stay. This environment's controlled entities are: Agent, Buttons and Lights.
Empty: this environment is copy of the clusters environment where all the boxes, blocks, spikes and rewards were removed.
17

---

Under review as a conference paper at ICLR 2022
Ground truth: Griddly provides access to each entity's state (x, y, light is on/off, etc): binary mask is produced for each controlled entity with any of its attributes changed when transitioning between two time steps. and y coordinates bounding box is generated using the size of that entity. Note that the coordinates may be different between steps thus the generated mask may enclose multiple locations. The resulting masks for each entity are combined into a single mask m by taking the maximum value among them. Since we want know what pixels were actually controlled, the final ground truth mask produced as: m eq
projected into pixel space and
B.2 ATARI MONTEZUMA'S REVENGE
The ALE (Bellemare al., 2012a) provides access to Atari 2600 games learning methods like RL. As it has been popular choice by methods using inverse models, in this work we use the game Montezuma's Revenge evaluate CEN. This environment provides uncontrolled as well as controlled effects with more complex entities. The environment typically generates observations of 210x160 pixels which we downscale to 64x64 pixels. Additionally the action space of size 10.
Ground truth: in contrast to Griddly, we can actually compute counterfactual worlds by saving and loading the state of the game (RAM) multiple times when taking different actions. For this, we directly compute Eq. 5 using ALE's special calls cloneSystemSState and restoreSystemmStat More precisely, we compute every possible perceived effect reachable from the current state and build normal effect using the mode over all possible effects. Then, we compare the perceived effect for the agent's chosen action against the normal effect. The ground truth mask will have Is where these two effects are different.
C TRAINING
C.I ARCHITECTURE
Encoder: is composed of two 2D convolutional layers with 4x4 kernels, stride and padding of 1. Additionally, we have 2 residual blocks each with two 2D convolutional layers with stride and padding of 1. The first layer has a kernel of 3x3 and the second layer of 1x1. ReLU used as activation function throughout the network; BatchNorm is used between each layer; and 64 channels on every convolutional layer. We project the resulting maps into a flatten vector of size using a linear layer with ReL.U activation function.
Decoder: this module is composed of six 2D transposed convolutional layers all having 4x4 kernels, stride 2 and padding of Each layer uses ReLU as activation function but the output layer which uses Tanh activation. Every layer uses 64 channels with the exception of the last layer which outputs a channel prediction of the perceived effects. Parameters are shared among the controlled and normal branch decoders.
Controlled and normal modules: both modules are composed of three linear layers with 32 hidden units, each with a ReLU as activation function. The input to the controlled branch are the encoded observation and an embedding of size 8 of the chosen action.
PPO Agent: uses an encoder consisting of 3 convolutional layers with (channels, padding, strides) equal to (32, 8, 4), (64, 4, 2), (64, 3, 1) respectively. The encoder is followed by two linear layers of sizes 512 and number actions respectively, transform the feature map to environment's number of actions.
C.2 MASK GENERATION
CEN's controlled masks generated using the encoder, controlled branch and decoder. The predicted controlled effect is binarized using a threshold as mcEN = (-T < ee) > T). In the case of ADM, its attention mask is thresholded in the same way and resized to the size of the effect.
18

---

Under review as a conference paper at ICLR 2022
C.3 HYPERPARAMETERS
Name
Value
Sweep [16, 32, 64, 128] [16, 32, 64, 128, 256]
hidden size
32
latent size
128
channels
64
[16, 32, 64, 128]
learning rate
0.0001
[0.0001, 0.0005, 0.001, 0.005]
0.01
[0.001, 0.01, 0.1, 1,5, 10, 20, 30, 50]
0.01
Table 2: CEN hyperparameter sweeps and final values used.
Name
Value
Sweep [0.01, 0.05, 0.1, 0.5, I, 5]
entropy
0.05
hidden size
64
[16, 32, 64, 128]
attention size
128
[32, 64, 128, 256]
learning rate
0.0001
[0.0001, 0.0005, 0.001, 0.005]
T
0.01
Table 3: ADM hyperparameter sweeps and final values used.
Name
Value
Sweep
encoder channels
32
[32, 64] [16, 32, 64, 128)
encoder hidden
32
latent size
128
[64, 128, 256]
learning rate
0.0001
[0.0001, 0.0005, 0.001, 0.005]
Table 4: Inverse model hyperparameter sweeps and final values used.
Name
Value
Sweep
batch size
512
[64, 128, 512, 1024]
latent size
32
[8, 16, 32]
CEN encoder output size
128
[16, 32, 64, 128]
learning rate
0,0005
[0.00005, 0.0001, 0.0002, 0,0005, 0.001] [0.0001, 0.001, 0.002, 0.005, 0.01, 0.1]
IR Beta
0.001
Rollout size
2048
[1024, 2048, 4096)
discount
0.95
epochs
10
Table 5: PPO hyperparameter sweeps and final values used.
19


Reviews:
[{"review_id": "n_QJWM9id-J", "reviewer": "Reviewer_Y2tc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduces a representation-learning system, CEN, that is both conceptually simple and well-motivated from the perspective of causal learning. The writing (up until the implementation and experimental details) was very clear and remarkably self-contained compared to other work at the intersection of causality and RL. However, I cannot currently endorse this paper's acceptance due to several significant issues with the experimental results, and (to a less extent) some methodological limitations that deserve discussion. As per the paper (and my summary), there are 3 sets of results and I'll go through my issues with them sequentially.\n\n(Section 4.1) The comparisons to the ground truth masks are unproblematic from an evaluation standpoint -- I truly believe CEN dramatically outperforms ADM on this task. But the significance of this is far less than the other experiments. The \"ground truth\" mask of controllable elements is *defined* to be a function of the difference between the average (across all actions) change in pixels and the change in pixels for the particular action actually taken. This is exactly what CEN attempts to model, and quite unlike what previous approaches consider. It is thus unsurprising that CEN wins out on this metric. Indeed, the utility of matching these masks is what needs to be shown; these results are a necessary sanity check, but can't stand alone without the other results. It should reframed in this light. This is a smaller point, but what's going on with the Montezuma result in Figure 5b? Why is the performance of CEN going down over time?\n\n(Section 4.2) I view Table 1 as not supporting the argument of the paper. The main results to my mind is that CEN is better than alternatives at modelling the agent. If that was the goal of CEN then fine, but it is not. The agent is the aspect of the environment that the authors claim to modelled by all of these methods and emphatically not the thing they're trying to improve upon. Both ADM and Inv model box, button, and light (the indirectly controllable objects) at least as well as they model the agent. Indeed, Inv actually matches CEN on button and light, despite being significantly worse on agent.\n\n(Section 4.3) Figure 7a is not the clean uniform vs corner-seeking distinction that it is written up to be. Early in training CEN appears to be as drawn to corners as Inv, and even at the end there is a significant bias towards edges that isn't explained by your account. Figure 7b is clearly significant, but it is not clear that this is due to the properties of CEN highlighted throughout. Table 1 shows that CEN outperforms Inv on modelling the agent, so perhaps that is the cause of the performance gain? And this might be down to a matter of tuning, since Inv was never trained on this environment before (at least in the NGU paper) -- why shouldn't agent modelling performance be maxed out here? And would the performance gap be eliminated if this were the case? I believe that it would.\n\nUnstated limitations. The reliance on training under the uniform random policy is a big issue e.g. preventing the usage of CEN to improve the NGU paper's results. The theoretical motivation doesn't have anything to say as to why this behavior policy is needed, but the action-agnostic predictions are highly reliant on the policy, which would continuously change if CEN was trained online. Even training from a fixed dataset of expert trajectories would be problematic e.g. experts near-deterministically kill enemies, therefore enemies are no longer controllable under your formulation (since their deaths are predictable without action information). Perhaps importance sampling could correct for this, but that is an empirical question that needs answering before CEN can be said to be scalable.\n\nA pair of small related concerns: why are the decoder weights tied? Why is Equation 6 a mixture loss instead of only trained the parameters of the action-agnostic stream with the second term and the parameters of the action-aware stream with the first time? Feels like you could eliminate a hyper-parameter without hurting performance.", "summary_of_the_paper": "The authors' introduce the Controlled Effects Network (CEN), an approach to separating the controlled aspects of the environmental dynamics from those that are agnostic to the agent's actions. This occurs on a per-pixel basis. In order to accomplish this separation, the network architecture forces the next step prediction to be the sum of two separate predictions (only one of which has access to the agent's action). To prevent the action-aware stream from simply modelling everything, a mixture loss is used the cares explicitly about the accuracy of the action-agnostic stream's predictions in addition to that of the aggregate. Training occurs on a fixed dataset collected with a uniform random policy, which is important since the best action-agnostic prediction is highly dependent upon the behavior policy. The utility of this approach is demonstrated in 3 ways: comparisons to the ground masks of controlled effects, decodability of attribute changes in controlled effects (and not un-controlled effects), and RL performance when used as the representation for an intrinsic motivation system.", "summary_of_the_review": "A clear and well motivated algorithm is let down by (1) experiments that fail to provide evidence to support the paper's claims. (2) An unstated reliance on an unrealistic behavior policy also limits this approaches applicability. In its current state, I must therefor recommend rejection. That said, I could definitely see accepting this with some significant revisions. To address (1), the experimental evidence must better match the motivation of the approach. You could perform new experiments that show CEN models more than just the agent whereas other approaches don't. You might also reconsider your framing; perhaps the agent/object distinction isn't relevant here and CEN should be better for a different reason.  To address (2), acknowledging the limitations (or correcting me if I'm mistaken) is enough. A first step towards a promising new direction is enough for acceptance; no need to solve everything in one paper, but being explicit about the state of things is a must.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "gadTciqE8gW", "reviewer": "Reviewer_fGPn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper makes a valuable contribution toward modeling controlled effects of agent behavior, which is important for causal RL and exploration—both recent hot topics in RL. The method is clearly motivated and well explained, and the paper is well-written. The experiments are presented in a logical order, and they systematically and thoroughly evaluate several important aspects of the algorithm.\n\nIn terms of improvement, I would like to see additional experiments for NGU + CEN, on more complex tasks. I was a bit disappointed that the only experiments on using CEN as intrinsic motivation for exploration were on an empty world and a grid world task.\n\nIn addition, I have a couple questions about the applicability of CEN. First, I am curious whether CEN would work for first-person cameras. Would it get confused by the entire observation / camera image changing as the agent’s gaze direction changes from frame to frame? If so, this a limitation that should be mentioned and discussed in terms of how it could be addressed.\n\nI also am curious about the ability of CEN to scale to environments with more rich observations. For Montezuma’s Revenge, the pixel prediction of controlled effects is less accurate. The hypothesis is that this is because this Atari game has more complex visual scenes, compared to the grid worlds, which affects the ability to predict accurate masks. But this Atari game is still quite simple compared to real-world environments. Does this imply that CEN would not work for more realistic domains?", "summary_of_the_paper": "This paper proposes an approach, called Controlled Effect Network (CEN), for agents to predict the controlled parts of the environment. CEN is a dynamics model that separately predicts the \"controlled\" and \"normal\" change in observations, given the current observation and action. The controlled part is the change caused by the agent's action, whereas the normal/counterfactual part is what would have happened normally (e.g. this could be computed by averaging over the outcomes of all possible actions of the agent).\n\nThe empirical experiments show that CEN outperforms previous approaches for predicting the controlled part of the observation, on grid world tasks and Montezuma’s Revenge where the observations are images. In addition, it improves exploration when combined with NGU (Never Give Up) on a couple simple tasks, compared to the inverse dynamics model used in prior work.", "summary_of_the_review": "CEN is a simple, elegant solution for disentangling controlled and normal effects, that is backed up by thorough empirical evaluation. However, it leaves a few open questions that I'm hoping the authors can answer in the rebuttal: 1) can CEN scale to domains with more rich and complex observations?, 2) can CEN handle first-person observations, and 3) does CEN improve exploration for more complex domains? Additional experiments and discussion on these areas would significantly strengthen the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q8vioEZzNQK", "reviewer": "Reviewer_hgSF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Simple method based on nice intuition\n- Clear writing and presentation\n\nWeaknesses\n- Limited empirical evaluation\n\nDetailed comments\n- Reason for the choice of a baseline is not properly provided in the paper. First of all, what is Attentive Dynamics Model? How does it learn representations? Which characteristic of ADM makes you consider it as a baseline? What's the expected message from the experiments comparing the proposed method to ADM? Why not just learn the representations from inverse dynamics and learn to predict the difference between observations using such representations? Just stating that '`ADM produces an attention mask for pixels controlled by the agent therefore it is used as baseline in these experiments.` does not convey meaningful information and makes paper not self-contained.\n- Ablation study to show that the proposed component is indeed important for the improved performance is missing. How does the method compare to using representations from a dynamics predictor without normal branch (objective should be the same by predicting the difference between observations)? How does the performance changes, and predictions differ by changing $\\alpha$?\n- Missing comparative evaluation in terms of episode returns / success rates on standard benchmarks. Such experiments are only done in Clusters environments, and not clear why the performance is only reported only on that task. More investigations on hard-exploration games in Atari and DeepMind Control Suite are required for demonstrating that the proposed method is indeed better than inverse dynamics model for learning representations useful for capturing the novelty of states without affected by random aspects of the observations.\n- In many cases, an easy solution for the network to solve the optimization problem might be to just ignore the normal branch and only utilize the controlled branch for prediction. Why do you think the method can work without collapsing into such case ignoring a normal branch? How do the predictions from controlled and normal branch differ qualitatively? (if I understand correctly, it should be possible to visualize the outputs from the decoders of both controlled and normal branches). How does the method perform on such environments, and compare to baseline of inverse dynamics predictor?", "summary_of_the_paper": "This paper proposes a method to identify the controlled effects from actions based on the concept of 'blame'. The proposed network consists of two branches that conditions on both current observation and action and only current observation for predicting the difference of two consecutive observations, respectively. By introducing these two branches, representations from the controlled branch can capture the controlled effects from actions which cannot be adequately captured by normal branch. The proposed method is evaluated on several benchmarks to predict the controlled effects obtained by utilizing internal states of simulators, but performance of the proposed method as an intrinsic motivator is only evaluated on a single task.", "summary_of_the_review": "I like the general idea of the paper, but several empirical evaluations are missing to support / justify the effectiveness of the proposed method. Hence my evaluation remains closer to the rejection of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CIhz_-Mm1hT", "reviewer": "Reviewer_MeF5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong:\n* I like the papers main insights about controllability (e.g. the examples in Figure 1), and that this requires a notion of ‘normal’ effects, from which controllable factors need to deviate. \n* The experiments are extensive and varied, and show good results. \n\nIssues with Sec 2:\n* I think the main weakness is in the conceptual part (Sec 2.2), because I think there is a problem in Eq. 5, being the use of the expectation over all other actions. For example, I think it already fails for your example in Figure 1, middle. I assume there are three actions (left = health decreases, do nothing = health decreases, right = health stays the same). In this case, the ICE of action ‘right’ would be the biggest, since the other two actions (in the expectation) suggest that a decreasing health is actually normal. You mention in the paper that ‘keeping full health’ is normal in this case, but I do not understand how you get to that conclusion. \n* Your practical experiments do work, which I think is caused by the fact that your neural network automatically generalizes over similar states. I think this is what needs to be corrected in Eq5: the expectation should probably also run over other states in the problem, not only over actions.\n* Moreover, I think the relation to Q(s,a) and V(s) in Eq 4 is also a bit stretched, since for a greedy policy, Q(s,a) = V(s). Now, it seems as if controllability depends on the type of policy you have, which should of course not be the case. In your Eq. 5, the policy does not seem to play a role anymore, and you simply weight each possible action equally? In any case, the jump from Eq 4 to 5 does not make sense, and I personally think Eq. 4 does not make sense in general (the dependence on your own policy). \n\nSmaller issues:\n* Sec 4.1: I do not get how you get the ground-truth masks exactly from the true state (you say ‘we use the internal environment state’, but how do you actually use this: to repeatedly sample the environment for all actions, or you manually code something?). \n* Fig 6: I think this figure needs more labels, already on the rows and columns, but also on the masks. Dark means that the mask is active, i.e. controllable? So what should I see in the picture. Right now I found it hard to learn something from the figure. \n* I find it a bit surprising that ‘the shape of lights and buttons’ (Sec 4.1.3) is so complex that performance decreases. For Montezuma’s I would buy this, but the lights environment is pretty straightforward right? Isn’t there something else going on? Did you investigate this? \n* Sec 4.1.4: Results are in Fig 5b I suppose, not Fig 8?\n* Table 1: Why is CEN for example so good at predicting the Skull in MZR (61%), for an object that is actually not controllable? This is weird right? \n* ‘Fig 7a shows how an inverse model does not’ - This sentence seems to be incomplete. ", "summary_of_the_paper": "This paper studies a new method to identify controllable effects in a learned dynamics model of the environment. While previous methods have focused on inverse models (predicting the correct action from a given state and next state), the authors argue that this method fails to properly identify controllability. Instead, they come up with a new specification of ‘blame’, based on the difference between the observed effect of an action and the average/normal effect otherwise observed (under other action). Experiments show that their approach is better able to disentangle controllable effects than the baseline, and can also aid as an intrinsic motivation reward.", "summary_of_the_review": "In general, I think this paper studies an interesting and relevant topic (controllable factors), and brings interesting new insights (inverse models do not work well, and there may be better methods). There are good conceptual illustrations of the proposed solution, but I do think there are some problems with the equations in Sec 2.2 that underpin their ideas. Their proposed network architecture and experiments do make sense, and show improved performance over inverse dynamics models. If the authors correct (or further elaborate on in the rebuttal) some parts of Sec 2, then I think the paper would be a good contribution to ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an attempt to infer controllable aspects of the environment dynamics by imposing an architecture on a pixel prediction model, such that the prediction is the sum of an action-aware and action-agnostic prediction, trained on a fixed dataset gathered under a uniform random policy, in such a way as to incentivize the action-aware predictor to model the residual of what the action-agnostic predictor is capable of predicting on its own. The utility of the method is demonstrated qualitatively and quantitatively by comparing both the effect-identification performance and performance when used in the context of intrinsically motivated RL.

Reviewers found the work to be well-motivated, highly relevant, clearly described and insightful. Several reviewers pointed to the relatively limited empirical evaluation, appropriateness of baselines, and questions as to how the method would apply to richer observation spaces. Reviewer hgSF, whose score improved by 2 points following discussion, in particular questioned the motivation for the baseline method (ADM) and asked for a hyperparameter ablation study which the authors provided. Reviewer Y2tc, whose score improved by 3 points, found that several experiments poorly supported the paper's claims and was concerned about the uniform policy requirement for training; after discussion, clarifications and a softening of some claims, the reviewer was satisfied. Reviewer fGPn raised concerns around metrics and questioned applicability to first person observations, and while the former were addressed to the reviewer's satisfaction, the latter were not, and the reviewer maintained their marginal accept-leaning appraisal. MeF5's biggest concern revolves around a conceptual issue around the expectation in equation 5 which judging by the discussion was clarified but not fully resolved.

The paper has improved significantly since submission through the diligent engagement of the reviewers. The method proposed is clever and effective in the domains considered but as fGPn points out is probably limited in applicability to observation spaces wherein non-controllable effects remain stationary, ruling out 3D locomotion or camera control for example. It is unclear whether this limitation is addressable, and the current version of the manuscript does not discuss this a perspective a great deal (in fact, direct mention of this  limitation is absent from the "limitations" section).

After discussion among the senior program committee, it was decided that the manuscript, while much improved by subsequent revisions, fell short of the acceptance bar this year, particularly in terms of rigorous evaluation of the method. We are therefore unable to recommend acceptance at this time.
Decision: Reject