Did I do that? Blame as a means to identify controlled effects in reinforcement learning

Abstract: Identifying controllable aspects of the environment has proven to be an extraordinary intrinsic motivator to reinforcement learning agents. Despite repeatedly achieving State-of-the-Art results, this approach has only been studied as a proxy to a reward-based task and has not yet been evaluated on its own. We show that solutions relying on action-prediction fail to model critical controlled events. Humans, on the other hand, assign blame to their actions to decide what they controlled. This work proposes Controlled Effect Network (CEN), an unsupervised method based on counterfactual measures of blame to identify effects on the environment controlled by the agent. CEN is evaluated in a wide range of environments showing that it can accurately identify controlled effects. Moreover, we demonstrate CEN's capabilities as intrinsic motivator by integrating it in the state-of-the-art exploration method, achieving substantially better performance than action-prediction models.

Reviews and author responses:

 I appreciate the authors' detailed answers to my questions. I understand now how despite a low F1 score, CEN is still correctly identifying the controlled objects in MZR. The follow-up experiment on MZR is also encouraging, in that engineering the network architecture could enable CEN to scale to domains with more rich observations.

I'm still concerned about points 2 and 3 though. In terms of first-person observations, I was thinking of a maze navigation setting (e.g. the VizDoom maze in Pathak et al. 2017). If the agent is navigating around in the environment, then I believe everything would be considered to be controlled, which is not very useful. This limits the kinds of settings that CEN could be applied to. I agree with the authors that it would be really interesting to see whether CEN can be extended to identify controlled effects in state space.

Regarding the exploration experiments, I agree with Reviewer hgSF that the experiment with Clusters is a bit contrived. Even if CEN is not expected to substantially outperform existing aproaches when controllability is not the most important factor, it would still be useful to see how it compares on standard baseline environments for studying exploration in RL (like Super Mario, MZR, or other Atari environments).

Overall I think this work is a great step towards identifying controlled effects, but in my view not all the claims are supported. Thus I will keep my original score of 6.

 Ok I see your point, the expectation over all states may also run into trouble. 

I mainly thought of the expectation to solve the issue that blame now depends on the number of available actions. I do not think that can be correct.

I do not know what the correct formulation should be, but I do think that you need a different formulation to remove this dependency, i.e., I think there is still room for conceptual improvement in the current paper. 

 Thanks for the response and clarification on the second point. Taking the expectation over all states would work in some cases but we believe it would be wrong in general, as shown by the following simple example.

If we have a switch that can be turned on or off and let us say its initial state is off. The agent can move around and press the switch. Taking expectation over all states would say that the most normal state of the switch is off. In this scenario, the agent would be correctly attributed the change from off to on since its action leads to an abnormal situation. The problem comes when going from on to off (the agent presses again the switch), in this case, what is normal is the same as the actual effect (or reality).

In general, taking the expectation over all states would only identify effects that go from normal to abnormal but not the other way around. The reason is that now we are not conditioning on the state but just taking a general view of the world. Although this may be desired in some cases, does not seem to work in general.

This paper proposes a method to identify the controlled effects from actions based on the concept of 'blame'. The proposed network consists of two branches that conditions on both current observation and action and only current observation for predicting the difference of two consecutive observations, respectively. By introducing these two branches, representations from the controlled branch can capture the controlled effects from actions which cannot be adequately captured by normal branch. The proposed method is evaluated on several benchmarks to predict the controlled effects obtained by utilizing internal states of simulators, but performance of the proposed method as an intrinsic motivator is only evaluated on a single task. Strengths
- Simple method based on nice intuition
- Clear writing and presentation

Weaknesses
- Limited empirical evaluation

Detailed comments
- Reason for the choice of a baseline is not properly provided in the paper. First of all, what is Attentive Dynamics Model? How does it learn representations? Which characteristic of ADM makes you consider it as a baseline? What's the expected message from the experiments comparing the proposed method to ADM? Why not just learn the representations from inverse dynamics and learn to predict the difference between observations using such representations? Just stating that '`ADM produces an attention mask for pixels controlled by the agent therefore it is used as baseline in these experiments.` does not convey meaningful information and makes paper not self-contained.
- Ablation study to show that the proposed component is indeed important for the improved performance is missing. How does the method compare to using representations from a dynamics predictor without normal branch (objective should be the same by predicting the difference between observations)? How does the performance changes, and predictions differ by changing $\alpha$?
- Missing comparative evaluation in terms of episode returns / success rates on standard benchmarks. Such experiments are only done in Clusters environments, and not clear why the performance is only reported only on that task. More investigations on hard-exploration games in Atari and DeepMind Control Suite are required for demonstrating that the proposed method is indeed better than inverse dynamics model for learning representations useful for capturing the novelty of states without affected by random aspects of the observations.
- In many cases, an easy solution for the network to solve the optimization problem might be to just ignore the normal branch and only utilize the controlled branch for prediction. Why do you think the method can work without collapsing into such case ignoring a normal branch? How do the predictions from controlled and normal branch differ qualitatively? (if I understand correctly, it should be possible to visualize the outputs from the decoders of both controlled and normal branches). How does the method perform on such environments, and compare to baseline of inverse dynamics predictor? I like the general idea of the paper, but several empirical evaluations are missing to support / justify the effectiveness of the proposed method. Hence my evaluation remains closer to the rejection of the paper.

 - Thanks again for the response and clarifying the difference between selected tasks and the tasks in Camporo et al., 2020. Now the goal of the selecting those tasks and the difficulty seems more clearer to me, and sorry for stating those kind of experiments are like toy experiments. I agree on that the method works well for identifying what is controlled by the agent as shown in the selected tasks.

- I understand that the authors want to show that `CEN can identify these controlled effects.`, as stated in the abstract. But at least to me it seems that the second main contribution of this paper is "CEN as an intrinsic motivator for RL", which is also emphasized in the last part of the abstract and introduction. Given that the paper only evaluates their method on a single RL task, it's difficult for me to make conclusion like `CEN achieves substantially better performance than action-prediction models` as an intrinsic motivator for RL.

Hence I raise score from 3 to 5 to appreciate the clarification in the response, and contribution of the method for identifying the controllable aspects of the environments, but my evaluation still remains in the negative side due to the concern on the RL part. I'm willing to update my score until the end of the discussion period if the authors can address my concern with the updated results (in a text if revision is not possible after today).

The authors' introduce the Controlled Effects Network (CEN), an approach to separating the controlled aspects of the environmental dynamics from those that are agnostic to the agent's actions. This occurs on a per-pixel basis. In order to accomplish this separation, the network architecture forces the next step prediction to be the sum of two separate predictions (only one of which has access to the agent's action). To prevent the action-aware stream from simply modelling everything, a mixture loss is used the cares explicitly about the accuracy of the action-agnostic stream's predictions in addition to that of the aggregate. Training occurs on a fixed dataset collected with a uniform random policy, which is important since the best action-agnostic prediction is highly dependent upon the behavior policy. The utility of this approach is demonstrated in 3 ways: comparisons to the ground masks of controlled effects, decodability of attribute changes in controlled effects (and not un-controlled effects), and RL performance when used as the representation for an intrinsic motivation system. This paper introduces a representation-learning system, CEN, that is both conceptually simple and well-motivated from the perspective of causal learning. The writing (up until the implementation and experimental details) was very clear and remarkably self-contained compared to other work at the intersection of causality and RL. However, I cannot currently endorse this paper's acceptance due to several significant issues with the experimental results, and (to a less extent) some methodological limitations that deserve discussion. As per the paper (and my summary), there are 3 sets of results and I'll go through my issues with them sequentially.

(Section 4.1) The comparisons to the ground truth masks are unproblematic from an evaluation standpoint -- I truly believe CEN dramatically outperforms ADM on this task. But the significance of this is far less than the other experiments. The "ground truth" mask of controllable elements is *defined* to be a function of the difference between the average (across all actions) change in pixels and the change in pixels for the particular action actually taken. This is exactly what CEN attempts to model, and quite unlike what previous approaches consider. It is thus unsurprising that CEN wins out on this metric. Indeed, the utility of matching these masks is what needs to be shown; these results are a necessary sanity check, but can't stand alone without the other results. It should reframed in this light. This is a smaller point, but what's going on with the Montezuma result in Figure 5b? Why is the performance of CEN going down over time?

(Section 4.2) I view Table 1 as not supporting the argument of the paper. The main results to my mind is that CEN is better than alternatives at modelling the agent. If that was the goal of CEN then fine, but it is not. The agent is the aspect of the environment that the authors claim to modelled by all of these methods and emphatically not the thing they're trying to improve upon. Both ADM and Inv model box, button, and light (the indirectly controllable objects) at least as well as they model the agent. Indeed, Inv actually matches CEN on button and light, despite being significantly worse on agent.

(Section 4.3) Figure 7a is not the clean uniform vs corner-seeking distinction that it is written up to be. Early in training CEN appears to be as drawn to corners as Inv, and even at the end there is a significant bias towards edges that isn't explained by your account. Figure 7b is clearly significant, but it is not clear that this is due to the properties of CEN highlighted throughout. Table 1 shows that CEN outperforms Inv on modelling the agent, so perhaps that is the cause of the performance gain? And this might be down to a matter of tuning, since Inv was never trained on this environment before (at least in the NGU paper) -- why shouldn't agent modelling performance be maxed out here? And would the performance gap be eliminated if this were the case? I believe that it would.

Unstated limitations. The reliance on training under the uniform random policy is a big issue e.g. preventing the usage of CEN to improve the NGU paper's results. The theoretical motivation doesn't have anything to say as to why this behavior policy is needed, but the action-agnostic predictions are highly reliant on the policy, which would continuously change if CEN was trained online. Even training from a fixed dataset of expert trajectories would be problematic e.g. experts near-deterministically kill enemies, therefore enemies are no longer controllable under your formulation (since their deaths are predictable without action information). Perhaps importance sampling could correct for this, but that is an empirical question that needs answering before CEN can be said to be scalable.

A pair of small related concerns: why are the decoder weights tied? Why is Equation 6 a mixture loss instead of only trained the parameters of the action-agnostic stream with the second term and the parameters of the action-aware stream with the first time? Feels like you could eliminate a hyper-parameter without hurting performance. A clear and well motivated algorithm is let down by (1) experiments that fail to provide evidence to support the paper's claims. (2) An unstated reliance on an unrealistic behavior policy also limits this approaches applicability. In its current state, I must therefor recommend rejection. That said, I could definitely see accepting this with some significant revisions. To address (1), the experimental evidence must better match the motivation of the approach. You could perform new experiments that show CEN models more than just the agent whereas other approaches don't. You might also reconsider your framing; perhaps the agent/object distinction isn't relevant here and CEN should be better for a different reason.  To address (2), acknowledging the limitations (or correcting me if I'm mistaken) is enough. A first step towards a promising new direction is enough for acceptance; no need to solve everything in one paper, but being explicit about the state of things is a must.

 Sorry about the confusion re:data; 500k steps from a uniform policy isn't too unreasonable. I think the issues around the inverse model are relatively resolved; perhaps with enough data and a different policy the inverse model would be sufficient, but that can be left for future work. The fact that your method works in this modest data regime is sufficiently interesting.

Combined with your other improvements, I'm raising my score: 3-->6

 Thanks for the quick response. We do believe that the representations created by the observation encoder of the inverse model are affected by cases where the model cannot predict the action since the loss will be bigger and the encoder will need to change to minimize the loss. Nevertheless, the reviewer is right that more experiments are needed to show this point. Just a clarification regarding the training for the results in Table 1, the inverse model is trained the same as CEN and ADM, 500K steps (we fixed the typo, before was saying 300K). The probing classifiers were trained on 35K steps. Ultimately, the point we are trying to make is that the inverse model trained for 5M steps in experiments 4.3 (clusters) works as intended, should have enough data to learn (at least as well as CEN) and that the differences in Fig. 7b are due to replacing the inverse model in NGU with CEN.

 Thank you for the additional response, I found it quite helpful.

That said, your argument against inverse models still confuses me:

>In general, action-prediction models cannot predict an action accurately when two or more actions lead to the same observation/effect.

This is fine. Since it embeds each state individually before combining them to make the prediction, the state embedding should contain all of the information needed to predict *the subset of actions that are predictable*. Even in the case of the agent being in a corner, this still means the embedding should contain information about the agent. This only fails to be the case when the data generating process is heavily skewed towards these no-op, unpredictable actions -- in which case the network might not have seen enough distinguishable actions in an area of state space to warrant representing anything. 

> In any case, the reason not to model the agent entirely (Table 1) is due to failing at unmovable objects and not because we did not fine-tune or implement the inverse model correctly.

You have not provided any evidence of this. Indeed, I've just provided a counter hypothesis about the failure mode that is not ruled out by any of your existing experiments. I don't think you've implemented the inverse model incorrectly, but I do think that the training regime (35k steps from a random policy?) is a plausible culprit. I'd be surprised if the inverse model performance didn't improve with more data and/or a better policy (e.g. reward seeking, but really any policy that isn't taking majority indistinguishable actions). You could still claim to be more agnostic to policy quality and/or more data efficient in that case, and I think that's a much more reasonable stance than your current one.


 Hi, I read your rebuttal, thanks for the feedback. 

- Regarding Eq. 2: I was not clear to me that the agent had 5 actions (up, down, left, right, nothing) instead of 3 (left, right, nothing). I now see the arrows in the thinking cloud. I do believe my point indeed stands, that the number of actions influences what is normal (which is weird), and I do not think your current explanation is fully convincing. 
- Regarding the generalization over other states: This follows from my previous comment. I think what makes "not being harmed by a bomb" normal is the fact that this is normal in most of the other states in a task. Therefore, I think you may need to take the expectation over all states in the state space (not just the next state). 

In any case, given these conceptual issues, I think the paper needs one more conceptual rewrite, and I will not change my score. 

 Thanks for the feedback. It is unclear what the policy for ICLR is, since the CallForPapers state that OpenReview generates a pdf diff with the changes.

> Now I understand that how including normal branch can be useful based on the additional qualitative results using masks, but the revised draft is still missing the comparison with the baseline of 'dynamics predictor with the same objective' that can show that the proposed architecture / objective is indeed important. 

We did not add this baseline since a forward model (only controlled branch) would model everything and not just what is controlled (which is what is trained for) and consequently, its score would be 0 for environments with dynamics.

> Sorry, but I can't understand what We chose Clusters due to having the largest difference to non-agent objects in Table 1 means. 

We mean that Clusters highlights the problem of controllability for non-agent objects.

> I can understand that you chose the task where the proposed method's strength can be highlighted, but this just feels like a bit of toy / motivating experiments. 

To highlight the complexity of these environments, PPO cannot solve the given task (even with extensive fine-tuning). Moreover it only learns to avoid spikes but not to put even a single box in place. On the other hand, PPO can achieve decent rewards in both Atari and DM Control Suite. We understand that comparing CEN in an environment like Atari would provide more context on its comparison to other intrinsic motivators. However, this comparison would not be fair nor realistic since CEN is a solution to a specific subproblem to the ones presented in Atari: controllability. Although we showcase CEN as an intrinsic motivator in section 4.3 experiments, CEN is not limited to this setting but a general method to identify what is controlled by the agent. Furthermore, it could be combined with most of these other methods or even used for other purposes like planning. Our experiments focus on showing that CEN can identify these controlled effects.

> why did you not evaluate your method on GridWorld tasks proposed in Camporo et al., 2020?

We chose to compare to the inverse model and ADM since they work on the controllability problem in an unsupervised setting. The method proposed in Campero et al. 2020 is quite different from this setting: 1) it is not unsupervised (uses the extrinsic reward) 2) uses the state of the environment instead of pixels 3) it is not designed to model controlled objects. The main reason not to use MiniGrid (the environment used in the paper) in our experiments is the limited visual requirements of the environment. Objects are limited to basic shapes (triangles, circles, etc.), whereas in our experiments Griddly and MZR use complex representations making the task more challenging and allowing us to showcase CEN's ability to generate accurate pixel masks for relatively complex objects.

We do evaluate CEN in five environments under three different settings to two state-of-the-art baselines.

 > I don't understand this reasoning. Unless all actions result in no change, surely action prediction would model the agent's position?...

The problem is not the number of actions but the action chosen by the agent and its effects. For example, if an agent has a wall on its left and chooses to move left, the inverse model cannot predict the action accurately since the actions "move left" and "do-nothing" lead to the same observation/effects. This is more problematic in corners where more actions lead to no effects. In general, action-prediction models cannot predict an action accurately when two or more actions lead to the same observation/effect.

This is not a fine-tuning issue but an inherent problem with the inverse model objective. Even when not using a random policy. Any stochastic policy would cause the inverse model not to predict the action accurately. This presents a problem for exploration as with intrinsic motivators, it is unreasonable to restrict these models to deterministic policies. In any case, the reason not to model the agent entirely (Table 1) is due to failing at unmovable objects and not because we did not fine-tune or implement the inverse model correctly.

> You've convinced me that this isn't a terribly useful metric, so why are you using it? Why not something aware of the screen geometry such that nearby classifications give partial credit?

Pixel F1 score is a standard and popular metric for image segmentation. Although it is not perfect, it is arguably the most accepted metric and provides a high-level view of how CEN compares to other models. In addition to the F1 score, we also provide qualitative results showing CEN's masks. Regarding taking into account nearby pixels, it is not the purpose of this work to develop a new metric that considers proximity to ground truth.

> I'm disappointed that many (most?) of my concerns weren't addressed in your response...

to our understanding, the reviewer provides two main problems:
1. Experiments that fail to provide evidence to support the paper's claims.
2. An unstated reliance on an unrealistic behavior policy also limits this approach's applicability.

With two suggested solutions:
1. reframing the paper
2. acknowledging the limitations (or correcting me if I'm mistaken) is enough.

To address 1) we did reframe the paper and removed the action-prediction claim to not modeling controlled non-agent objects since we do not provide enough evidence supporting this claim. For 2) We added this limitation to a new "limitations" section. Please do clarify if these changes are not the ones expected. We do realize that some of our answers may not be obviously linked to the reviewer's additional points:

> The comparisons to the ground truth masks are unproblematic from an evaluation standpoint -- I truly believe CEN dramatically outperforms ADM on this task...

To our knowledge, there is no definition of controllability that we can use to produce ground truth. Although one may be eager to mark the pixels for the agent in Montezuma as controlled, it is not true that all pixels are controlled since some pixels do not change or even if they change, they may not be controlled; for example, the agent is not controlled when falling after jumping. Thus, we use our proposed definition in Eq. 5 principled by measures of causality. Indeed, ADM was not trained to meet this definition; however, it was trained to model what is controlled. We believe pixel-level comparisons are valuable and provide insight into how CEN differs from ADM. Attribute-level experiments complement these results.

> (Section 4.2) I view Table 1 as not supporting the argument of the paper.

As mentioned before, we agree that Lights environment does not prove our claim that action-prediction models ignore non-agent objects and have amended that claim in the paper. However, Table 1 shows that CEN is better at modeling the agent in every environment than any baseline. It also shows that it is better than the baselines at modeling boxes for the Inverse model and boxes and lights for ADM.

> why are the decoder weights tied? 

There is no requirement to share weights. We did this to save memory and compute. 

 Thanks for the response. One minor suggestion is to highlight the modified parts of the draft for better clarity during the rebuttal period. 

- Now I understand that how including normal branch can be useful based on the additional qualitative results using masks, but the revised draft is still missing the comparison with the baseline of 'dynamics predictor with the same objective' that can show that the proposed architecture / objective is indeed important. 
- Sorry, but I can't understand what `We chose Clusters due to having the largest difference to non-agent objects in Table 1` means. 
- I can understand that you chose the task where the proposed method's strength can be highlighted, but this just feels like a bit of toy / motivating experiments. I still think the evaluation on standard benchmarks are necessary in order to evaluate how the proposed method works compared to prior works that also includes the other exploration method which are not focused on measuring the controllability, and in order to understand the limitation of the method. Furthermore, based on what you have commented, why did you not evaluate your method on GridWorld tasks proposed in Camporo et al., 2020?

Overall, my concern on the limited empirical evaluation still remains, especially on the limited evaluation setups which makes it difficult to compare the strength of the proposed method in the context of prior works.

 Thank you for your response, it addresses a few of my concerns, but requires more clarification:

>The inverse model, and action-prediction models in general, struggle to predict actions near unmovable objects (e.g. blocks, walls, and corners). This is because the effect of moving to a wall and staying still is the same, consequently, the model cannot differentiate between actions (this is more exaggerated the more actions map to the same effect). This is why, in general, our model performs better than ADM and Inverse model at modeling the agent. Consequently, the Inverse model cannot perfectly model the agent even with more tuning.

I don't understand this reasoning. Unless *all* actions result in no change, surely action prediction would model the agent's position? Could you unpack this: "this is more exaggerated the more actions map to the same effect"? I could see it as meaning something like 'as the number of actions go to infinity, the chance of sampling an action that does anything goes to zero'. But if that's the argument, the counter is that no reasonable should sample that many no-op actions! This feels either an implementational/tuning issue or a consequence of a random policy for training. 

>the F1 score is extremely sensitive for empty ground truth (no controlled effects), a single pixel marked as controlled takes the score from 1 to 0. 
You've convinced me that this isn't a terribly useful metric, so why are you using it? Why not something aware of the screen geometry such that nearby classifications give partial credit?

## unanswered inquiries
I'm disappointed that many (most?) of my concerns weren't addressed in your response. I know time is limited, but if there are too many problems to address, then perhaps its worth considering resubmitting after you've been able to address them.

My score remains unchanged unless these remaining issues are addressed.

 > **Sec 4.1: I do not get how you get the ground-truth masks exactly from the true state (you say ‘we use the internal environment state’, but how do you actually use this: to repeatedly sample the environment for all actions, or you manually code something?).**

This is explained in appendix A, under each environment suit there is a subsection "ground truth" explaining it. For MZR, we compute Eq. 5 directly by restarting the game to the current state and performing each action. Griddly does not provide this option so we compute the bounding boxes of controlled objects.

> **I find it a bit surprising that ‘the shape of lights and buttons’ (Sec 4.1.3) is so complex that performance decreases. For Montezuma’s I would buy this, but the lights environment is pretty straightforward right? Isn’t there something else going on? Did you investigate this?**

Thanks for the comment, we did find an error in Lights where CEN was only predicting one of the image channels. Due to the Lights' background, this was only a problem in Lights but not in the other environments. We have fixed it and rerun our experiments leading to performance more similar to other environments.
Regarding MZR, the quality of CEN's masks greatly depends on how powerful the decoder's CNN is. There are two main points to why MZR's score behaves differently:

* *F1 decreases over time:* the F1 score is extremely sensitive for empty ground truth (no controlled effects), a single pixel marked as controlled takes the score from 1 to 0. At the beginning of training, CEN produces empty masks correctly but performs poorly at controlled effects, which makes the F1 score artificially high. Over time it improves at controlled effects but starts marking few pixels as controlled for effects where nothing is controlled.

* *Lower F1 for MZR:* the sensitivity of the F1 score also affects CEN's performance on controlled effects since pixels outside the ground truth are penalized the same no matter how close to the ground truth they are. The masks in the appendix show that CEN typically fails by pixels close to the controlled object instead of identifying the wrong object as controlled. Even with a low F1 score, CEN identifies the controlled objects correctly. 
We have preferred to have the same network for every environment but prompted by the referee's question we have tried a more powerful network for MZR. Having a dedicated decoder (no parameter sharing) for each branch improves the quality of the masks leading to an F1 score of \~0.6 for non-empty effects where before was \~0.4. The F1 score for empty effects stays the same but this time the metric increases over time since it makes more mistakes at the beginning. Qualitatively the masks are sharper and closer to the agent. We believe that the score could be improved with more engineering of the CNN architecture.

We have added additional masks for MZR including typical failures of CEN for empty ground truth to the appendix. This discussion around MZR is also added to the main text.

> **Table 1: Why is CEN for example so good at predicting the Skull in MZR (61\%), for an object that is actually not controllable? This is weird right?**

The skull moves all the time except when the agent dies. In this case, CEN and the inverse model can predict the skull using information from the agent. If we do not take into account the skull when the agent dies, the F1 score goes below 36. We have updated the manuscript with this clarification.

 > **I think the main weakness is in the conceptual part (Sec 2.2), because I think there is a problem in Eq. 5, being the use of the expectation over all other actions. For example, I think it already fails for your example in Figure 1, middle. I assume there are three actions (left = health decreases, do nothing = health decreases, right = health stays the same). In this case, the ICE of action ‘right’ would be the biggest, since the other two actions (in the expectation) suggest that a decreasing health is actually normal. You mention in the paper that ‘keeping full health’ is normal in this case, but I do not understand how you get to that conclusion.**

The agent in Figure 1, middle, has five actions represented as arrows for moving and a circle for the do-nothing action. Three out of two actions lead to no changes in health levels making this the normative world for health. The main point here is that if we were to replace $\beta_Y$ from Eq. 3 with the do-nothing action, the decrease in health when performing do-nothing would be wrongly identified as not controlled.

This comment leads to an interesting question, how does the number of actions affect our definition of Blame (Eq. 5)? It does make sense to us that, for example, an agent with only one action is not to blame for what happens $e^a_p = e_n$ (even if this action is going to the bomb as in Fig 1).
Similarly, your example with three actions instead of five would lead to the opposite blame; the agent would not be to blame when health decreases. In this case, the agent is less "empowered" to not decrease its health, in other words, it has fewer options and thus, its blame would be higher.
We believe that the chosen normative world in Eq. 5 is not perfect. Although, this work explores a fixed rule to create a normative world, this does not need to be the case. More research in this direction needs to be done to explore how normality can be made more flexible.
We have added this discussion to a new "limitations" section.

> **Your practical experiments do work, which I think is caused by the fact that your neural network automatically generalizes over similar states. I think this is what needs to be corrected in Eq5: the expectation should probably also run over other states in the problem, not only over actions.**

We do not seem to fully understand this concern, maybe we can give a better answer if provided with more details. If the concern is regarding stochastic environments, we do agree that for Eq. 5 to work in stochastic domains the expectation should include possible next states. We have corrected this in the manuscript. Feel free to clarify the question if the answer does not go in the right direction.

> **I think the relation to Q(s,a) and V(s) in Eq 4 is also a bit stretched, since for a greedy policy, Q(s,a) = V(s). Now, it seems as if controllability depends on the type of policy you have, which should of course not be the case. In your Eq. 5, the policy does not seem to play a role anymore, and you simply weight each possible action equally? In any case, the jump from Eq 4 to 5 does not make sense, and I personally think Eq. 4 does not make sense in general (the dependence on your own policy).**

It is true that the advantage function (Eq. 4) computes its normative world using the policy and that our version (Eq. 5) does not.
Note that Blame, as in Eq. 3, does not impose any restriction on how the normal world is computed, thus it is up to us to define what normal world to use, it could be a constant.
We do agree that the advantage function is not computing the causal effect of an action on the full return but the relative effect the action would have on the return following a policy, i.e., what are the gains of performing action $a$ for a given policy in state $s$.
Ultimately, the causal effect of an action on the return still depends on the policy, otherwise it would only account for the effect on the reward at a single time step since the return depends on the policy. In other words, there would not be recurrence in V(s). We have clarified this point in Section 2.2. An interesting research question is to study the benefits (if any) of RL agents learning with a normal world similar to Eq. 5 instead of to Eq. 4. Learning flexible rules to approximate what is normal is certainly an active area of research in social sciences and psychology looking to underpin how humans judge in causal terms.

 > **1) experiments that fail to provide evidence to support the paper’s claims.**

Although the inverse model performs worse in Clusters at non-agent objects, we do agree that its performance in Lights for non-agent objects is similar to CEN. 
We have removed our assertion that action-prediction models do not model non-agent objects since our results do not provide enough evidence for this claim.

> **Figure 7b is clearly significant, but it is not clear that this is due to the properties of CEN highlighted throughout...**

* Table 1 shows that CEN is indeed better than the Inverse model at modeling boxes which is one of the main reasons CEN performs better as intrinsic motivator in Clusters. Of course, CEN rewards the agent for moving, which may lead to different exploratory behavior than the inverse model, but we cannot forget that the agent is also a controlled object of the environment.

* The inverse model, and action-prediction models in general, struggle to predict actions near unmovable objects (e.g. blocks, walls, and corners). This is because the effect of moving to a wall and staying still is the same, consequently, the model cannot differentiate between actions (this is more exaggerated the more actions map to the same effect). This is why, in general, our model performs better than ADM and Inverse model at modeling the agent. Consequently, the Inverse model cannot perfectly model the agent even with more tuning.

* To ensure the inverse model performs well, we did extensive hyperparameter tuning, exact ranges are in the appendix. We also checked that the action accuracy of the model was high; in the RL experiments was not below 95\% for most of the training (only at the beginning was lower). In fact, since we started implementing NGU with the inverse model it got more care from our side when making NGU work.

> **2) An unstated reliance on an unrealistic behavior policy also limits this approaches applicability.**

We concur that since Eq. 5 does not depend on the policy, CEN needs diverse data for each action to be able to approximate Eq. 5. This is an important problem not just to CEN but to any forward or world model. Probably even the representations of the inverse model will degrade using deterministic policies. This issue deserves more attention from our research community. We have added an explanation of this issue to a "limitations" section.
The RL experiments train CEN with the same PPO policy used to solve the task, not a random policy. We believe that PPO's entropy regularization helps to alleviate this issue and facilitates the use of CEN as intrinsic reward.

> **This is a smaller point, but what's going on with the Montezuma result in Figure 5b?*

* *F1 decreases over time:* the F1 score is extremely sensitive for empty ground truth (no controlled effects), a single pixel marked as controlled takes the score from 1 to 0. At the beginning of training, CEN produces empty masks correctly but performs poorly at controlled effects, which makes the F1 score artificially high. Over time it improves at controlled effects but starts marking few pixels as controlled for effects where nothing is controlled.

* *Lower F1 for MZR:* the sensitivity of the F1 score also affects CEN's performance on controlled effects since pixels outside the ground truth are penalized the same no matter how close to the ground truth they are. The masks in the appendix show that CEN typically fails by pixels close to the controlled object instead of identifying the wrong object as controlled. Even with a low F1 score, CEN identifies the controlled objects correctly. 
We have preferred to have the same network for every environment but prompted by the referee's question we have tried a more powerful network for MZR. Having a dedicated decoder for each branch improves the quality of the masks leading to an F1 score of \~0.6 for non-empty effects where before was \~0.4. We believe that the score could be improved with more engineering of the CNN architecture.

We have added additional masks for MZR including typical failures of CEN for empty ground truth to the appendix. This discussion around MZR is also added to the main text.

> **Why is Equation 6 a mixture loss instead of only trained the parameters of the action-agnostic stream with the second term and the parameters of the action-aware stream with the first time? Feels like you could eliminate a hyper-parameter without hurting performance.**

Each branch optimizing its own independent objective is equivalent to having a very large $\alpha$ since the normal branch would ignore the loss's first term. In this case, the normal branch would try to model controlled effects too, forcing the controlled branch to counter wrong predictions made by the normal branch.
In this work, we chose the expectation as the normative world but this is probably not perfect and more research is required to find more flexible measures of normality.

 > **Reason for the choice of a baseline is not properly provided in the paper.**

We do agree that a more self-contained description of ADM would help the reader. We now describe ADM and the reasons to choose it as one of the baselines in the paper.

> **Ablation study to show that the proposed component is indeed important for the improved performance is missing.**

We have included an ablation study for the $\alpha$ hyperparameter in the appendix, including masks from controlled and normal branches. The study shows that the larger the $\alpha$ the more work the controlled branch needs to perform leading to more errors for empty ground truth.

> **Missing comparative evaluation in terms of episode returns / success rates on standard benchmarks.**

We chose Clusters due to having the largest difference to non-agent objects in Table 1. Our goal is to show that in environments where controllability is important CEN surpasses action-prediction models like the inverse model in NGU.
Controllability may not be the most important factor as is the case in Super Mario or Atari Skiing. Arguably, most Atari games do not require much interaction with non-agent objects which would obfuscate our method with the environment demands.
As shown in Campero et al. 2020, popular exploration methods used in Atari do not perform well in grid worlds where the agent needs to interact with multiple objects. Consequently, we chose environments with two requirements 1) control is important and 2) they have an elaborated visual scene (not just simple shapes).

> **An easy solution for the network to solve the optimization problem might be to just ignore the normal branch and only utilize the controlled branch for prediction.**

The normal branch would be ignored if CEN was trained only with the typical reconstruction loss used by forward models. We explain how we avoid this collapse in Section 3. In essence, Eq. 6 includes an additional term that forces the normal branch to model the environment as much as possible. Masks from controlled and normal branches are now included in the appendix, showing that this collapse does not happen.

 We thank the reviewer for the time spent on the review and the feedback provided.

> **1) Can CEN scale to domains with more rich and complex observations?**

In general, the quality of CEN's masks greatly depends on how powerful the decoder's CNN is. As shown in our attribute-level experiments, CEN can disentangle controlled effects at attribute level remarkably well. In the following we explain the problems with the masks seen in MZR: 

* **F1 decreases over time:** the F1 score is extremely sensitive for empty ground truth (no controlled effects), a single-pixel marked as controlled takes the score from 1 to 0. At the beginning of training, CEN produces empty masks correctly but performs poorly at controlled effects, which makes the F1 score artificially high. Over time it improves at controlled effects but starts marking few pixels as controlled for effects where nothing is controlled.

* **Lower F1 for MZR:** the sensitivity of the F1 score also affects CEN's performance on controlled effects since pixels outside the ground truth are penalized the same no matter how close to the ground truth they are. The masks in the appendix show that CEN typically fails by pixels close to the controlled object instead of identifying the wrong object as controlled. Even with a low F1 score, CEN identifies the controlled objects correctly. 
We have preferred to have the same network for every environment but prompted by the referee's question we have tried a more powerful network for MZR. Having a dedicated decoder (no parameter sharing) for each branch improves the quality of the masks leading to an F1 score of \~0.6 for non-empty effects where before was \~0.4. The F1 score for empty effects stays the same but this time the metric increases over time since it makes more mistakes at the beginning. Qualitatively the masks are sharper and closer to the agent. We believe that the score could be improved with more engineering of the CNN architecture.
We have added additional masks for MZR including typical failures of CEN for empty ground truth to the appendix. This discussion around MZR is also added to the main text.

> **2) Can CEN handle first-person observations?**

This is an interesting scenario, note that our formulation of ICE uses perceived effects, $e_p$, as opposed to states. Thus, CEN explains changes in pixel space i.e. when CEN indicates that some change is controlled by the agent this is not (necessarily) equivalent to stating that the objects represented by those pixels are controlled.
For example, let us assume an agent is standing on a platform that can rotate. Assume the platform rotates at the same speed as the agent's turning speed. In other words, the agent would get the same input whether the platform rotates one step or the agent turns. Our formulation would identify the rotating platform case as NOT controlled, and the case of the agent turning as controlled since turning depends on the action but rotating does not.
An exciting avenue to explore is combining CEN with a state representation learning model, e.g., LSTM or Recurrent State-Space Model by Hafner et al. 2018, where changes would happen at state level instead of pixel level.
We have added this case to the paper.

> 3) **Does CEN improve exploration for more complex domains?**

We chose Clusters due to having the largest difference to non-agent objects in Table 1. Our goal is to show that in environments where controllability is important CEN surpasses action-prediction models like the inverse model in NGU.
Controllability may not be the most important factor as is the case in Super Mario or Atari Skiing. Arguably, most Atari games do not require much interaction with non-agent objects which would obfuscate our method with the environment demands.
Although Campero et al. 2020 does not use action-prediction models nor unsupervised learning, it demonstrates that popular exploration methods used in Atari do not perform well in grid worlds where the agent needs to interact with multiple objects. Consequently, we chose environments with two requirements 1) control is important and 2) they have an elaborated visual scene (not just simple shapes).

This paper proposes an approach, called Controlled Effect Network (CEN), for agents to predict the controlled parts of the environment. CEN is a dynamics model that separately predicts the "controlled" and "normal" change in observations, given the current observation and action. The controlled part is the change caused by the agent's action, whereas the normal/counterfactual part is what would have happened normally (e.g. this could be computed by averaging over the outcomes of all possible actions of the agent).

The empirical experiments show that CEN outperforms previous approaches for predicting the controlled part of the observation, on grid world tasks and Montezuma’s Revenge where the observations are images. In addition, it improves exploration when combined with NGU (Never Give Up) on a couple simple tasks, compared to the inverse dynamics model used in prior work. This paper makes a valuable contribution toward modeling controlled effects of agent behavior, which is important for causal RL and exploration—both recent hot topics in RL. The method is clearly motivated and well explained, and the paper is well-written. The experiments are presented in a logical order, and they systematically and thoroughly evaluate several important aspects of the algorithm.

In terms of improvement, I would like to see additional experiments for NGU + CEN, on more complex tasks. I was a bit disappointed that the only experiments on using CEN as intrinsic motivation for exploration were on an empty world and a grid world task.

In addition, I have a couple questions about the applicability of CEN. First, I am curious whether CEN would work for first-person cameras. Would it get confused by the entire observation / camera image changing as the agent’s gaze direction changes from frame to frame? If so, this a limitation that should be mentioned and discussed in terms of how it could be addressed.

I also am curious about the ability of CEN to scale to environments with more rich observations. For Montezuma’s Revenge, the pixel prediction of controlled effects is less accurate. The hypothesis is that this is because this Atari game has more complex visual scenes, compared to the grid worlds, which affects the ability to predict accurate masks. But this Atari game is still quite simple compared to real-world environments. Does this imply that CEN would not work for more realistic domains? CEN is a simple, elegant solution for disentangling controlled and normal effects, that is backed up by thorough empirical evaluation. However, it leaves a few open questions that I'm hoping the authors can answer in the rebuttal: 1) can CEN scale to domains with more rich and complex observations?, 2) can CEN handle first-person observations, and 3) does CEN improve exploration for more complex domains? Additional experiments and discussion on these areas would significantly strengthen the paper.

This paper studies a new method to identify controllable effects in a learned dynamics model of the environment. While previous methods have focused on inverse models (predicting the correct action from a given state and next state), the authors argue that this method fails to properly identify controllability. Instead, they come up with a new specification of ‘blame’, based on the difference between the observed effect of an action and the average/normal effect otherwise observed (under other action). Experiments show that their approach is better able to disentangle controllable effects than the baseline, and can also aid as an intrinsic motivation reward. Strong:
* I like the papers main insights about controllability (e.g. the examples in Figure 1), and that this requires a notion of ‘normal’ effects, from which controllable factors need to deviate. 
* The experiments are extensive and varied, and show good results. 

Issues with Sec 2:
* I think the main weakness is in the conceptual part (Sec 2.2), because I think there is a problem in Eq. 5, being the use of the expectation over all other actions. For example, I think it already fails for your example in Figure 1, middle. I assume there are three actions (left = health decreases, do nothing = health decreases, right = health stays the same). In this case, the ICE of action ‘right’ would be the biggest, since the other two actions (in the expectation) suggest that a decreasing health is actually normal. You mention in the paper that ‘keeping full health’ is normal in this case, but I do not understand how you get to that conclusion. 
* Your practical experiments do work, which I think is caused by the fact that your neural network automatically generalizes over similar states. I think this is what needs to be corrected in Eq5: the expectation should probably also run over other states in the problem, not only over actions.
* Moreover, I think the relation to Q(s,a) and V(s) in Eq 4 is also a bit stretched, since for a greedy policy, Q(s,a) = V(s). Now, it seems as if controllability depends on the type of policy you have, which should of course not be the case. In your Eq. 5, the policy does not seem to play a role anymore, and you simply weight each possible action equally? In any case, the jump from Eq 4 to 5 does not make sense, and I personally think Eq. 4 does not make sense in general (the dependence on your own policy). 

Smaller issues:
* Sec 4.1: I do not get how you get the ground-truth masks exactly from the true state (you say ‘we use the internal environment state’, but how do you actually use this: to repeatedly sample the environment for all actions, or you manually code something?). 
* Fig 6: I think this figure needs more labels, already on the rows and columns, but also on the masks. Dark means that the mask is active, i.e. controllable? So what should I see in the picture. Right now I found it hard to learn something from the figure. 
* I find it a bit surprising that ‘the shape of lights and buttons’ (Sec 4.1.3) is so complex that performance decreases. For Montezuma’s I would buy this, but the lights environment is pretty straightforward right? Isn’t there something else going on? Did you investigate this? 
* Sec 4.1.4: Results are in Fig 5b I suppose, not Fig 8?
* Table 1: Why is CEN for example so good at predicting the Skull in MZR (61%), for an object that is actually not controllable? This is weird right? 
* ‘Fig 7a shows how an inverse model does not’ - This sentence seems to be incomplete.  In general, I think this paper studies an interesting and relevant topic (controllable factors), and brings interesting new insights (inverse models do not work well, and there may be better methods). There are good conceptual illustrations of the proposed solution, but I do think there are some problems with the equations in Sec 2.2 that underpin their ideas. Their proposed network architecture and experiments do make sense, and show improved performance over inverse dynamics models. If the authors correct (or further elaborate on in the rebuttal) some parts of Sec 2, then I think the paper would be a good contribution to ICLR. 

Meta-review: This paper presents an attempt to infer controllable aspects of the environment dynamics by imposing an architecture on a pixel prediction model, such that the prediction is the sum of an action-aware and action-agnostic prediction, trained on a fixed dataset gathered under a uniform random policy, in such a way as to incentivize the action-aware predictor to model the residual of what the action-agnostic predictor is capable of predicting on its own. The utility of the method is demonstrated qualitatively and quantitatively by comparing both the effect-identification performance and performance when used in the context of intrinsically motivated RL.

Reviewers found the work to be well-motivated, highly relevant, clearly described and insightful. Several reviewers pointed to the relatively limited empirical evaluation, appropriateness of baselines, and questions as to how the method would apply to richer observation spaces. Reviewer hgSF, whose score improved by 2 points following discussion, in particular questioned the motivation for the baseline method (ADM) and asked for a hyperparameter ablation study which the authors provided. Reviewer Y2tc, whose score improved by 3 points, found that several experiments poorly supported the paper's claims and was concerned about the uniform policy requirement for training; after discussion, clarifications and a softening of some claims, the reviewer was satisfied. Reviewer fGPn raised concerns around metrics and questioned applicability to first person observations, and while the former were addressed to the reviewer's satisfaction, the latter were not, and the reviewer maintained their marginal accept-leaning appraisal. MeF5's biggest concern revolves around a conceptual issue around the expectation in equation 5 which judging by the discussion was clarified but not fully resolved.

The paper has improved significantly since submission through the diligent engagement of the reviewers. The method proposed is clever and effective in the domains considered but as fGPn points out is probably limited in applicability to observation spaces wherein non-controllable effects remain stationary, ruling out 3D locomotion or camera control for example. It is unclear whether this limitation is addressable, and the current version of the manuscript does not discuss this a perspective a great deal (in fact, direct mention of this  limitation is absent from the "limitations" section).

After discussion among the senior program committee, it was decided that the manuscript, while much improved by subsequent revisions, fell short of the acceptance bar this year, particularly in terms of rigorous evaluation of the method. We are therefore unable to recommend acceptance at this time.

Decision: Reject