Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an approach, called Controlled Effect Network (CEN), for agents to predict the controlled parts of the environment. CEN is a dynamics model that separately predicts the "controlled" and "normal" change in observations, given the current observation and action. The controlled part is the change caused by the agent's action, whereas the normal/counterfactual part is what would have happened normally (e.g. this could be computed by averaging over the outcomes of all possible actions of the agent).

The empirical experiments show that CEN outperforms previous approaches for predicting the controlled part of the observation, on grid world tasks and Montezuma’s Revenge where the observations are images. In addition, it improves exploration when combined with NGU (Never Give Up) on a couple simple tasks, compared to the inverse dynamics model used in prior work.
main_review: This paper makes a valuable contribution toward modeling controlled effects of agent behavior, which is important for causal RL and exploration—both recent hot topics in RL. The method is clearly motivated and well explained, and the paper is well-written. The experiments are presented in a logical order, and they systematically and thoroughly evaluate several important aspects of the algorithm.

In terms of improvement, I would like to see additional experiments for NGU + CEN, on more complex tasks. I was a bit disappointed that the only experiments on using CEN as intrinsic motivation for exploration were on an empty world and a grid world task.

In addition, I have a couple questions about the applicability of CEN. First, I am curious whether CEN would work for first-person cameras. Would it get confused by the entire observation / camera image changing as the agent’s gaze direction changes from frame to frame? If so, this a limitation that should be mentioned and discussed in terms of how it could be addressed.

I also am curious about the ability of CEN to scale to environments with more rich observations. For Montezuma’s Revenge, the pixel prediction of controlled effects is less accurate. The hypothesis is that this is because this Atari game has more complex visual scenes, compared to the grid worlds, which affects the ability to predict accurate masks. But this Atari game is still quite simple compared to real-world environments. Does this imply that CEN would not work for more realistic domains?
summary_of_the_review: CEN is a simple, elegant solution for disentangling controlled and normal effects, that is backed up by thorough empirical evaluation. However, it leaves a few open questions that I'm hoping the authors can answer in the rebuttal: 1) can CEN scale to domains with more rich and complex observations?, 2) can CEN handle first-person observations, and 3) does CEN improve exploration for more complex domains? Additional experiments and discussion on these areas would significantly strengthen the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to identify the controlled effects from actions based on the concept of 'blame'. The proposed network consists of two branches that conditions on both current observation and action and only current observation for predicting the difference of two consecutive observations, respectively. By introducing these two branches, representations from the controlled branch can capture the controlled effects from actions which cannot be adequately captured by normal branch. The proposed method is evaluated on several benchmarks to predict the controlled effects obtained by utilizing internal states of simulators, but performance of the proposed method as an intrinsic motivator is only evaluated on a single task.
main_review: Strengths
- Simple method based on nice intuition
- Clear writing and presentation

Weaknesses
- Limited empirical evaluation

Detailed comments
- Reason for the choice of a baseline is not properly provided in the paper. First of all, what is Attentive Dynamics Model? How does it learn representations? Which characteristic of ADM makes you consider it as a baseline? What's the expected message from the experiments comparing the proposed method to ADM? Why not just learn the representations from inverse dynamics and learn to predict the difference between observations using such representations? Just stating that '`ADM produces an attention mask for pixels controlled by the agent therefore it is used as baseline in these experiments.` does not convey meaningful information and makes paper not self-contained.
- Ablation study to show that the proposed component is indeed important for the improved performance is missing. How does the method compare to using representations from a dynamics predictor without normal branch (objective should be the same by predicting the difference between observations)? How does the performance changes, and predictions differ by changing $\alpha$?
- Missing comparative evaluation in terms of episode returns / success rates on standard benchmarks. Such experiments are only done in Clusters environments, and not clear why the performance is only reported only on that task. More investigations on hard-exploration games in Atari and DeepMind Control Suite are required for demonstrating that the proposed method is indeed better than inverse dynamics model for learning representations useful for capturing the novelty of states without affected by random aspects of the observations.
- In many cases, an easy solution for the network to solve the optimization problem might be to just ignore the normal branch and only utilize the controlled branch for prediction. Why do you think the method can work without collapsing into such case ignoring a normal branch? How do the predictions from controlled and normal branch differ qualitatively? (if I understand correctly, it should be possible to visualize the outputs from the decoders of both controlled and normal branches). How does the method perform on such environments, and compare to baseline of inverse dynamics predictor?
summary_of_the_review: I like the general idea of the paper, but several empirical evaluations are missing to support / justify the effectiveness of the proposed method. Hence my evaluation remains closer to the rejection of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors' introduce the Controlled Effects Network (CEN), an approach to separating the controlled aspects of the environmental dynamics from those that are agnostic to the agent's actions. This occurs on a per-pixel basis. In order to accomplish this separation, the network architecture forces the next step prediction to be the sum of two separate predictions (only one of which has access to the agent's action). To prevent the action-aware stream from simply modelling everything, a mixture loss is used the cares explicitly about the accuracy of the action-agnostic stream's predictions in addition to that of the aggregate. Training occurs on a fixed dataset collected with a uniform random policy, which is important since the best action-agnostic prediction is highly dependent upon the behavior policy. The utility of this approach is demonstrated in 3 ways: comparisons to the ground masks of controlled effects, decodability of attribute changes in controlled effects (and not un-controlled effects), and RL performance when used as the representation for an intrinsic motivation system.
main_review: This paper introduces a representation-learning system, CEN, that is both conceptually simple and well-motivated from the perspective of causal learning. The writing (up until the implementation and experimental details) was very clear and remarkably self-contained compared to other work at the intersection of causality and RL. However, I cannot currently endorse this paper's acceptance due to several significant issues with the experimental results, and (to a less extent) some methodological limitations that deserve discussion. As per the paper (and my summary), there are 3 sets of results and I'll go through my issues with them sequentially.

(Section 4.1) The comparisons to the ground truth masks are unproblematic from an evaluation standpoint -- I truly believe CEN dramatically outperforms ADM on this task. But the significance of this is far less than the other experiments. The "ground truth" mask of controllable elements is *defined* to be a function of the difference between the average (across all actions) change in pixels and the change in pixels for the particular action actually taken. This is exactly what CEN attempts to model, and quite unlike what previous approaches consider. It is thus unsurprising that CEN wins out on this metric. Indeed, the utility of matching these masks is what needs to be shown; these results are a necessary sanity check, but can't stand alone without the other results. It should reframed in this light. This is a smaller point, but what's going on with the Montezuma result in Figure 5b? Why is the performance of CEN going down over time?

(Section 4.2) I view Table 1 as not supporting the argument of the paper. The main results to my mind is that CEN is better than alternatives at modelling the agent. If that was the goal of CEN then fine, but it is not. The agent is the aspect of the environment that the authors claim to modelled by all of these methods and emphatically not the thing they're trying to improve upon. Both ADM and Inv model box, button, and light (the indirectly controllable objects) at least as well as they model the agent. Indeed, Inv actually matches CEN on button and light, despite being significantly worse on agent.

(Section 4.3) Figure 7a is not the clean uniform vs corner-seeking distinction that it is written up to be. Early in training CEN appears to be as drawn to corners as Inv, and even at the end there is a significant bias towards edges that isn't explained by your account. Figure 7b is clearly significant, but it is not clear that this is due to the properties of CEN highlighted throughout. Table 1 shows that CEN outperforms Inv on modelling the agent, so perhaps that is the cause of the performance gain? And this might be down to a matter of tuning, since Inv was never trained on this environment before (at least in the NGU paper) -- why shouldn't agent modelling performance be maxed out here? And would the performance gap be eliminated if this were the case? I believe that it would.

Unstated limitations. The reliance on training under the uniform random policy is a big issue e.g. preventing the usage of CEN to improve the NGU paper's results. The theoretical motivation doesn't have anything to say as to why this behavior policy is needed, but the action-agnostic predictions are highly reliant on the policy, which would continuously change if CEN was trained online. Even training from a fixed dataset of expert trajectories would be problematic e.g. experts near-deterministically kill enemies, therefore enemies are no longer controllable under your formulation (since their deaths are predictable without action information). Perhaps importance sampling could correct for this, but that is an empirical question that needs answering before CEN can be said to be scalable.

A pair of small related concerns: why are the decoder weights tied? Why is Equation 6 a mixture loss instead of only trained the parameters of the action-agnostic stream with the second term and the parameters of the action-aware stream with the first time? Feels like you could eliminate a hyper-parameter without hurting performance.
summary_of_the_review: A clear and well motivated algorithm is let down by (1) experiments that fail to provide evidence to support the paper's claims. (2) An unstated reliance on an unrealistic behavior policy also limits this approaches applicability. In its current state, I must therefor recommend rejection. That said, I could definitely see accepting this with some significant revisions. To address (1), the experimental evidence must better match the motivation of the approach. You could perform new experiments that show CEN models more than just the agent whereas other approaches don't. You might also reconsider your framing; perhaps the agent/object distinction isn't relevant here and CEN should be better for a different reason.  To address (2), acknowledging the limitations (or correcting me if I'm mistaken) is enough. A first step towards a promising new direction is enough for acceptance; no need to solve everything in one paper, but being explicit about the state of things is a must.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies a new method to identify controllable effects in a learned dynamics model of the environment. While previous methods have focused on inverse models (predicting the correct action from a given state and next state), the authors argue that this method fails to properly identify controllability. Instead, they come up with a new specification of ‘blame’, based on the difference between the observed effect of an action and the average/normal effect otherwise observed (under other action). Experiments show that their approach is better able to disentangle controllable effects than the baseline, and can also aid as an intrinsic motivation reward.
main_review: Strong:
* I like the papers main insights about controllability (e.g. the examples in Figure 1), and that this requires a notion of ‘normal’ effects, from which controllable factors need to deviate. 
* The experiments are extensive and varied, and show good results. 

Issues with Sec 2:
* I think the main weakness is in the conceptual part (Sec 2.2), because I think there is a problem in Eq. 5, being the use of the expectation over all other actions. For example, I think it already fails for your example in Figure 1, middle. I assume there are three actions (left = health decreases, do nothing = health decreases, right = health stays the same). In this case, the ICE of action ‘right’ would be the biggest, since the other two actions (in the expectation) suggest that a decreasing health is actually normal. You mention in the paper that ‘keeping full health’ is normal in this case, but I do not understand how you get to that conclusion. 
* Your practical experiments do work, which I think is caused by the fact that your neural network automatically generalizes over similar states. I think this is what needs to be corrected in Eq5: the expectation should probably also run over other states in the problem, not only over actions.
* Moreover, I think the relation to Q(s,a) and V(s) in Eq 4 is also a bit stretched, since for a greedy policy, Q(s,a) = V(s). Now, it seems as if controllability depends on the type of policy you have, which should of course not be the case. In your Eq. 5, the policy does not seem to play a role anymore, and you simply weight each possible action equally? In any case, the jump from Eq 4 to 5 does not make sense, and I personally think Eq. 4 does not make sense in general (the dependence on your own policy). 

Smaller issues:
* Sec 4.1: I do not get how you get the ground-truth masks exactly from the true state (you say ‘we use the internal environment state’, but how do you actually use this: to repeatedly sample the environment for all actions, or you manually code something?). 
* Fig 6: I think this figure needs more labels, already on the rows and columns, but also on the masks. Dark means that the mask is active, i.e. controllable? So what should I see in the picture. Right now I found it hard to learn something from the figure. 
* I find it a bit surprising that ‘the shape of lights and buttons’ (Sec 4.1.3) is so complex that performance decreases. For Montezuma’s I would buy this, but the lights environment is pretty straightforward right? Isn’t there something else going on? Did you investigate this? 
* Sec 4.1.4: Results are in Fig 5b I suppose, not Fig 8?
* Table 1: Why is CEN for example so good at predicting the Skull in MZR (61%), for an object that is actually not controllable? This is weird right? 
* ‘Fig 7a shows how an inverse model does not’ - This sentence seems to be incomplete. 
summary_of_the_review: In general, I think this paper studies an interesting and relevant topic (controllable factors), and brings interesting new insights (inverse models do not work well, and there may be better methods). There are good conceptual illustrations of the proposed solution, but I do think there are some problems with the equations in Sec 2.2 that underpin their ideas. Their proposed network architecture and experiments do make sense, and show improved performance over inverse dynamics models. If the authors correct (or further elaborate on in the rebuttal) some parts of Sec 2, then I think the paper would be a good contribution to ICLR.
review_title: null

Meta-review: This paper presents an attempt to infer controllable aspects of the environment dynamics by imposing an architecture on a pixel prediction model, such that the prediction is the sum of an action-aware and action-agnostic prediction, trained on a fixed dataset gathered under a uniform random policy, in such a way as to incentivize the action-aware predictor to model the residual of what the action-agnostic predictor is capable of predicting on its own. The utility of the method is demonstrated qualitatively and quantitatively by comparing both the effect-identification performance and performance when used in the context of intrinsically motivated RL.

Reviewers found the work to be well-motivated, highly relevant, clearly described and insightful. Several reviewers pointed to the relatively limited empirical evaluation, appropriateness of baselines, and questions as to how the method would apply to richer observation spaces. Reviewer hgSF, whose score improved by 2 points following discussion, in particular questioned the motivation for the baseline method (ADM) and asked for a hyperparameter ablation study which the authors provided. Reviewer Y2tc, whose score improved by 3 points, found that several experiments poorly supported the paper's claims and was concerned about the uniform policy requirement for training; after discussion, clarifications and a softening of some claims, the reviewer was satisfied. Reviewer fGPn raised concerns around metrics and questioned applicability to first person observations, and while the former were addressed to the reviewer's satisfaction, the latter were not, and the reviewer maintained their marginal accept-leaning appraisal. MeF5's biggest concern revolves around a conceptual issue around the expectation in equation 5 which judging by the discussion was clarified but not fully resolved.

The paper has improved significantly since submission through the diligent engagement of the reviewers. The method proposed is clever and effective in the domains considered but as fGPn points out is probably limited in applicability to observation spaces wherein non-controllable effects remain stationary, ruling out 3D locomotion or camera control for example. It is unclear whether this limitation is addressable, and the current version of the manuscript does not discuss this a perspective a great deal (in fact, direct mention of this  limitation is absent from the "limitations" section).

After discussion among the senior program committee, it was decided that the manuscript, while much improved by subsequent revisions, fell short of the acceptance bar this year, particularly in terms of rigorous evaluation of the method. We are therefore unable to recommend acceptance at this time.
Decision: Reject