Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a world model that can decompose controlled and uncontrolled factors in the environment. Concretely, their world model is composed of a stream that takes in actions as input and its state is encouraged to contain mostly controlled factors in the video by an inverse dynamics loss on the learned states; the stream that models uncontrolled factors does not take actions as input so that the environment is predicted purely on observations. The two streams as separated in the observation space by masks generated from each stream. In experiments the authors show that the proposed method outperforms other world models in a modified version of DMC and the CARLA driving environment. Finally, they also show the proposed method is able to perform action conditioned video prediction in the BAIR robot push and RobotNet datasets.
strengths_and_weaknesses: Strengths:
+ A world model that decomposes uncontrolled and controlled factors in the input observation.
+ Outperforms recent world models when the environment involves mostly controlled factors.


Weaknesses:
- Performance comparisons in similar battleground.
The authors clearly show an advantage of their model over the baselines in the setting the proposed model was designed for. For completeness, and to see whether the proposed model performs similar to the baselines, the authors may want to provide experiments in a setting where the baselines were tested. For example, the authors can just test their method on the non-modified DMC environment, and can provide results of this to show that proposed model is able to improve on the baselines on the settings in this paper, but doesn't make them much worse in the original settings the baselines were tested.


- Special treatment for different environments.
As far as I understand, the authors choose which version of the proposed world model to use depending on whether the uncontrolled dynamics serve a purpose for the environment (Lines 143-147). Should the model learn this without the need of being told/trained depending on the environment. To me, this takes from the generality of the model and could imply that the model is simply good in specific scenarios. Can the authors comment on this? Or provide results without special treatment?




- Video prediction setup, results and other baselines:
The video prediction setup in this paper is unusual. The authors deliberately put bouncing balls on top of the frames in order to show the proposed network has an advantage over other networks in the task of video prediction. I would suggest the authors directly test on the clean data and show the performance there. The model should be able to handle both the original and the modified version of the data.


There are also more up to date baselines missing from this comparison such as FitVid (https://arxiv.org/pdf/2106.13195.pdf)



Suggestions:

- Modeling relationships between current controlled state and future uncontrolled states.
In equation 7, the authors model relationships between the uncontrolled and controlled steps where the uncontrolled states are unrolled into the future to compute a representation that is used to make decisions on what action to take next. While this is sensible to provide the model a look into the future, the future also should consider what actions the agent may take. The uncontrolled environment also depends on what actions the controlled part of the environment take. For example, if a controlled car turns left, we should expect parts of the environment that are not seemingly controlled, to react to the left turn, and this may cause a chain reaction with all cars in the environment. Therefore, it seems more sensible to unroll both the controlled and uncontrolled states, and then equation 7 should be based on the result of this. If time permits, could the authors unroll both states together and see if this results in improvements? This is more of a suggestion out of curiosity and not a weakness of the model.
questions: Please see the weaknesses section.
limitations: There is no negative societal impact discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents Iso-dream, which uses a variant of RSSM as a world model that explicitly represents controllable and non-controllable aspects of the world state.  Because these factors are modeled independently, Iso-dream allows its policy to condition on samples of future non-controlled state rollouts from the model.  This method is experimentally evaluated on several benchmarks, some to evaluate the policy and world model together, and some to evaluate the performance of dynamics predictions.  In all benchmarks, Iso-dream outperforms examined baselines.

##Update after responses

I'd like to thank the authors for the clear responses, both through discussion and experimentation, to my questions and concerns.  I have raised my score.
strengths_and_weaknesses: ### Originality
The approach, to the best of my knowledge, is novel.  Some aspects of the separability of controllable and non-controllable states have been explored (InfoPower, as cited in the paper), and use of inverse dynamics losses are not new on their own.  However, the conditioning of the policy on future rollouts of the model is an interesting idea that I have not seen in this form before.  It is somewhat reminiscent of methods that incorporate planning into policy learning, and I would like to see the authors discuss the differences between this aspect of their approach and work like [Wang and Ba, Exploring Model-based Planning with Policy Networks, ICLR 2020].

### Quality & Clarity
The paper is well-written, technically sound, and mostly clear.  The diagrams are informative and help scaffold a quick understanding of Iso-dream. I'll list a few suggestions and questions that may aid in clarifying a few points, or which may be typos
- In line 130, the paper mentions a training collapse, but this collapse is not described
- Figure 5 contains a label that reads 'Truth futher'
- The image results in Figure 3 are somewhat small and difficult to make out

### Significance
The changes proposed in Iso-dream are incremental improvements to both the world model and policy learning of a prominent model-based RL approach.  These changes are evaluated in a thorough set of experiments.  This paper is a solid contribution.

questions: Some issues to help with my understanding, or for the authors to address:
- Comparison to prior work as mentioned above
- In the CARLA environment, agent actions have the potential to control all pixels (turning, or accelerating, would change the positions of other cars in the observations as well as the background).  Can you clarify what controllable and non-controllable aspects of the state would be in this scenario?  What training signal makes the non-controllable mask represent other cars, and is that signal brittle?
- When rolling out the state for conditioning the policy, is there a reason this cannot be done with controllable state as well?  I wonder how a baseline that used undifferentiated world state like RSSM, but conditioned its policy on imagined future rollouts would perform (and also how well Iso-dream would work without this future rollout conditioning)  I am curious how much the controllable vs non-controllable split contributes to the improvement in results compared to the conditioning and would love to see these additional experiments in the ablations in Fig 4b.  Since I see this as the main contribution of the paper, I'd like to see this explored a bit more.
- For the BAIR and RoboNet datasets, I'm curious whether the improved results for Iso-dream are attributable more to the action-free or action-dependent parts of the input.  Is Iso-dream doing better reconstructing the 'distractions' than other approaches but performing similarly for the 'relevant' parts of the observations? 

My score depends mostly on the fact that I think this is a solid, incremental contribution that is well evaluated but could be better.  My rating would be higher if my questions either result from some misunderstanding on my part or can be addressed in a response.
limitations: Computational limitations are mentioned, societal impact is not discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper expands on the Dreamer framework to explicitly model controllable vs non-controllable dynamics, leveraging the fact that controllable dynamics can be explained away using an inverse dynamics model, while non-controllable dynamics ought to be predictable with no action change.
strengths_and_weaknesses: Strengths:
- strong academic contribution: the approach is novel, well-motivated, and well supported by evidence.
- the problem of modeling controllable vs. non-controllable effects is at the center of the problem of learning visual dynamics model, and this general area of research is highly relevant to a number of robotics applications.

Weaknesses:
- the limitations section is short. It would have been interesting to expand on things attempted but that didn't work for instance.
questions: - Have the authors looked into how the factorization deals with 'partially controllable' dynamics? When driving, other cars have their own (uncontrollable) intents, but are also affected by one's driving decisions. I suspect the model would still largely treat them as entirely uncontrollable due to the assumption that inverse dynamics ought to be able to resolve them if they were controllable, but I don't have a strong intuition for the actual outcome. The question may be relevant to leveraging this model to build models of other agents in the scene, besides the main one, and using those for behavior prediction.
- I understand the used of GRUs and LSTMs follows previous papers on the topic, but since everyone seems to have moved on to Transformer-based architectures, it begs the question of whether replacing the backbones with that kind of architecture would further improve the model's efficiency?
- I may have missed this from the discussion, but is it well-established that the background branch is necessary now that controllable and non-controllable effects are modeled separately? I didn't see an ablation for that.

limitations: Limitations section is terse and is a non-limitation in disguise.
review_title: null

Meta-review: metareview: This paper studies the problem of building world models that can decouple controllable and uncontrollable factors in the environment. The paper received reviews that generally tended towards acceptance. However, the reviewers had difficulty understanding some details and had concerns that the setup might not be the same across environments. The authors provided a rebuttal that addressed most of the reviewers' concerns. The paper was discussed and all the reviewers updated their reviews in the post-rebuttal phase. Reviewers generally agree that the paper should be accepted. AC agrees with the reviewers and suggests acceptance. However, the authors are urged to look at reviewers' feedback and incorporate their comments into the camera-ready.
confidence: Certain
Decision: Accept