Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to leverage models of the environment and rewards as regularizers to perform explicit transfer between different observation spaces in RL.

Being able to quickly adapt to different observations is quite a different problem setting than what the rest of the literature is tackling (and we could argue on how relevant it is, but let’s leave that aside). They provide some theoretical analysis of this setting, especially one interesting result on when their method can converge to a near-optimal solution through approximate policy iteration.

They tackle it using ideas very similar to the recent wave of work leveraging bisimulation metrics, hence novelty / comparison on this aspect will be important.
They assess it on simple Mujoco control tasks, where their results are promising but perhaps still a bit early.
main_review: Overall, I found this work interesting but slightly lacking in focus, and especially the theoretical parts are slightly divorced from the effective implementation. The empirical work is also quite similar to previous work, despite the rather different problem setting considered.

Here are additional comments and questions which would be worth clarifying:

1. The problem considered is quite novel and I feel could lead to novel applications and approaches. However, in my opinion this would be more interesting if only the environment dynamics were considered to be shared between the source and target task, and when action spaces would differ. In its current instantiation, this restricts this to very particular situations.
   1. A more generic solution would be to consider MDP homomorphism, something that was tackled by van der Pol et al, 2020.
   2. This paper isn’t cited and discussed and is quite clearly relevant to this current work, alongside more recent works from Amy Zhang.
   3. Some other works in bisimulation metrics and behavioral similarity might also deserve to be contrasted and discussed.
2. I liked how Section 4.1 explicitly spells out that a representation needs to support the policy improvement path and not just the fixed point. However, I am not yet fully convinced of how novel Definition 3 and Lemma 4 really are. The proofs appear quite similar in construction to many existing results in the (recent) bisimulation literature, and the end of this section makes statements which can be related to the Value Equivalence Principle from Grimm et al 2020. Similarly, the assumption of linearity brings the setting closer to successor features and their more novel versions, something that has been thoroughly explored over the years.
   1. Again, I’m looking to understand how novel and different the current work is compared to previous work which aren’t explicitly discussed in this draft.
3. The loss used in practice is not that related to the theoretical derivations, and hence the paper isn’t as focused as it could be. 
   1. In effect, L_P and L_R are rather standard in the model-based literature (e.g. they are used exactly as is in Van der Pol 2020), so the only novelty is the constraint to still use the same frozen Transition and Reward models in the target task (which is interesting).
4. Figure 3 presents an interesting setting (transfering from joints to pixel observations), and the method does help.
   1. However, Figure a and b do not appear to have converged for the “Pixel Obs, Without Transfer” condition. I would not be comfortable drawing conclusions given the current curves.
   2. This is also performed on Cartpole, which is rather simplistic.
5. I wished Figure 4 was also covering a similar setting (joints -> pixels), instead of the rather arbitrary addition of extra joints. 
   1. Results are also not as promising, no setting apart from HalfCheetah can convincingly demonstrate the benefit of the method.
6. I found Section D.2 in the Appendix quite interesting, especially the ablation about transferring P or R was something that I was about to ask about, so I wished it was more discussed. Recent work indicates that matching R in bisimulation metrics is prone to overfitting, so it’s interesting that these results do seem to indicate that it is still helpful to capture it.


References:
   * Van der Pol 2020: https://arxiv.org/abs/2002.11963 
   * Zhang 2020, https://arxiv.org/abs/2003.06016 
   * Castro 2020, https://arxiv.org/abs/1911.09291 
   * Grimm 2020, https://arxiv.org/abs/2011.03506 
   * Also slightly related: https://arxiv.org/abs/1911.12247 , https://arxiv.org/abs/2107.11676 


summary_of_the_review: In conclusion, I found this paper interesting, and tackling a rather novel problem setting. However, it currently is quite theory centric albeit without contrasting the novelty of their analysis compared to previous works as much as I’d like it to.
In addition, the empirical instantiation is rather divorced from the analysis, and is not novel compared to the large body of literature assessing model-based regularization and representation learning.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the problem of observation space change for RL problems, i.e., the observation changes while the dynamics remain similar. To deal with this problem, the authors propose to learn a latent dynamics models based on encoded observations. When the observation space is changed, the learnt latent dynamics model can serve as a knowledge prior to transfer known dynamics information when learning with the new observation space. 


main_review: ## Pros
* The novel problem formation is well motivated from real-life applications.
* The structural similarity assumption 1 made for the problem setting is reasonable and the proposed solution elegantly solves the problem by leveraging the structural similarity. 

## Cons/questions/suggestions
* __Comparisons that demonstrate the practical benefits when observation quality is improved are missing__ The example of upgrading sensors to improve observation quality is a great motivating example for the problem of transfer RL between different observation spaces. For some RL applications, by reducing the noise or augmenting the observations, one might be able to learn a better policy than the policy trained on the original representation. Based on this understanding, I think an experiment that demonstrates the usefulness of the proposed algorithm in this setting would greatly strengthen the current work. Specifically, one could study the transfer between pixel inputs or vector inputs with different levels of background noise or image resolution. The knowledge transferred from poorer observations should enable an efficient learning when the observation quality is improved (_which is shown in Fig. 4_), and the asymptotic performance of the policy under better representations should improve upon the poorer representation's corresponding policy (_this is missing_). If such results can be included, I believe readers can better appreciate the importance of this novel problem, because it might allow one to improve observation qualities to get better policies, without having to train from scratch on the new observations. 
* __Details on observation encoder initialization__ For vector-image transfer, the encoder architectures would be different and the latent output $z$ could shift a lot for the same underlying state under the vector observation and the image observation. Would this large latent vector shift lead to unstable training? Does one need to control how these encoders are initialized? 

## Minor comments
* The authors claim that the proposed method can make it easier to learn a good representation from the new observation space. The results in figure 3 and 4 indirectly support the authors' claim. I wonder whether it would be possible to directly compare the representation learning progress with/without the latent dynamics transfer. For example, use the learned representations for some auxiliary tasks such as clarification/prediction/clustering and compare the performance.

summary_of_the_review: The novel problem setting proposed by the authors is of practical interests and the authors's proposed solution is well justified. Though the proposed method does not provide significant benefit when it's hard to learn the transition dynamics, the novel problem formulation could inspire future work along this line to develop new methods that can achieve better transfer for more challenging environments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of adapting a policy to a novel representation of the environment. Different from most previous work the paper focuses on the problem of a change in the representation of the observation provided by the environment, i.e., assuming that the underlying dynamics P and R remains mostly similar. The idea is to learn source and target models of the environment jointly with the policy. This serves as a model-based regularization that guides policy and representation towards the learned models. This effectively leads to a learned representation that is shared for the policy for the source and the target environment representation.
main_review: I generally like the idea that the paper follows and its theoretical rigorousness. However, I have a few concerns about the exposition of the paper as well as its empirical evidence. Also, I think the number of real-world applications this solution addresses is very limited, making it a very narrow field of application.

While I especially like the insights to non-linear policy heads at the end of Sec. 4, I suggest shortening Secs. 4.1 and 4.3 and reduce them to their general insights (putting the rest into the appendix). This makes more space available for the presentation and discussion of additional experimental results.


*** Motivation of the paper and real-world application. ***
At points I was confused by the motivation of the paper. This started in the very beginning with “the observation space is specified by human developers and restricted by physical realizations and may thus be subject to dramatic changes” – I wonder in which real-world use-cases this really is the case (I mean – without considering simulation environments etc.). Even more: switching from vector-based observation to image-based observation sounds interesting but might finally be a fictional problem setup. 

A much more intuitive scenario would be a sensor fusion setup where you have different sensors that measure different physical properties but which, in their sum, are redundant. If you remove a sensor (breakage?) the policy becomes useless as it expects a different input encoding while still (as the authors propose) the dynamics remains the same.


*** Related work ***
-	The papers argues that it is not easy to scale up the work of Sun et al (2020) for continuous domains. However, while not originally being proposed it seems more or less straight-forward? This is also lined out in the original paper. Hence, this work might also serve as a baseline
-	The paper argues that the work proposed by Raiman et al (2019) does not work for drastic changes. I cannot completely agree. The originally proposed gradient mapping could also be applied to the embedding where there is a sufficient representation. This work could hence also serve as a baseline. Even if it fails for the ‘drastic change’ it might work on the proposed Mujoco experiments.


*** Experimental Results ***

Although the authors claim that there is no previous work to which they can compare their algorithm to, I have a few suggestions/remarks:
-	(see lsao before) In the introduction you mention previous work of Raiman et al. (2019) who add or remove individual observation features. This is exactly what the authors do in their Mujoco experiments. Hence, this work serves as a baseline to which the authors should compare their approach to
-	Also an option would be to not starting to train the policy from scratch but to fine-tune the policy on the new observation representation. At least for the Mujoco setup this should be a viable starting point.
-	As the authors effectively propose a model-based RL algorithm/extension they should also run their algos against MBRL baselines.

Questions on DQN & Cartpole:
-	What exactly has been used in DQN? Dueling Double DQN with prioritized experience replay and noise? What about exploration parameters? As the results are not clearly showing the approach to outperform I wonder how well it would perform under perfect DRL algorithm configuration….
-	Presenting a screenshot of the cartpole to the agent forms a POMDP as we do not get information on the angular velocity and the velocity of the pole’s tip – which are both the most important components. I wonder why the authors formulate the experiment like this. Or did you stack frames to ensure Markovian state representation?

Questions on SAC & Mujoco/3DBall:
-	In my opinion, the observation representation change in the Mujoco environments is too simplistic. There are only features stacked on top of the existing features. Here, I expect a fine tuning of the policy whily freezing anything but the early layers should outperform both approaches, right?
-	The 3DBall modification is much more challenging and effectively also resembles the motivation of the paper. However, the results are not interpreted at all? Moreover, the results seem not convincing. It looks like both perform en par while the “without-transfer” approach does not reach the final performance. (Or does it towards the end???) There is too few information on the task and environment to interpret the results….

In general, there is a lack of discussion of the results of the approach and its limitations. It would also be interesting to see an ablation study that shows how much the representation effectively needs to change so that the proposed approach beats a fine-tuning of the policy. Also: what is the motivation of using DQNs in the first experiment and SAC in the last? I have the feeling that in some/many combinations of DRL agents and environments the proposed method also might lack behind and retraining of the policy…

Minor remarks:
-	Typo in the introduction: automotive[-s-]
-	The concept of the target networks (p5 bottom) is confusing at first as it refers to target/evaluation nets known from DQN. At this point the readers first thinks that the paper talks about the network that encodes target environment dynamics.

summary_of_the_review: I generally like the idea of the paper but in my opinion the paper is not ready yet for publication in ICLR. The main reason for that is the lack of experimental evidence and lack of comparison to previous work exposed to ablated variants of the proposed problem setting. Moreover, the application of the methods seems quite narrow given the assumptions of drastic representation changes during evolvement of environments.

*** update of the review ***

I would like to thank the authors for their thorough submission update. The other reviews also share some of my criticism and the authors imho did a very good job updating their work. Exposition and motivation are much clearer know the the experiments have improved considerably.

Bottom line: an increase of my score is definitely justified and I will increase my score to 5. The reason while I am still not leaning towards acceptance is because I still have concerns:
The fine-tuning experiment is not what I actually meant. In the experiments only the policy head is kept but I suggested to keep on training with the new representation (on the cases where the observation just gets bigger; just add the missing neurons at the beginning and initialise them). This should be faster/better. 
And against such a baseline, I am still missing an ablation study that answers the following question: How much different must a representation be such that the transfer approach is better?

P.S. there are two typos:
- feature is corresponding [to] which old observation feature.
- Page 8, last line: 'Fine-tine' instead of 'fine-tune'
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers the problem of RL environments where the observations space can change dramatically, e.g. from proprioceptive to pixel observations. The authors introduce various definitions for this setting and provide proofs regarding convergence (in the sense of Q-values converging to true values). Over 5 environments in continuous control, the authors demonstrate that their method enables transfer learning from proprioceptive to pixel observations and that such transfer learning can be beneficial.
main_review: Overall, the paper is well-written, although some technical points are unclear. The problem that the authors consider is interesting and relatively novel. Specifically, the setting is close to many previously studied settings, but not identical to the best of my knowledge. The paper combines several theoretical results with empirical evaluation. There are three main issues with the paper: motivation, technical clarity, and experimental evaluation. 

## Motivation

The setting the authors consider, transferring between observation spaces, has not been tried before to the best of my knowledge. The authors claim that it is an important problem, but I do not find the motivation for this very persuasive. The authors state that “Observation change is common in practice due to hardware upgrading, data restriction or curriculum design.”. Could specific citations be provided here? Figure 1 shows an example where observation spaces could change, but it seems to be an abstract toy example. It would be better to use a real-world example here. Maybe sim-to-real is a better example since simulated environments can expose data structures for the simulator, which real environments cannot. 

Overall, I think the sim-to-real problem is the setup closest to what the authors are trying to achieve. I would encourage the authors to discuss sim-to-real more, and maybe use some baselines from there. See e.g. ​​Sim-to-Real: Learning Agile Locomotion For Quadruped Robots.


## Technical Clarity

The role of the approximation operator is a little unclear. What does this operator model? Does it just mean the best Q-value estimates available under given features? The paper would be easier to read if a motivating example was given when it was introduced. 

Overall, the theoretical arguments appear to be somewhat circular. The authors define a representation mapping to be sufficient if it enables accurate Q-values. It is then shown that if a representation mapping is sufficient, one can obtain accurate q-values. I have a hard time assessing what the theoretical results really prove. 

Is it possible to measure if some representation mapping is sufficient? If not, the definition might not be useful in practice.

Also, do the theoretical results apply to the empirical setting considered later? 

For the experiment section, the setup details are relegated to Appendix D. There, however, there are not enough details to really understand what is going on. E.g., what is the size of the screen, and what is the size of the crop? What parameters, beyond learning rate, are used for adam? What settings are used for the replay buffer? For continuous control, are the actions discretized in any way to enable an argmax policy?

## Experimental evaluation
		
The authors state “We implement a DQN learner”. There are many high-quality DQN implementations available online, and doing your own implementation can introduce bugs. It is not clear that the DQN implementation works well, and if it does not, improving upon it is not very meaningful.

In Figure 4, results on tasks hopper and walker do not appear to be statistically significant. 

How are the tasks selected? How are the architectures selected? How are the hyperparameters selected? 

For latent space models, there are many excellent implementations available online e.g. dreamerV2. Why not use one of these?

Why are the proprioceptive states of the environments modified? E.g. in Appendix D, the authors write: “For the target task of MuJoCo environments, we add the center of the mass based inertia and velocity into the observations of the agent”.



## Post rebuttal update

I thank the authors for the many clarifications and improvements to the paper! I want to retain my score of 5 for two reasons. Firstly, I still think that the empirical results are not strong enough. In Figure 3, it seems like the improvements are only significantly outside the error bars for two tasks (Vec-to-pixel: Cheetah-Run and More-sensor: HalfCheetah). Secondly, while the authors have improved the motivation by giving relevant real-world scenarios where their methods apply, the experimental evaluation is pretty far away from these examples. Thus, while their methods improve performance for these toy tasks, I am not convinced their methods will improve a real-world problem. One specific direction to pursue here might be that of this paper (https://arxiv.org/abs/1912.12294). 





summary_of_the_review: While the paper often is well-written and considers an interesting problem, the motivation for the problem is lacking. Additionally, lack of technical clarity on both the empirical and theoretical sides makes it hard to assess what the paper really proves. At last, the empirical evaluation is problematic -- some results are not statistically meaningful, many parameter choices are not motivated and it is not clear that the baseline DQN implementation works well in the first place.
review_title: null

Meta-review: This work suggests using models of the environment as regularizers for performing explicit transfer in RL. Here are some of the highlights from the reviews and subsequent discussions:
  * Novel problem 
  * Unclear to some of the reviewers why the problem setting is in fact important.
  * Well-written
  * Interesting theoretical results
  * Somewhat limited experimental results
Post-rebuttal, while there is not necessarily a great consensus, the reviewers all feel that it's an improved piece of work. While I am myself not fully convinced that the problem setting motivation truly aligns with the kind of empirical results that the work provides, on the balance I think this work is interesting and has sufficient novel contributions to be accepted at ICLR.
Decision: Accept (Poster)