Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper studies the problem setting in which one has a "source" simulator with low-level state information in which one can efficiently train an agent to perform a task, and a "target" simulator that shares the initial state distribution and transition dynamics as the source simulator but only offers high-dimensional observations. The goal is to leverage trajectories sampled from a policy in the target domain to learn a mapping $q_\phi$ from the stream of high-dimensional observations in the target domain to states in the source domain. One can then perform the task in the target domain by feeding observations through $q_\phi$ and then through the source policy. The authors propose to use variational inference to train $q_\phi$ and thoroughly demonstrate what adaptation tasks their framework (CODAS) can and cannot perform on MuJoCo and Hand Manipulation. Impressively, using CODAS, the model in the target domain generally achieves 80-90% the performance of the model in the source domain.
main_review: I enjoyed reading this paper. The problem domain is largely very practical; the assumption in 125-127 that the source and target policies are identical is somewhat limiting in that in the practical setting, we have an optimal policy in simulation but no optimal policy in the real world target domain. However, the authors do a good job of investigating this limitation and also providing intuition on what makes the alignment problem difficult even when it is assumed that the source policy matches the target policy. The experimental results are impressive and the writing is very clear. The algorithm does seem to be somewhat unstable and possibly difficult to train (based on section 3.3) but the authors provide a comprehensive description of their implementation details in the paper and supplementary.

Some minor comments below:

- Figure 1 - this figure seems to suggest that the "shape" of the 3-point trajectory is preserved from one domain to another, and that additional structure is what makes the problem easier. However, trajectory shapes in one domain do not necessarily correspond to the same shape in another domain. Is this interpretation and criticism of the figure correct?
- Figure 7 - is it also worth considering ablating the importance of the initial state distributions matching?
- 98 - how well is $q_\phi$ able to extract the correct state from an observation quantitatively? A training curve would be interesting to see. Figure 5 is just qualitative.
- 230-231 - what is the intuition for image->action requiring a higher sample complexity to learn than image->state?
- 245-247 - is there some suggestion for how one could fine-tune either the source policy or $q_\phi$ to bump the performance from 85% to 100%?
- 306-307 - is it not easy to collect a larger dataset and test this reasoning?
limitations_and_societal_impact: The authors do a good job of discussing the importance of the assumption in the derivation of CODAS that the dynamics and data collection policies match between source and target. They also offer an example of a task for which CODAS performs poorly (relocate in Table 1) and claim the reason is the "goal-conditional nature" of the task.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduces an interesting work that applies UDA technique to connect the visual gap between the real and simulation domain in RL simulation tasks. An inference function is learned with variational inference to transfer the policy learned from the simulator instance to a realistic instance.

main_review: + This basic idea of this paper is easy to follow.
+ This paper shows an easy solution to reduce the visual gap between real and simulate environments when solving the RL tasks. The idea to transfer status and policy from status domain to visual domain with domain adaptation is interesting.
 
 
This paper shows some interesting ideas and novelty. But it still contains some uncleared descriptions and concerns. There are a few questions or problems as listed below:
 
	1. L135 mentioned that related works learn the mapping function by minimizing the divergence between distributions. What related works exactly here are talking about? Is there any comparison of these works in your experiments? 
	2. L146 What is the Gaussian distribution here descript? The structure of q() and the output distribution should be clarified. 
	3. L158 The difference of ELBO loss with VAEGAN should be more specific list here.
	4. The definition of D in Eq4. is unclear. 
	5. L178 What are the structure of the DM and embedded DM model? This is not clear. 
 
Why name your method CODAS instead of CMDAS ?

limitations_and_societal_impact: This paper has limited impact on the related community since similar works have been explored.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper provides a variational formulation to train a observation-to-state mapping, as a potential solution to cross domain transfer. The proposed approach is evaluated in MuJoCo control and hand manipulation tasks, showing its advantage over GAN-based approaches.
main_review: Strength
- The method shows empirical improvement over baselines across a range of control tasks. 
- The study of changes in environmental dynamics is very interesting and shows that the proposed method can tolerate some degree of dynamic mismatch.
- While the baselines are rather naive, the paper addresses an interesting problem and proposes a feasible solution. 

Weakness
- The method relies on the ability to reset the simulator and query the oracle one-step transition, while the GAN baselines do not have this information. It strongly favors the proposed method. Stacking the images could already improve vanilla GAN significantly, and what if transitional information is also taken into accounts? Requiring the ability to reset could also be a problem when scaling to real robots. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper purpose to solve sim to real problem from a new viewpoint. Instead of using image or feature level domain adaptation, the authors use sequential information（trajectories) to adapt the image to state, and define a differential and stabilize object for domain adaptation in reinforcement learning, and results perform better than other baselines in all experiments.
main_review: Strength:
+ This paper has a solid theoretical derivation for the objective of domain adaptation in RL using Variational Inference.
+ Lots of experiment is conducted to show the effect of this method including baseline, an experiment in a complex task, ablation study, mapping state error, etc.

Weakness:
- Without an image, the agent cannot know the geometry of itself or object which can be hard to train a task need to know the shape (for example grasp for different objects).
- For a simulation environment that has a simple and unrealistic image, it still needs 600 episodes (minimum 100 steps each episodes) data to train the mapping function, which means need more data for the true sim to real.
- This method is proposed to solve sim2real, no experiment show this method can map a real image to the corresponding state, even for a demonstration.

Detail comments：
The main motivation of this paper is for general domain adaptation in sim2real under the same modal. It has a mismatching problem and is costly, so this paper purpose to solve the problem by cross-modal domain adaptation. 
For the method, there is some previous work also try to adapt between image and state. For example, in [1], the authors adapt image and state to hidden state and use that to train the policy.  [2] also uses sequential information to unsupervised adapt the policy. It would be better to show the advantage compare with these methods.
For the “cost” problem, the main idea of sim2real is to use a simulator to reduce the cost of real data collection, for this method although it reduces the cost in simulation, it seems to require more complex real data collection than general domain adaptation(trajectory need action). Besides, this paper does not show any demonstration in real-word scenario.  Therefore, it is hard to evaluate whether this method still remains low-cost in real-world scenarios. From this point, it would be better to conduct some experiments in real world or evaluate the performance of this method with different amounts of target data episodes to show some potential for low real data requirement.
For baselines, there is also another domain adaptation method that can be used for cross-modal domain adaptation, such as[1] and [3], it would be better also use this kind of method as stronger baseline.

[1] Wilson, Matthew, and Tucker Hermans. "Learning to manipulate object collections using grounded state representations." Conference on Robot Learning. PMLR, 2020.

[2] Hansen, Nicklas, et al. "Self-supervised policy adaptation during deployment." arXiv preprint arXiv:2007.04309 (2020).

[3] Tzeng, Eric, et al. "Adversarial discriminative domain adaptation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

limitations_and_societal_impact: The authors have addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: This paper tackles the problem of visual-input sim-to-real and tries to overcome the reality gap between images rendered in simulators and those from the real world. It presents an approach that, instead of building and running simulators that render costly high-quality images, generates only low-dimensional states in simulation to learn the policy, and then learns encoders from and decoders to observations that handle sequential observations. This enables to train and transfer state dynamics from a source environment to a target environment with only high-dimensional observations. It then proposes a mathematical derivation for the cross-modal unsupervised domain adaptation problem. The method is then evaluated on MuJoCo tasks (OpenAI Gym and robot hand manipulation) and compared to several GAN, CycleGAN and temporally stacked GAN baselines, where it shows better domain adaptation and transfer performance.

Reviewer NDFM praised the paper and their minor comments were addressed. Reviewer cBxr’s most negative comment was about the limited impact of the work due to similarity with existing work, but the authors contest this review and justify their contribution as a combination of mathematical derivation of the problem as variational inference and the addition of sequential modeling of the dynamics (as opposed to VAE-GAN). Reviewer 6QJy complained about missing real-world data evaluations to make this work relevant (a claim disputed by the authors and reviewer NDFM), as well as missing comparisons to two recent sim2real techniques that “adapt image and state to hidden state and use that to train the policy'' and that modeled sequences of states from observations (the authors provide a clarification).

Review scores are (5, 5, 6, 8, average 6). Reviewers cBxr (score 5) did not respond to the rebuttal and did not update their scores but I believe they could promote their score. While waiting on this reviewer, I am therefore willing to challenge them and promote this paper to an acceptance.

Decision: Accept (Poster)