Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents an approach for unsupervised learning of a goal-conditioned policy that is robust to changes in dynamics and can be transferred to a target domain that is different from the source domain in terms of the underlying dynamics. There are several components. First, a latent-conditioned probing policy is used to generate goals (either in the source or target domain). This probing policy is trained to generate diverse goals by maximising the information gain between the goals and the resulting trajectories generated by the policy, i.e. we want diverse goals that are predictable given the resulting trajectories. This “predictability” is approximated using a discriminator that scores the likelihood of the goal latent given the resulting trajectory; this is also trained alongside the goal-generating policy based on rollouts in the source domain. 

Next, the probing policy is used to generate goals in the source and target domains (these are either the latents themselves or the final states of trajectories from the probing policy), and the goal-conditioned policy (which is conditioned on these goals) is executed in both these domains to generate rollouts. A key assumption here is that it is costly to execute the policy in the target domain (e.g. real world), so the approach tries to minimise this by training the probing policy, discriminator and to a major extent, the policy, primarily on the source domain. The policy is again trained to maximise the log-likelihood of the discriminator, thereby leading to predictable trajectories in the source domain. To ensure that the policy does not overfit to the source domain, and consequently fail on the target domain, an additional KL regularisation term is added to the policy training objective. This regularisation penalises trajectories that provide inconsistent results between the source and target domains, as measured by the KL divergence of the resulting transition distributions when executing the policy for the same goal in both these domains. This effectively ensures that the policy learns trajectories that are executable consistently in both domains, thereby ensuring success even in the presence of differences in dynamics between the two domains. Note that two key assumptions that are necessary for this to work (and the theoretical results) is that there is no transition in the target domain that is not possible in the source domain, and that the goal distribution in the source is also valid in the target (a relaxation of the latter is provided in the supplementary material). This regularisation term is approximated by a pair of classifiers that are trained on data from both domains; these are also trained alongside the policy. The final algorithm alternates training the probing policy & discriminator that parameterise the goal distribution and rewards against training the goal-conditioned policy and the classifiers for approximating the dynamics difference KL.

The approach is tested on several simulated tasks of varying complexity. First, the approach is tested on a 2D point mass with pairs of source and target tasks where the arena has walls of different configurations. Next, the approach is evaluated on continuous control tasks from the OpenAI Gym with the ant, half-cheetah and humanoid embodiments. The source tasks are the standard gym tasks and the target tasks have certain joints not working or additional perturbations in the form of obstacles. Lastly, the approach is also tested in a sim2real setup with a quadruped; the source task is in simulation while the target task is on the real robot. The resulting policy and goal distribution show good performance on all these tasks; the skills and goal distribution learned by the method is able to successfully transfer across changes in wall configuration, joint issues, stability problems and also to real world robots. Several visualisations of different parts of the algorithm (e.g. discriminator, skills) etc. are provided along with comparisons to baselines, namely variants of DARC, SMiRL, DIAYN and GPIM. The supplement also provides additional analysis and proofs of the theoretical guarantees of the near-optimality of the proposed method under the pertinent assumptions.
main_review: This paper presents an approach to unsupervised RL that trains a goal-conditioned policy which is capable of adapting to changes in dynamics between a source and target domain. A key advantage of the method is that it is able to train in an unsupervised fashion in target domains where data collection can be hard or expensive. This is done by training the goal distribution, goal-based reward function and goal-conditioned policy primarily on the source domain where data collection is inexpensive, while regularising for differences in dynamics between the two domains. This regularisation encourages the policy to find skills that lead to consistent results in the source and target domains while utilising an order of magnitude lower amount of data from the target domain. This is particularly promising for real world robotics, especially in the context of sim2real methods, where simulated data is easy and cheap to generate but has differences in dynamics to the real world where data is also expensive to generate. 

The approach is tested on several simulated and real world source and transfer tasks and shows promising results. There are several assumptions that are key for the method to work. Key amongst those is the assumption that there is no transition that is possible in the target domain but impossible in the source domain; interestingly, one of the point-mass experiments does violate this assumption and while the approach does work on this transfer setting it effectively over-constrains the search space leading to a potentially sub-optimal solution. It would be interesting to see if this assumption can be relaxed. Another important assumption is that the goal distribution is consistent under dynamics changes between the source and target domains; an approach to relax this assumption is derived in the supplementary material wherein the goal distribution training can be regularised to respect the source/target dynamics changes and is validated with a set of experiments (it would be useful in this context to see (Map-a, Map-e) discriminator visualisations in Figure 6). 

Overall, I am quite encouraged by the results presented in the paper, which are strong and show that the method is able to do well on a variety of control tasks both in simulation and in the real world. I am particularly excited about the applications to sim2real settings where the data limitations can make learning quite hard. It would be interesting to see if further reductions in the data is possible, and it would be great if there was an ablation study with respect to the number of real world data points used (i.e. fraction of target domain data) on some of these tasks. To add further clarity to the paper it would be useful if the approach for training the classifiers to estimate the dynamics regularisation is also detailed either in the main text or supplementary material. 
limitations_and_societal_impact: The authors have clearly explained the limitations of the approach, explaining in detail the assumptions that are needed to get the approach working (and for the theoretical results), the resulting restrictions on the method and potential relaxations for some of the assumptions with experimental results for these relaxations. Brief discussions of the broader impact is provided in the supplement, though some more details could be provided with regards to the social impact (especially since the approach is applicable for sim2real).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors address the problem of unsupervised Domain Adaptation in Reinforcement Learning. Combining ideas from previous work, they devise a method able to utilize an easy to sample source environment (e.g., a simulator) in order to learn a set of skills without supervision, while at the same time guarantee that these skills will be suited to the target environment (e.g., a real robotic system). To achieve this, they modify the reward structure of the typical Unsupervised Reinforcement Learning setup to enable the agent to build skills that are compatible with the target environment dynamics, while penalizing exploration in regions of the source environment where the dynamics differ.
The proposed approach is benchmarked against state-of-the-art algorithms in a wide variety of environments with increased complexity and manages to outperform previous baselines both in simulation as well as in three walking tasks on a real quadruped robot.

main_review: The authors address the problem of unsupervised Domain Adaptation in Reinforcement Learning. Combining ideas from previous work, they devise a method able to utilize an easy to sample source environment (e.g., a simulator) in order to learn a set of skills without supervision, while at the same time guarantee that these skills will be suited to the target environment (e.g., a real robotic system). To achieve this, they modify the reward structure of the typical Unsupervised Reinforcement Learning setup to enable the agent to build skills that are compatible with the target environment dynamics, while penalizing exploration in regions of the source environment where the dynamics differ.
The proposed approach is benchmarked against state-of-the-art algorithms in a wide variety of environments with increased complexity and manages to outperform previous baselines both in simulation as well as in three walking tasks on a real quadruped robot.

limitations_and_societal_impact: The first part of the paper starts with a very clear and well-written introduction in Sections 1 and 2 that introduces the problem and the proposed approach. Figures 1 and 2 provide an excellent overview and help understand the intuition behind the setup and the proposed algorithm. This holds for Sections 3.1 and 3.2 (aided by the visual explanation in Figure 3), where the problem is described in detail along with the derivation of the modified objective in Equation 4.
Unfortunately, Sections 3.3 – 3.5 which describe the details of the proposed approach are difficult to follow, as the authors omit critical information. The definition and purpose of the discriminator q_\phi is never explained (or discussed). The same goes for the two classifiers q_\psi: they are introduced directly in Algorithm 1 and referenced in one line in Section 3.5. Also, it is not clear by reading the text how they are used to shape the reward \Delta r. If the reader follows the references and reads the relevant papers, these design choices become clear, but the paper should be made more self-contained.
I would recommend the authors to bring Figure 10 of Section F.1. of the supplementary material in the main text and use it to explain what is the role of the discriminator, how the classifiers are trained and how they are used to calculated \Delta r. 
The experiments Section is quite dense and difficult to follow at some points. For example, I would recommend moving the comparisons shown in Figures 8 and 9 in the appendix and expand the discussion there a bit.

Pros:
- Very well written for the most part (see discussion on problematic Sections above)
- Figures providing intuition on the proposed method, as well as visualization of skills (both trajectories and heatmaps) are very well thought and complement the text in an excellent way
- Very extensive experimental evaluation and impressive results on real robot

Limitations:
- One of the main limitations is that the approach requires full knowledge and a-priori access to the target environment, while at the same time assuming that the goal distribution is the same between the two. Nevertheless, the authors discuss these limitations in Appendices B and D providing insights and extensions to the algorithm.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper handles the problem of domain adaptation in RL, specifically the setting where rewards and prior goal representations are not available. They do so by combining a skill learning method (in the source environment) with a transfer learning method that aims to match the joint $(goal, \tau)$ distribution arcoss environments. By doing so they show that skills can both be learned in the source domain, and they transfer more easily to the target domain.
main_review: The authors want to use domain adaptation to learn a policy that will perform well in a target environment, assuming access to a source environment where it is easier to get many samples. In addition, the authors want a method that will do so without access to rewards in either the source or target environment.

They propose a method called DARS, standing for dynamics-aware rewards. It is based off 2 primary components. First, there is a skill learning policy. For this policy, they use a common trick in the literature: an exploratory policy $\pi$ is conditioned on some latent skill vector $\omega$. As trajectories are generated, a learned $q_\phi(\omega|s)$ acts as a discriminator that tries to identify $\omega$ from the episode. Intuitively, if the mutual information between $\omega$ and episode $\tau$ is maximized, then different $\omega$ should lead to distinguishable trajectories in the environment. (The specific citations for this are DIAYN and GPIM but something analogous to this has shown up in other recent papers as well.)

They then combine this with a dynamics based reward. The aim is to match distributions $(goal, \tau)$ across the source environment and target environment, given the same policy $\pi$. Through some derivation, you can show that when taking the KL divergence $p(g,\tau)$, all terms between the two cancel out except for $p_{source}(s_{t+1}|s_t,a_t)$ and $p_{target}(s_{t+1}|s_t,a_t)$. The ratio of these probabilities can then further be approximated by learning 2 classifiers $p(target|s_t,a_t,s_{t+1})$ and $p(target|s_t,a_t)$. This closely follows the Domain Adaptation with Rewards from Classifiers (DARC) paper.

Combined, they demonstrate that skills learned in the source domain transfer better to the target domain.

I found the presentation in this paper a bit dense, although that is to be expected since they are aiming to explain 2 papers worth of ideas in 1. At a surface level, I personally found it confusing that $\Delta r$ was defined opposite to how it was defined in the DARC paper (i.e as (source - target)), and I initially thought this was an error until I noticed that it was subtracted instead of added to the reward. I think negating it and treating $\Delta r$ as an additive reward bonus is slightly more intuitive.

This is a fairly minor issue, my more detailed reservation with the paper is that it doesn't feel like a very interesting research contribution. To me, it feels like it is primarily about taking DARC then replacing the source rewards with a skill learning method. The 2 working together doesn't seem novel to me because they are somewhat orthogonal methods (one is skill learning in 1 env and the other is transfer learning between 2 envs). Both are based on defining rewards for an RL method to optimize. A priori I would assume the two would work together as long as you balanced their reward objectives appropriately, and this is basically what the paper shows (adjusting the $\beta$ weight is what affects whether the method works or not). Moreover it doesn't seem like their comparisons are very strong - they are essentially comparing against a skill learning / intrinstic motivation (SMiRL, GPIM), or they are comparing against a transfer learning method (DARC with a proxy reward), and again it seems like the method is supposed to work if tuned properly. And it does, by a margin that appears small.

To me, if it a priori is supposed to work, and the improvement is fairly small, then I'm not sure the paper needs to exist. This is especially true because the target problem they are solving feels narrow, as it is the intersection of 2 other problems. They aim to solve "transfer learning, between 2 environments, where we have ready access to 1 and not the other, but we also don't have rewards in the source env we have ready access to", and this doesn't feel like a common scenario of interest, which weakens significance.

Without an especially strong empirical result, to be strong enough for acceptance the paper needs some argument that the combination of the two is better than the sum of its parts. It attempts to argue this by saying that the dynamics from the target domain are affecting the skill learning process (in the Map env visualizations). I think this is basically just a consequence of "weighting the objectives correctly" - in the limit, at $\beta = 0$ you recover DIAYN and at $\beta = \infty$ you recover DARC, so at some $\beta$ in between you expect to see episodes that 1) are closer to the target episodes but 2) still have diversity within that restricted part of state space, and that's effectively what we see here.

Edit: I have read the other reviews and rebuttals. Please see the comment later in the thread for more detailed replies. In short,

* I agree that $\beta$ does not inherently limit diversity of skills, it is primarily about alignment in the target domain. This may reduce diversity of states that are reached in the source domain, but this is fine because the end goal is to learn diverse skills for the target domain - anything that is closed off by the $KL$ objective is something we should not learn anyways.
* One of my comparisons was comparing an unsupervised (no observed reward) method against a supervised (observed reward) method, which was a mistake on my part.
* After some more consideration, I believe the problem the paper aims to solve is more important than I previously thought.

These were significant misunderstandings so I am updating from 4 to 7.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The reviewers highly appreciated the author response and the additional details. Reviewer UYK4 didn't update the review but indicated in the discussion to have read the replies.
We had quite an extensive discussion between the reviewers on whether the setting of this paper too narrow and whether that would actually speak against publishing it. In the end they came up with a few examples (please add some motivating examples that have practical relevance to the paper) and everybody was convinced that this is a sound and interesting paper.
Decision: Accept (Poster)