Title: Third Person Imitation Learning
Authors: Bradly C Stadie
Pieter Abbeel
Ilya Sutskever
Venue: ICLR 2017

Abstract: Reinforcement learning (RL) makes it possible to train agents capable of achieving
sophisticated goals in complex and uncertain environments. A key difficulty in
reinforcement learning is specifying a reward function for the agent to optimize.
Traditionally, imitation learning in RL has been used to overcome this problem.
Unfortunately, hitherto imitation learning methods tend to require that demonstrations
are supplied in the first-person: the agent is provided with a sequence of
states and a specification of the actions that it should have taken. While powerful,
this kind of imitation learning is limited by the relatively hard problem of collecting
first-person demonstrations. Humans address this problem by learning from
third-person demonstrations: they observe other humans perform tasks, infer the
task, and accomplish the same task themselves.
In this paper, we present a method for unsupervised third-person imitation learning.
Here third-person refers to training an agent to correctly achieve a simple
goal in a simple environment when it is provided a demonstration of a teacher
achieving the same goal but from a different viewpoint; and unsupervised refers
to the fact that the agent receives only these third-person demonstrations, and is
not provided a correspondence between teacher states and student states. Our
methods primary insight is that recent advances from domain confusion can be
utilized to yield domain agnostic features which are crucial during the training
process. To validate our approach, we report successful experiments on learning
from third-person demonstrations in a pointmass domain, a reacher domain, and
inverted pendulum.

TL;DR: Agent watches another agent at a different camera angle completing the task and learns via raw pixels how to imitate. 

Review discussion:
In this paper, I didn't see how the expert (teacher) data is acquired. Is this demonstrated from human or optimized (given the known physical parameters)?
Also, there is no detailed specification of network architecture (e.g., what layers did you use? How complicated is D_F, D_R and D_D) in the paper besides Fig. 2. Please elaborate.
Question: How is the expert data generated?
Answer: The expert data is generated via an RL algorithm (TRPO) that has access to full state information and reward signal.
In fact, this RL algorithm is just the out of the box TRPO from RLLab being used to solve the reacher, point, and pendulum environments from RLLab.
Question: Also, there is no detailed specification of network architecture (e.g., what layers did you use? How complicated is D_F, D_R and D_D) in the paper besides Fig. 2. Please elaborate.
Answer: The joint feature extractor is 2 convolutional layers with 5 filters of size 3 each. Each layer is followed by a max-polling layer with size 2. The input images are size 50 x 50 with 3 channels, RGB.
Both the domain classifier and the class discriminator take as input the domain-agnostic image features and pass them through 2 MLP layers of size 128 before going through the final MLP layer of size 2 and a softmax layer. We have clarified this in the updated paper in Appendix A.
In addition, if any further architecture questions should arise, the discriminator and generator code is available here:
https://github.com/bstadie/third_person_im/blob/master/sandbox/bradly/third_person/discriminators/discriminator.py
On p.5 you choose to instantiate the mutual information by introducing another classifier.  Is this common? Can you add a reference?  Or is it novel?  What are the impacts of this choice?
If I understand correctly, the imitation task is reduced to predicting whether the demonstrations come from Expert or Non-expert.
Then the label is associated with a (known/stored) reward function, and that reward function is used in TRPO.  Is that correct?  It seems that is a lot of information, compared to the standard imitation learning setup, where the policy must be fully learned from demonstrations.  Am I misunderstanding something?
With so many terms being simultaneously optimized, it is not so surprising that it is difficult to get stable learning. How do you optimize hyper-parameters in this setting?   Presumably you cannot use a hold-out set.  Fig.6 & 7 show some sensitivity analysis, but nothing is shown about the learning rates.
I would have expected to see comparison to the following methods added to Figure 3:
1)  Standard 1st person imitation learning using agent A data, and apply the policy on agent A.  This is an upper-bound on how well you can expect to do, since you have the correct perspective.
2)  Standard 1st person imitation learning using agent A data, then apply the policy on agent B.  Here, I expect it might do less well than 3rd person learning, but worth checking to be sure, and showing what is the gap in performance.
3)  Reinforcement learning using agent A data, and apply the policy on agent A.  I expect this might do better than 3rd person imitation learning but it might depend on the scenario (e.g. difficulty of imitation vs exploration; how different are the points of view between the agents). It would be nice to show this.
Question: On p.5 you choose to instantiate the mutual information by introducing another classifier.  Is this common? Can you add a reference?  Or is it novel?  What are the impacts of this choice?
Answer: This is not novel. A similar idea is proposed in for instance
J. S. Bridle, A. J. Heading, and D. J. MacKay, “Unsupervised classifiers, mutual information and ’phantom targets’,” in NIPS, 1992.
D. Barber and F. V. Agakov, “Kernelized infomax clustering,” in NIPS, 2005, pp. 17–24.
A. Krause, P. Perona, and R. G. Gomes, “Discriminative clustering by regularized information maximization,” in NIPS, 2010, pp. 775–783.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. CoRR, abs/1606.03657, 2016.
We have updated the paper to explicitly mention these references in section 5.1.
Question: If I understand correctly, the imitation task is reduced to predicting whether the demonstrations come from Expert or Non-expert.
Then the label is associated with a (known/stored) reward function, and that reward function is used in TRPO.  Is that correct?  It seems that is a lot of information, compared to the standard imitation learning setup, where the policy must be fully learned from demonstrations.  Am I misunderstanding something?
Answer: This is incorrect. TRPO receives as a reward the (learned) log probability that a set of demonstration frames belongs to the expert class, and attempts to maximize this number. In doing so, it is trying to learn a policy that generates more frames that look like they belong to the expert class. TRPO never has access to the raw reward of the system. This problem setup is identical to that presented by Ho in Generative Adversarial Imitation Learning. Line 29 of Algorithm 1 makes this explicit. It is also shown and specified in figure 2.
Question: With so many terms being simultaneously optimized, it is not so surprising that it is difficult to get stable learning. How do you optimize hyper-parameters in this setting?   Presumably you cannot use a hold-out set.  Fig.6 & 7 show some sensitivity analysis, but nothing is shown about the learning rates.
Answer: Hyperparameters were optimized with a standard hold-out procedure, where the same hyperparameter choices were required across all problem instances.  For the discriminator, we use ADAM and a learning rate of 0.001. This worked out of the box. For the RL generator, we used the standard TRPO implementation from RLLab and made no changes.
Question: I would have expected to see comparison to the following methods added to Figure 3:
1)  Standard 1st person imitation learning using agent A data, and apply the policy on agent A.  This is an upper-bound on how well you can expect to do, since you have the correct perspective.
2)  Standard 1st person imitation learning using agent A data, then apply the policy on agent B.  Here, I expect it might do less well than 3rd person learning, but worth checking to be sure, and showing what is the gap in performance.
3)  Reinforcement learning using agent A data, and apply the policy on agent A.  I expect this might do better than 3rd person imitation learning but it might depend on the scenario (e.g. difficulty of imitation vs exploration; how different are the points of view between the agents). It would be nice to show this.
Answer: 1) and 2) are great suggestions for further calibrating the results, thank you, we plan to incorporate them in the next revision of the paper.
3) is actually already how the expert demonstration data is generated.
The paper extends the imitation learning paradigm to the case where the demonstrator and learner have different points of view. This is an important contribution, with several good applications.  The main insight is to use adversarial training to learn a policy that is robust to this difference in perspective.  This problem formulation is quite novel compared to the standard imitation learning literature (usually first-order perspective), though has close links to the literature on transfer learning (as explained in Sec.2).
The basic approach is clearly explained, and follows quite readily from recent literature on imitation learning and adversarial training.
I would have expected to see comparison to the following methods added to Figure 3:
1)  Standard 1st person imitation learning using agent A data, and apply the policy on agent A.  This is an upper-bound on how well you can expect to do, since you have the correct perspective.
2)  Standard 1st person imitation learning using agent A data, then apply the policy on agent B.  Here, I expect it might do less well than 3rd person learning, but worth checking to be sure, and showing what is the gap in performance.
3)  Reinforcement learning using agent A data, and apply the policy on agent A.  I expect this might do better than 3rd person imitation learning but it might depend on the scenario (e.g. difficulty of imitation vs exploration; how different are the points of view between the agents). I understand this is how the expert data is collected for the demonstrator, but I don’t see the performance results from just using this procedure on the learner (to compare to Fig.3 results).
Including these results would in my view significantly enhance the impact of the paper.
Question:
I would have expected to see comparison to the following methods added to Figure 3:
1)  Standard 1st person imitation learning using agent A data, and apply the policy on agent A.  This is an upper-bound on how well you can expect to do, since you have the correct perspective.
2)  Standard 1st person imitation learning using agent A data, then apply the policy on agent B.  Here, I expect it might do less well than 3rd person learning, but worth checking to be sure, and showing what is the gap in performance.
3)  Reinforcement learning using agent A data, and apply the policy on agent A.  I expect this might do better than 3rd person imitation learning but it might depend on the scenario (e.g. difficulty of imitation vs exploration; how different are the points of view between the agents). I understand this is how the expert data is collected for the demonstrator, but I don’t see the performance results from just using this procedure on the learner (to compare to Fig.3 results).
Answer:
Thank you for suggesting these experiment. We have added all of them to Appendix A. We feel like this suggestion significantly improved the quality of the paper. Please see earlier responses/the paper itself for a discussion of these experiments.
The paper presents an interesting new problem setup for imitation learning: an agent tries to imitate a trajectory demonstrated by an expert but said trajectory is demonstrated in a different state or observation space than the one accessible by the agent (although the dynamics of the underlying MDP are shared). The paper proposes a solution strategy that combines recent work on domain confusion losses with a recent IRL method based on generative adversarial networks.
I believe the general problem to be relevant and agree with the authors that it results in a more natural formulation for imitation learning that might be more widely applicable.
There are however a few issues with the paper in its current state that make the paper fall short of being a great exploration of a novel idea. I will list these concerns in the following (in arbitrary order)
- The paper feels at times to be a bit hurriedly written (this also mainly manifests itself in the experiments, see comment below) and makes a few fairly strong claims in the introduction that in my opinion are not backed up by their approach.
For example: "Advancements in this class of algorithms would significantly improve the state of robotics, because it will enable anyone to easily teach robots new skills"; given that the current method to my understanding has the same issues that come with standard GAN training (e.g. instability etc.) and requires a very accurate simulator to work well (since TRPO will require a large number of simulated trajectories in each step) this seems like an overstatement.
There are some sentences that are ungrammatical or switch tense in the middle of the sentence making the paper harder to read than necessary, e.g. Page 2: "we find that this simple approach has been able to solve the problems"
- The general idea of third person imitation learning is nice, clear and (at least to my understanding) also novel.
However, instead of exploring how to generally adapt current IRL algorithms to this setting the authors pick a specific approach that they find promising (using GANs for IRL) and extend it. A significant amount of time is then spent on explaining why current IRL algorithms will fail in the third-person setting. I fail to see why the situation for the GAN based approach is any different than that of any other existing IRL algorithm. To be more clear: I see no reason why e. g. behavioral cloning could not be extended with a domain confusion loss in exactly the same way as the approach presented.
To this end it would have been nice to rather discuss which algorithms can be adapted in the same way (and also test them) and which ones cannot. One straightforward approach to apply any IRL algorithm would for example be to train two autoencoders for both domains that share higher layers + a domain confusion loss on the highest layer, should that not result in features that are directly usable? If not, why?
- While the general argument that existing IRL algorithms will fail in the proposed setting seems reasonable it is still unfortunate that no attempts have been made to validate this empirically. No comparison is made regarding what happens when one e.g. performs supervised learning (behavioral cloning) using the expert observations and then transfers to the changed domain. How well would this work in practice ?
Also, how fast can different IRL algorithms solve the target task in general (assuming a first person perspective) ?
- Although I like the idea of presenting the experiments as being directed towards answering a specific set of questions I feel like the posed questions somewhat distract from the main theme of the paper. Question 2 suddenly makes the use of additional velocity information to be a main point of importance and the experiments regarding Question 3 in the end only contain evaluations regarding two hyperparameters (ignoring all other parameters such as the parameters for TRPO, the number of rollouts per iteration, the number of presented expert episodes and  the design choices for the GAN). I understand that not all of these can be evaluated thoroughly in a conference paper but I feel like some more experiments or at least some discussion would have helped here.
- The presented experimental evaluation somewhat hides the cost of TRPO training with the obtained reward function. How many roll-outs are necessary in each step?
- The experiments lack some details: How are the expert trajectories obtained? The domains for the pendulum experiment seem identical except for coloring of the pole, is that correct (I am surprised this small change seems to have such a detrimental effect)? Figure 3 shows average performance over 5 trials, what about Figure 5 (if this is also average performance, what is the variance here)? Given that GANs are not easy to train, how often does the training fail/were you able to re-use the hyperparameters across all experiments?
UPDATE:
I updated the score. Please see my response to the rebuttal below.
Question:
- The paper feels at times to be a bit hurriedly written (this also mainly manifests itself in the experiments, see comment below) and makes a few fairly strong claims in the introduction that in my opinion are not backed up by their approach.
For example: "Advancements in this class of algorithms would significantly improve the state of robotics, because it will enable anyone to easily teach robots new skills"; given that the current method to my understanding has the same issues that come with standard GAN training (e.g. instability etc.) and requires a very accurate simulator to work well (since TRPO will require a large number of simulated trajectories in each step) this seems like an overstatement.
Answer:
The claim is that advancements in this class of algorithms [i.e., third person imitation learning algorithms] would significantly improve the state of robotics. Ergo, it is a problem worthy of considering, stating formally, and offering a first solution attempt. We do not claim that our specific solution significantly advances the state of robotics.  This paper is an attempt to draw attention to the problem, not to conclusively solve it, but rather providing a contribution that can be built on.
Question:
There are some sentences that are ungrammatical or switch tense in the middle of the sentence making the paper harder to read than necessary, e.g. Page 2: "we find that this simple approach has been able to solve the problems"
Answer:
We will extra carefully edit the final version of the paper (up to and including hiring outside grammar and style police to ensure the highest possible quality).
Question:
The general idea of third person imitation learning is nice, clear and (at least to my understanding) also novel.
However, instead of exploring how to generally adapt current IRL algorithms to this setting the authors pick a specific approach that they find promising (using GANs for IRL) and extend it. A significant amount of time is then spent on explaining why current IRL algorithms will fail in the third-person setting. I fail to see why the situation for the GAN based approach is any different than that of any other existing IRL algorithm. To be more clear: I see no reason why e.g. behavioral cloning could not be extended with a domain confusion loss in exactly the same way as the approach presented.
Answer:
Thank you, and this is an excellent suggestion (which we hope to pursue in future work), but beyond the scope of this current paper, which already studies many factors even while staying within the IRL setting.
Indeed, to fully flesh out everything you suggest here could easily fill a 30 page paper if done thoroughly, making it far too large for the venue! At some point, a narrowing of focus is required. And we felt IRL was the appropriate scope for this work. As a compromise, we would be willing to rename the paper “Third Person Inverse Reinforcement Learning” if the reviewer feels this distinction is essential to maintain.
It is worth noting that behavioral cloning requires access to the raw actions taken by both agents.
Hence the requirements for behavioral cloning are significantly more constraining in terms of how data is collected.  In this work, we are considering the setting where there is only access to a video of the behavior we are trying to imitate.   We will discuss this difference more explicitly in the next revision of the paper.
Question:
To this end it would have been nice to rather discuss which algorithms can be adapted in the same way (and also test them) and which ones cannot. One straightforward approach to apply any IRL algorithm would for example be to train two autoencoders for both domains that share higher layers + a domain confusion loss on the highest layer, should that not result in features that are directly usable? If not, why?
Answer:
We actually tried this in our initial investigations for 3rd person imitation learning.  Our findings suggested it wasn’t as promising a direction, hence we proceeded along the reported line of research.   We did not mention these attempts in the paper as we discarded it for our own purposes as not as promising, but don’t want to necessarily discourage others from pushing this harder (than we believe is warranted for our own time).
Question:
- While the general argument that existing IRL algorithms will fail in the proposed setting seems reasonable it is still unfortunate that no attempts have been made to validate this empirically. No comparison is made regarding what happens when one e.g. performs supervised learning (behavioral cloning)
Answer:
Wrt behavioral cloning, see above.
Wrt existing IRL algorithms -- as discussed in the paper, generally speaking there are two categories: (i) using hand-engineering/crafted features of which the reward is a linear combination, (ii) using arbitrary reward functions (as could be represented by a generic neural net, or in earlier work, by a generic Gaussian process).   For (i), as discussed in the paper, if carefully choosing these features, one might be able to choose them to be domain invariant in some situations, but this will require extensive engineering effort in every new setting.  We are pursuing (ii), as has recent work by Ho et al (nips16), and Finn et al (icml16), but their work assumed demonstrator and agent act in the same view / environment, and their work would fail -- indeed, when we remove the multi-time frame input and the domain confusion, ours corresponds to Ho et al, and performance drops significantly.
Question:
using the expert observations and then transfers to the changed domain. How well would this work in practice ?
Also, how fast can different IRL algorithms solve the target task in general (assuming a first person perspective) ?
Answer:
Thank you for this suggestion. We have added further comparisons accordingly in appendix A. We see that first person IRL and RL both solve the task moderately faster (requiring for instance 7 epochs of training to reach the final performance attained by third person imitation after 20 epochs).
However, we see that the final performance of our algorithm is still comparable.
Further, we see that attempting to apply the policy learned from the first person cost on the third person agent utterly fails for all three environments, suggesting that something like our algorithm is necessary in the case of third-person imitation learning.
Question:
- Although I like the idea of presenting the experiments as being directed towards answering a specific set of questions I feel like the posed questions somewhat distract from the main theme of the paper. Question 2 suddenly makes the use of additional velocity information to be a main point of importance and the experiments regarding Question 3 in the end only contain evaluations regarding two hyperparameters (ignoring all other parameters such as the parameters for TRPO, the number of rollouts per iteration, the number of presented expert episodes and  the design choices for the GAN). I understand that not all of these can be evaluated thoroughly in a conference paper but I feel like some more experiments or at least some discussion would have helped here.
Answer:
Somewhat happily, there was no hyper-parameter tuning required for TRPO. The hyper-parameters that are supplied in RLLab by default worked out of the box.
While we understand that this is sometimes a potential warning sign, please note that the TRPO hyper-parameters have been tuned extensively in past work. The GAN discriminator neural network architecture choices and hyper-parameters were taken from Levine et al, JMLR 2016, who extensively investigated these choices to deal with similar (robotics) tasks in the past.  We will clarify this in the paper.
Question:
- The presented experimental evaluation somewhat hides the cost of TRPO training with the obtained reward function. How many roll-outs are necessary in each step?
Answer:
Between 10 and 30.   (This has been clarified in the paper.)
Question:
- The experiments lack some details: How are the expert trajectories obtained?
Answer:
Via TRPO with full state information. (This has been clarified in the paper.)
Question:
The domains for the pendulum experiment seem identical except for coloring of the pole, is that correct (I am surprised this small change seems to have such a detrimental effect)?
Answer:
Indeed, we chose this domain to investigate how the discriminator and reward function would behave with such clearly distinguishing visual appearance.  If not using domain confusion loss, the reward function immediately uses this bit of information to distinguish between expert vs non-expert trajectories (since all of the expert trajectories have a blue pole and the non-experts have a purple pole).
Consequently, without the domain confusion loss, the learned reward function is meaningless for the test domain.  We will further clarify this in the paper.
Question:
Were you able to re-use the hyperparameters across all experiments?
Answer:
Yes, aside from the number of expert sample rollouts utilized. This number varied between 10 and 30.
We thank you for this very thorough and comprehensive review! Carefully considering your comments has allowed us to improve the quality of our paper.
Overall I believe the revised version was improved significantly over the initial submission through a series of additional experiments/evaluations which highlight the difficulty of the third person imitation learning setting. I have increase my score in response. I believe the paper could be published as is but am still unsure of its significance: On the plus side the considered setting is novel and interesting and the experiments now clearly show the effectiveness of the domain confusion loss. On the negative side the paper does not contain any baseline besides the GAN based approach favored by the authors and the solution mainly consist of combining two existing techniques.
Regarding the claim about advancements of robotics through third person imitation learning: your point is well taken. I misread the sentence.
Regarding grammar and style: No need to involve the police ... I am sure you will be able to remove these issues before publication. I merely meant that some of the text was harder to read than it needed to be, also now some of the plots are quite tiny and for example the plots in the Appendix (e.g. Figure 10) make a comparison between methods unnecessarily hard as they don't combine experiments in one plot and all y-axes of the presented plots have different scales.
Regarding the answer on exploring a more general setting: I understand that covering the whole battery of IRL algorithms and how they could be adapted to the third person setting is impossible in a conference submission. It is clear that focus is necessary. My main issue is/was that I would have liked to see any baselines beyond the GAN based approached and that I somewhat fail to see why they should be so hard to come by.
Other than that my comment regarding the fact that I would have liked to see the idea explored more broadly should not hold up publication of the paper (it is merely a matter of preference).
Regarding your answer as to why behavioral cloning cannot be adapted: "It is worth noting that behavioral cloning requires access to the raw actions taken by both agents.
Hence the requirements for behavioral cloning are significantly more constraining in terms of how data is collected."
I agree that your algorithm is more general in this regard!
However, in all experiments you consider (or at least for the pendulum and the reacher) the action space is the same both for the teacher and the student.
As such adding a behavioral cloning baseline would consist in collecting the actions for the frames you collect anyway, adding a domain confusion loss and doing supervised learning.
Of course it is nice that your method could potentially also work in cases where the actions differ but your experiments don't currently exploit/show this anyway.
Regarding the Auto-encoder idea: good to know that this does not work / is not easy to make work. Thanks for the clarification.
Regarding your answer to "using the expert observations and then transfers to the changed domain"
Thank you for carrying out additional experiments. They do paint a more complete picture and support your claims (note that the plots in the appendix make comparison a bit hard)! I have raised my score as a result.
Regarding clarification of the used hyperparameters: Your responses make sense and I appreciate the additional details. In principle, as you said, a hyperparameter search should have been carried out for each setting to rule out bias in your evaluation but of course I understand that theses experiments are quite expensive to run.
Regarding the pendulum experiment (difference only in color): I am actually not sure the additional experiments are as conclusive as you claim for this experiment. Figure 8 shows an increase in performance also for simply transferring the first person policy to the third person agent (in contrast to the other domains) and the variance seems to be quite high. Since the experiment was stopped early it is hard to make any conclusion about the final performance. I agree that in the third person imitation learning setting the color makes the task for the discriminator trivial but that does not apply to the first person setting (with following transfer).
I apologize for missing the opportunity to ask a pre-review questions. Please see the main review for my questions.
This paper proposed a novel adversarial framework to train a model from demonstrations in a third-person perspective, to perform the task in the first-person view. Here the adversarial training is used to extract a novice-expert (or third-person/first-person) independent feature so that the agent can use to perform the same policy in a different view point.
While the idea is quite elegant and novel (I enjoy reading it), more experiments are needed to justify the approach. Probably the most important issue is that there is no baseline, e.g., what if we train the model with the image from the same viewpoint? It should be better than the proposed approach but how close are they? How the performance changes when we gradually change the viewpoint from third-person to first-person?
Another important question is that maybe the network just blindly remembers the policy, in this case, the extracted feature could be artifacts of the input image that implicitly counts the time tick in some way (and thus domain-agonistic), but can still perform reasonable policy. Since the experiments are conduct in a synthetic environment, this might happen. An easy check is to run the algorithm on multiple viewpoint and/or with blurred/differently rendered images, and/or with random initial conditions.
Other ablation analysis is also needed.
For example, I am not fully convinced by the gradient flipping trick used in Eqn. 5, and in the experiments there is no ablation analysis for that (GAN/EM style training versus gradient flipping trick). For the experiments, Fig. 4,5,6 does not have error bars and is not very convincing.
Question:
While the idea is quite elegant and novel (I enjoy reading it), more experiments are needed to justify the approach. Probably the most important issue is that there is no baseline, e.g., what if we train the model with the image from the same viewpoint?
Answer:
We have added new experiments, including training the model with the image from the same viewpoint.   We have also added RL in the test domain and standard first person imitation (i.e. our approach, but without the domain confusion).  All discussed in Appendix A. As we see from these graphs, simply using the cost/policy recovered from first person imitation on the third person agent is not sufficient to learn the task presented. We see that generally RL and first person imitation learning do perform better on these tasks (in terms of sample efficiency and overall performance).
However, we feel that third person imitation performs comparably, and this fact is significant enough to warrant strong consideration for accepting this paper.
Question:
How the performance changes when we gradually change the viewpoint from third-person to first-person?
Answer:
We have added this experiment to appendix A. We see that for the point env, performance linearly declines with the difference in camera angle between the expert and the novice. For reacher, the story is more complex and the behavior is more step like. Thank you for suggesting this experiment!
Question:
Another important question is that maybe the network just blindly remembers the policy, in this case, the extracted feature could be artifacts of the input image that implicitly counts the time tick in some way (and thus domain-agonistic), but can still perform reasonable policy. Since the experiments are conduct in a synthetic environment, this might happen. An easy check is to run the algorithm on multiple viewpoint and/or with blurred/differently rendered images, and/or with random initial conditions.
Answer:
We sample different initial conditions as in the underlying RL Lab environments. We see that the controls generated by different initial conditions are quantitatively different.
Further, when we cross-apply the controls generated from one set of initial conditions to another set of initial conditions, we see that performance is generally poor. We are hesitant to add these graphs to the paper, as there are already 16 additional graphs as a result of this rebuttal (on top of the 15 graphs in the original paper for a total of 31).
Question:
Other ablation analysis is also needed.
For example, I am not fully convinced by the gradient flipping trick used in Eqn. 5, and in the experiments there is no ablation analysis for that (GAN/EM style training versus gradient flipping trick).
Answer:
Figure 5 contains an ablation analysis for the gradient trick, concretely, it compares what happen with and without domain confusion, and with and without the velocity information.  The experiments show that having both domain confusion (i.e. gradient flipping trick) and velocity information outperforms ablated variants.
Figure 6 analyzes performance as a function of the parameter lambda, which weighs the domain confusion loss (against the other losses).  It shows the approach is robust to choices of lambda, however, extreme choices don’t perform well: lambda too small results in domain confusion loss largely ignored and poor performance; lambda too large results in domain confusion loss dominating too much (at the expense of the other losses).
Question:
For the experiments, Fig. 4,5,6 does not have error bars and is not very convincing.
Answer:
We ran these experiments multiple times, and results were consistent, as in the reported graphs.  We felt that the error bars were distracting for these experiments because they cluttered the graphs. For the final, we will add error bars, and increase the size of the graphs to maintain readability.
Our additional experiments (added in rebuttal phase into Appendix A) have error bars, which further suggest the robustness of the algorithm.
Note that there are now 31 experiments detailed in this paper. We feel that adding any more may cause it to burst at the seams! :)
We thank you for your constructive feedback, which significantly improved the quality of this paper.  We were able to incorporate 16 new experiments and other improvements.  We believe this has further strengthened the work and the paper (beyond its state at initial submission) and hope that you will potentially consider updating your score with this in mind.

Meta-review:
role: PC
data: comment: pros:
 - new problem
 - huge number of experimental evaluations, based in part on open-review comments
 
 cons:
 - the main critiques related to not enough experiments being run; this has been addressed in the revised version
 
 The current reviewer scores do not yet reflect the many updates provided by the authors.
 I therefore currently learn in favour of seeing this paper accepted.

Decision: Accept (Poster)