Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces two methods for imitation learning from pixel observations based on adversarial learning and optimal transport. Representation from the RL encoder are adopted for calculation imitation rewards. P-SIL (Pixel Sinkhorn Imitation Learning) learns and compares latent representations of the encoded expert and agent behaviors; and P-SAC (Pixel Discriminator Actor Critic) augments the representations of the agents and the experts. The key component of this work is the RL encoder used in both P-SIL and P-SAC. 
main_review: Pros: 

Two imitation learning methods based on RL encoder are proposed. The experimental results show that both of the methods outperform the DAC and SIL baselines. 

Cons: 
1. Data augment is important for improving the performance of P-DAC. However, it is not clear how to augment the data in practice.
2. It is not always possible to used the RL encoder for many tasks. This is the key issue of this work.  
3. The It is necessary to compare the proposed methods with other extentions of DAC and SIL.  
summary_of_the_review: Imitation learning can be applied in tasks where a reward function is hard to specify or too sparse to be used in practice. The methods introduced in this work is useful for imitation learning from images directly. However, the learning from pixels can be ambiguous in certain cases since the actions cannot be fully described in a video. 

A paper to appear in the Proceeding of Deep Reinforced Learning Workshop in NeurIPS 2021 has the same tile with this submission: Imitation Learning from Pixel Observations for Continuous Control (https://nips.cc/Conferences/2021/Schedule?showEvent=21848).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors have considered the problem of imitation learning using only visual observations, and they've explored the paradigms of using methods based on adversarial learning and optimal transport to solve this problem. In particular, the authors propose two new methods--P-SIL and P-DAC--presumably as a way to explore design choices and tradeoffs in this space. Some experiments designed to demonstrate the superiority of P-SIL and P-DAC are presented.
main_review: STRENGTHS

(S1) The stated thrust of the paper--that of exploring the design choices and tradeoffs within the space of algorithms designed for imitation learning from visual observations--is of great importance to the community studying these methods. To date, I can think of no paper that provides a compelling discussion of these topics.

(S2) The authors have provided an open-source implementation of their methods, which bolsters the story of the paper being one of exploring "recipes" for visual imitation.

WEAKNESSES

(W1) Unfortunately, I don't feel that the authors really lived up to the goals that they stated at the outset of the paper. That is, even with proposing and experimenting with P-SIL and P-DAC, I still don't have a clear understanding of the "design choices and tradeoffs" in this algorithmic space. Nowhere in the paper can I find a clear and explicit discussion of these things--the five experimental questions stated at the beginning of Section 4 seem to ultimately focus more on whether or not P-SIL and P-DAC will perform well, as opposed to quantifying or discovering which design choices are more important. For example, with respect to data augmentation, Figure 6 really only serves to show that P-DAC and P-SIL are better than BC rather than exploring the effect of data augmentation as a specific design choice across the suite of available methods. Moreover, with respect to the experiment depicted in Figure 7, it does not seem to me that the presented data matches up with the conclusion. Many of the domains show that all the methods compared perform about as well no matter what distance function is used--this doesn't at all support the claim that the authors have made that "OT alignment significantly outperform [sic] non-OT approaches." Finally, if P-SIL and P-DAC are both built atop DrQ-v2 and the other methods compared against are not, what evidence is there that the performance gain is not simply due to the use of DrQ-v2 as opposed to the "representation sharing" and other more general factors that the authors claim account for this difference? I do note that there _is_ unused space at the end of Page 9--I wonder if the authors might be able to address this comment with an enhanced Conclusion for example.

(W2) Again, for a paper purporting to discuss design choices and tradeoffs, I was disappointed that more experimental details weren't given. For example, with respect to the data augmentation experiments, I could not find anywhere in the paper where it was specified what exact types of data augmentation were performed for these experiments.

MINOR COMMENTS

(MC1) The bottom half of Figure 1 failed to render in an application on my iPad, though it did render correctly in Chrome. Perhaps the authors could pursue a more reliable way to generate that figure to avoid this problem in the next version of the paper.
summary_of_the_review: I agree wholeheartedly with the goal that the authors have stated in the early portions of the paper, but I think the work that has been done--or at the very least, the way it has been presented--fails to live up to that goal. I'd encourage the authors to focus more on the design decisions and tradeoffs in the existing space of algorithms rather than trying to show that their proposed algorithms dominate others.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents two algorithms for imitation learning P-SIL and P-DAC which are built on top of SIL (Papagiannis & Li, 2020) and DAC (Kostrikov et al, 2019). The first one is an algorithm based on the Sinkhorn distance and the second one uses an adversarial approach, by training a discriminator. These two approaches use state-only trajectories to generate reward sequences to match the agent state-trajectories and learn through that reward. By combining the advances in image-based reinforcement learning by the usage of data augmentation and target encoders, the two new algorithms, P-SIL and P-DAC, are tested and reach good results on visual control tasks.

main_review: Pros:

- Easy to implement and understand algorithms, albeit knowing the algorithms they are built upon.

- Good results on pixel-based imitation learning

- Open-source implementation is provided.
 
##########################################################################

Cons:

- Adding data augmentation and DrQ-v2 as backbone is a sound and effective approach but there is little novelty in this paper, as auxiliary losses and target encoders to improve pixel observations have been studied before. To my understanding the contributions for P-SIL with respect to SIL are using the DrQ-v2 RL encoder to embed states instead of an adversarial approach. Instead for P-DAC, the contributions with respect to DAC are again composing it with the DrQ-v2 RL encoder and computing the rewards at the end of episodic rollouts instead of recomputing them at each training iteration.  

- Paper is hard to read:
    - pseudocode of algorithm is not detailed enough to understand your contribution. You could expand on the `rewarder` function which should be your actual contribution.
    - pseudocode of algorithm doesn't have any initialization and definition of variables.
    - in equation 3, $\Psi$ and $\psi$ are not defined.
    - a background on DrQ-v2 would be highly appreciated to understand how the training pipeline works and the gradient flows, which I had to hypotesize after reading DrQ and DrQ-v2.

##########################################################################

Questions during rebuttal period:

- Address and/or clarify the cons above.
summary_of_the_review: I am towards rejection of this paper. Sound and interesting approach which achieves good results, but there is little novelty. Furthermore, paper is quite hard to read and to understand what are the main contributions, which makes an approach that seems easy quite complex.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on learning visual policies by imitating video data without observing the expert actions. The authors introduce a method to compute the imitation reward based on the representation from the expert RL encoder. Besides, a data augmentation method is proposed to scale to non-trivial control tasks. They conduct comprehensive experiments on DeepMind control suite tasks, showing the versatility and strong performance of the two approaches (P-DAC and P-SIL).
main_review: Strength:
+ The paper is well-motivated. It is attractive to enable agents to learn from demo videos without actions.
+ The paper is well-written and easy to follow.
+ Two approaches (P-SIL and P-DAC) are proposed based on stinkhorn imitation learning (SIL) and discriminator actor-critic (DAC), respectively. The introduced data augmentation method is simple yet effective in improving performance.
+ The designed experiments comprehensively answered common questions on effectiveness, efficiency, robustness, and necessity.

Weakness:
- The setting is impractical. The proposed methods rely on an RL expert for computing the reward. However, such an RL expert is unavailable in most video demos. Notably, if the RL expert is available, it will be convenient to access the expert action for training. This is my main concern for this paper. 
- It seems that the data augmentation method contributes most to the improvement in the performance. How about the performance of DAC and SIL with the data augmentation methods?

summary_of_the_review: My main concern for this paper is the setting. In my view, it is unreasonable to use an encoder from the RL-based expert for learning from videos without actions.
review_title: null

Meta-review: Learning policies from video demonstrations alone without paired action data is a promising paradigm for scaling up Imitation Learning. As such the paper is well-motivated. Two approaches P-SIL and P-DAC train rewards for RL training, based on learning Sinkhorn distances between trajectory embeddings and an adversarial approach.  The reviews brought up lack of clarity in presentation and experimental results and ablation studies falling short of convincingly demonstrating value of distance functions used and other design tradeoffs. As such the paper does not meet the bar for acceptance at ICLR.
Decision: Reject