Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
This paper proposes to use Wasserstein distance in the primal form for imitation learning. Compared with its dual form and f-divergence minimization variants, it avoids the unstable minimax optimization. In order to compute the Wasserstein distance in primal form, they also propose a greedy approximation. Their experiments demonstrate that this method has a better performance compared with baseline methods.

Pros:
+ The greedy approximation of the primal Wasserstein distance is a clean solution, and it works well for MuJoCo tasks, even if in the LfO setting
+ The paper presents a series of experiments and ablation studies, which showed quite strong performance. In the ablation, the experiments about PWIL-support is quite convincing. The agent may stay on the supports for other the density/support estimation-based methods, while PWIL doesn't.
+ The paper is well-written and easy to follow. It provides enough implementation details and codes, which is reproducible

Some questions & concerns:
+ I'm wondering if the uniform distribution assumption can be improved, by incorporating certain density estimation methods. 
+ Why the complexity of the algorithm is O((|S| + |A|)D)? It should also be linear in T
+ One limitation is that you need carefully selected metrics, e.g. L1, standardized L2. And it's generally hard to compute Wasserstein distance in the image domain. 
+ For the visual imitation experiments, the figures only contain PWIL without the results of other baselines. They can still be trained on the feature from TCC. 


review_title: Review for paper "Primal Wasserstein Imitation Learning"

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ### Summary 
The authors proposed an imitation learning algorithm that utilizes the primal form of Wasserstein distance to match agent’s and expert’s state-action visitation distributions. They considered the upper bound of the primal form and devise the optimization method based on greedy coupling which makes learning suitable for sequential problems. With standardized Euclidean distance and exponential smoothing, the proposed method PWIL is shown to perform well for both MuJoCo and Door Opening Tasks and highly outperforms the baseline (DAC) for Humanoid. 

### Quality 
Theoretical derivations are sound, and experiments are well-tuned and designed and highly support the authors’ claims.

### Clarity 
The submission is clearly written overall, but I added minor comments on clarity in `Detailed Comments`

### Originality 
The idea of submission is novel from the use of the primal form of Wasserstein distance in concatenation with greedy coupling. 

### Significance 
I think the contribution of this work is significant in the sense that it considers the primal form of Wasserstein distance and eventually show the decrease of Wasserstein distance through experiments, whereas the existing works haven’t explicitly shown the performance in terms of probabilistic metrics. 

### Detailed comments 
(p. 1, Abstract) We present a reward function which is derived offline, as opposed to recent adversarial IL algorithms that learn a reward function through interactions with the environment, and which requires little fine-tuning.
- I agree that PWIL doesn’t require neural networks and relevant gradient updates for reward learning but wasn’t sure about the term “offline” is appropriate to describe this. PWIL involves the internal loop for reward updates during environment simulation. This is a bit confusing at the first glance since I believe “offline” is used when the environment interaction is not allowed, e.g., offline reinforcement learning. (My first bias was that the reward is learned before the interaction, fixed, and then used during training without any updates.) I guess there’ll be a better expression for it (like a non-parameter reward?) or simply ignoring the term “offline” is much better. 
- A sentence in the introduction “The inferred reward function is non-stationary, like adversarial IL methods, but it is not re-evaluated as the agent interacts with the environment, therefore the reward function we define is computed offline.” tries to clarify this term, but I don’t understand why it is said that the reward is *not re-evaluated* since the most inner loop of **Algorithm 1** is related to the reevaluation of rewards. 

(p. 1, Introduction) Our method recovers expert behaviour better than existing state-of-the-art methods while being based on significantly fewer hyperparameters
- In Figure 1, PWIL seems to perform comparably to DAC, not better than DAC, except for Humanoid. 

(p. 2, Background and Notations) $\delta$ is the discount factor
- It seems like $\delta$ is used instead of $\gamma$ since $\gamma$ is used to indicate a stochastic matrix. However, since most potential readers may be familiar with the terms in RL and imitation learning, I’d rather use $\gamma$ for the discount factor and a different letter for coupling. 
- Also, it collides with Drac distribution in the following paragraph. 

(p. 2, Background and Notations) from maximizing to minimizing its return
- $\rightarrow$ from maximizing its return to minimizing cumulative costs?

(p. 2, Background and Notations) requires the definition of a metric in the space
- $\rightarrow$ requires the definition of a metric $d$ in the space (to easily link with one in the definition of Wasserstein distance)

(p. 2, Method) We introduce a reward based on an upper-bound of the Wasserstein distance
- $\rightarrow$ We introduce a reward based on an upper-bound of the Primal form of  Wasserstein distance

(p. 3, Wasserstein distance minimization) This can be problematic if an agent learns in an online manner or in large time-horizon tasks. Thus, we introduce an upper bound to the Wasserstein distance that yields a cost we can compute online, based on a suboptimal coupling strategy.
- I agree that this is a key idea of the submission, but I was wondering if it isn’t available to calculate the cost even when off-policy RL is applied as was done in the experiments. 

(p. 3, Greedy Coupling) Eq (4)
- It seems to me removing $\inf_{\pi\in\Pi}$ is possible and fits the whole equation inside the format. 

(p. 4, Figure 1) Note that the total cost with the greedy coupling is 7 whereas the total cost with the optimal coupling is 5.
- With the definition in Eq (5), I think the multiplication of $\gamma_\pi^q[i, j]$ (which is the uniform distribution) is missing. 
- Also, it would be better to add detailed calculations in the Appendix for curious readers. 

(p.4, Greedy Coupling)  The algorithm computes the greedy coupling with a complexity $\mathcal{O}((|\mathcal{S}|+|\mathcal{A}|)D)$
- How’s the complexity calculated in detail?

(P.4, Experiments) As DAC is based on TD3 which is a variant of DDPG, we use a DDPG-based agent for fair comparison.
- This makes me a bit confused since it was non-trivial for me to link off-policy RL with Algorithm 1 directly. One thing I’m really curious about is how the reward comes from the replay buffer used in a DDPG-based agent. 

(P.4, Experiments) Figure 2
- How’s the computational cost for overall evaluation? Specifically, I think there’s no problem with the evaluation of Wasserstein distance when a small number of the expert demonstration is given, but the calculation time will hugely increase for a large number of expert data (which I believe can be in practical scenarios).




review_title: Review for Paper1564

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper develops an imitation-learning (IL) method starting from the primal form of the Wasserstein distance, creating an upper-bound (by replacing optimal coupling with a greedy coupling), and converting that into a practical, scalable algorithm (PWIL). Experiments on the standard MuJoCo tasks and a pixel-based door opening task show that the method is competitive to the recent IL approaches in terms of performance and sample-efficiency.

I am currently on the fence about this paper. I would like the authors to address the following concerns:

1.	The experiments in Figure 2 compare PWIL (Distributional DDPG) with DAC (TD3). The difference in the underlying off-policy RL algorithm somewhat muddles the comparative evaluation of the reward-deriving strategy – greedy coupling in PWIL vs. discriminator training in DAC. From what I understand, PWIL should work seamlessly with TD3 as well. Is there a particular reason why the authors went with distributional-RL? Providing PWIL+TD3 results would be an equitable comparison to DAC.

2.	Discussion of potential limitations of the current approach -- the paper does strong relaxations when moving from theory to practice. I don’t have issues about this, but it would certainly be useful to include comments about the potential pitfalls of the approach. For instance, are there MDPs where the following the greedy coupling strategy provides deceptive rewards leading to sub-optimal imitation? Another example is the handling of the reward bias. Equation (6) is explicitly biased to provide a positive reward, and therefore, cannot handle cases where the episodes should be terminated at the goal (like the popular LunarLander Gym environment). Section 4.4 adds a manually designed constraint to get around this, but unlike DAC, I do not see a principled way to manage this with PWIL.

3.	Comparing rewards to those from adversarial IL – In the case of adversarial IL, the rewards are non-stationary as the discriminator parameters change throughout the training. However, for discriminator parameters at any particular iteration, the rewards are Markovian (depend only on current state and action). The rewards for PWIL seem to be non-Markovian (Equation 5), yet they have been integrated with the RL machinery (policy-gradients like DDPG) designed for Markovian rewards. Is this observation correct, and if yes, is there an intuition for why this should work?

4.	I assume the blue curve in Figure 3 is the actual Wasserstein distance. It would be helpful to add a line or two regarding the method used to obtain the optimal coupling to plot this.


===== POST REBUTTAL UPDATE =====


I have updated my score based on the response. I hope the authors can add PWIL+TD3 results (point 1. in review) in the next revision.
review_title: Updated review for PWIL

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present a well written paper with many detailed experiments.  I am happy to revisit my evaluation after a fruitful discussion :)

Pros

Wesserstein distance between demonstration and policy is setup
Using the earth mover’s analogy with the knowledge of T piles of dirt, distances to the holes are computed. Then the piles of dirt are moved to the closest holes in a greedy fashion.
An offline learning policy is presented
Sample complexity for learning is suggested to be low 
The model seems to be able to learn from a single example to run

Cons

The Sample complexity for training is still high. While I do agree that the number of demonstrations required is lower.

The win is not obvious or strong except for the humanoid condition. This is my biggest concern. Can we explore more complex environments? Or the effects of more demonstrations? It would be nice to understand where this model truly succeeds or fails.

In the door opening experiment it would be great to know how the model performs while varying the number of expert demonstrations (say). 

Might be tangential, but this paper seems relevant at least in introducing the idea of inverse reinforcement learning and W-distance
https://arxiv.org/pdf/1906.08113.pdf


review_title: Wassertein Distance based immitation learning with greedy approximation learns from fewer exemplars.

Meta-review: It is common in imitation learning to measure and minimize the differences between the agent’s and expert’s visitation distributions. This paper proposes using Wasserstein distance for this, named PWIL, by considering the upper bound of its primal form and taking it as the optimization objective. The effectiveness of the approach is demonstrated by an extensive set of experiments. 

Overall, reviewers reached general agreement that this paper makes a good contribution to the conference, and given the overall positive reviews, I also recommend accepting the paper.

Decision: Accept (Poster)