Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors propose an offline imitation learning algorithm that annotates rewards by using Optimal Transport (w.r.t. expert trajectories) solver and uses those reward with the offline RL to mimic expert behavior without environment interactions. The algorithm is evaluated on D4RL benchmarks (which is a widely adopted benchmark for offline RL and IL algorithms) and is shown to outperform existing offline IL algorithms. 
strength_and_weaknesses: ### Strength

- The proposed algorithm outperforms baselines. 

### Weaknesses

- The contribution seems incremental. Using OT to acquire IL rewards was proposed already by PWIL (by Dadashi et al) although the idea was for online imitation learning. The submission extends PWIL's idea to offline IL by using offline RL instead. 
- The complexity of using OT is not well-described. When the number of expert demonstrations is large, the reward estimation may not be scalable due to solver's complexity. 

clarity,_quality,_novelty_and_reproducibility: ### Clarity
- The submission is clearly written, and their contribution is well-summarized. 
- All mathematical notations seem correct. 

### Quality
- Various experiments were done.
- Experiment results are well-summarized. 

### Novelty
- The submission can be regarded as a simple offline extention of PWIL.
- Most of ideas regarding OT (e.g., using exponentiated rewards, computing optimal alignment with solver) are already presented by PWIL. 

### Reproducibility
- Source code is not given, but hyperparameters are shared, which seems to be reproducible. 
summary_of_the_review: Although the algorithm empirically outperforms baselines, the idea doesn't seem novel, and the contribution seems incremental.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new algorithm for reward-free offline reinforcement learning when a few expert demonstrations are available. The core idea is to assign rewards to the unlabelled trajectories based on their distance to the expert demonstrations such that the closer the trajectories they are to the expert demonstrations, the higher rewards they get assigned. The proposed approach (OTR) uses an optimal transport formulation where each trajectory is treated as a discrete distribution with uniform density over the states in the trajectory and the distance between two trajectories is defined as the Wasserstein distance between these two discrete distributions. The paper demonstrates that OTR can be combined with existing offline RL methods with little implementation and computation overhead to tackle reward-free offline RL with demonstrations and often matches the performance of offline RL algorithms when rewards are given on a range of MuJoCo locomotion tasks and Adroit manipulation tasks.
strength_and_weaknesses: *Strength*
- The proposed approach is conceptually simple and elegant with strong empirical results on a range of tasks.

*Weaknesses*
- OTR introduces additional hyper-parameters such as $\alpha, \beta$ in the squashing function (for reward labeling). The paper currently has no sensitivity analysis on how these parameters could affect the performance.
- There is uncertainty over how the cost between states is defined in the optimal transport formulation (see the section below), which could impact the reproducibility of the paper. 
- I found the claim that the proposed approach (OTR) can consistently match the performance of offline RL with ground-truth rewards is not well-supported by the current set of experiments (see the section below). 
clarity,_quality,_novelty_and_reproducibility: - It is unclear to me how the cost function is defined between states exactly. The authors mentioned in the 4.1 Setup that they use the cosine distance and reference the paper by Cohen et al. (2021). However, the referenced paper deals with pixel observations and the cosine distance is computed in the encoded observation. As far as what I understand, all the experiments in the paper considers state observations which adds uncertainty on how the distance is computed. This is a very important detail as the distance function (or the cost function) directly affects how the Wasserstein distance between trajectories is defined, which is the main objective that the proposed uses to minimize to provide the shaped reward function. 

- Are the expert demonstrations used for OTR also included in the dataset for the IQL oracle (the baseline with ground-truth rewards)? The most relevant sentence I could find is this -- "For each environment, we use the medium-v2, medium-replay-v2 and medium-expert-v2 datasets to construct the expert demonstrations and the unlabeled dataset." I found it to be vague as it could be interpreted as that the expert demonstrations are constructed by selecting the best trajectories from all three datasets combined for each environment. If it is indeed the case that the best trajectories are selected from all three datasets combined for each environment, I think these expert trajectories should also be added to the individual datasets (medium-v2, medium-replay-v2 and medium-expert-v2) for the IQL oracle to ensure a fair comparison since the paper tries to claim that the OTR+IQL could consistently match the performance of offline RL with ground-truth rewards (e.g., the last sentence of the abstract). Otherwise, the claim is not well-supported because both IQL oracle and OTR+IQL have advantages in their own way (IQL oracle has access to ground truth reward but possibly worse trajectories, OTR+IQL has access to possibly better trajectories but with much fewer reward labels). 
summary_of_the_review: The paper is generally well-written with strong empirical results. Although I have some concerns over the reproducibility of the work and empirical comparisons with respect to the baselines (see above), they do not occur to me to be major issues as long as they are addressed appropriately. Overall, I would recommend acceptance of the paper (borderline).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors focused on the offline reinforcement learning setting where a small set of expert demonstrations and a larger dataset without reward are available. 
The proposed method labels the reward for the larger dataset with optimal transport theories. Specifically, the authors considered the distribution over the state space and use the optimal coupling to decide the closeness of states. 
The unlabeled trajectory will be assigned a higher reward if its state distribution is closer to the state distribution from experts' data. The authors demonstrated the effectiveness of this method in several environments

strength_and_weaknesses: Pros:
- Overall, the paper is well-written and quite easy to follow.
- The idea of combining optimal transport with reinforcement learning is pretty novel. Given that there are only limited previous studies.
- The experimental results are encouraging. 

Cons:

- The novelty is limited, considering some previous studies. I have included more detailed questions in the next section.

- In Figure 2, the cost matrix and the optimal coupling do not seem to correspond. In addition, the authors may want to annotate $s^e_3$ and $s^e_4 $in the leftmost figure.

- The authors used cosine similarity as the ground metric. Is there a principle to choose the ground cost since this is very important for the OT objective? The authors may want to refer to bisimulation metrics.

- One of the pitfalls of offline RL approach is the distribution shift problem. However, the authors did not discuss how the proposed method can handle this issue. When the expert trajectories state distribution is far away from the unlabeled ones, I am wondering whether the proposed method will still annotate the closest states by high rewards. 


clarity,_quality,_novelty_and_reproducibility: The paper writing is pretty satisfactory, and the authors explained their idea clearly. However, I have concerns regarding the novelty of this work.

The authors had better justify the benefit of this method more comprehensively. The proposed method directly applied the Wasserstein distance (Sinkhorn algorithm) to the empirical distribution of the spaces. However, it seems that the proposed reward annotating method ignores the transition kernel, which is an important ingredient for RL. MDP is a decision process, how does this method take the information from transition kernels into account? 
In the contrast, PWIL proposed to find the coupling according to the sequential orders. In addition, in UDS, the authors claimed that, under certain conditions, setting rewards to zero still improves an offline RL task and provided theoretical justifications. Is it possible to elaborate more about how the proposed method can leverage the geometric information with OT?

summary_of_the_review: The paper is overall well-written. However, I would suggest the authors elaborate more on the benefits of labeling rewards with optimal transport coupling.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for performing imitation learning by first computing rewards using a method called Optimal Transport Reward labeling (OTR) that computes the Wasserstein distance between an expert demonstration and an unlabeled one. The rewards are then used in combination with existing offline RL algorithms, for example, implicit Q-learning (IQL). The process is made computationally efficient through the use of GPU-accelerated optimal transport solvers. The method is tested on locomotion tasks from the D4RL benchmark, and compared to DemoDICE, and ORIL and UDS for computing rewards. Through additional experiments, the authors show that OTR is able to effectively use up to 10 demonstrations to boost performance.
strength_and_weaknesses: Strengths:
- The method is conceptually simple, and the quantitative results are encouraging – OTR+IQL is able to recover the performance of IQL using oracle rewards for many of the D4RL tasks.
- It’s quite surprising and impressive that a single good expert demonstration can be used for the expert trajectory in this setup.
- The empirical evaluation compares to sensible baselines on the D4RL benchmark.
- The paper is particularly clearly written and easy to understand! 

Weaknesses:
- The introduction states that for behavior cloning, “generalization to new situations typically does not work well”, but I’m not sure I found the authors’ intuition in the paper for which aspect of OTR will improve generalization to new situations? It seems that labeling rewards based on a single demonstration will always make it more challenging to develop robust policies, as a single demonstration cannot provide signal about how to detect or recover from bad states. I’m curious if the authors could elaborate or if this would continue to work well in more challenging environments.
- I think it would be helpful to put the performance of the method in context by comparing the performance to behavior cloning on the top K% of dataset trajectories (like 1% or 10% that previous works like IQL report).

clarity,_quality,_novelty_and_reproducibility: Quality: I have a few questions that I think could make the analysis more complete if addressed:
- Can the method still work well if instead of providing the best expert demonstration in the dataset, one in for example the 90th percentile - is selected?
- How important is it that the cosine distance is used as opposed to alternatives?

Clarity:
I think the paper is generally well written and clear. However, I have a few questions:
- In Figure 2, why are there so many fewer samples from the expert policy than the other? I understand that two trajectories may have different lengths, but it seems that there are many more from the behavior policy. Yet the matrix C is drawn fairly square.
- In 4.1, it says “When there are more than one episode of expert demonstration, we perform a top K aggregation strategy with K = 1”, but I think it would be more clear just to remove the part about the top K aggregation and say “we compute the optimal transport with respect to each episode independently and use the rewards from the expert trajectory that give the best episodic return.”

summary_of_the_review: While there are related works with a similar reward relabeling idea such as [1], this paper tackles a challenging and (as far as I am aware) novel situation of learning from a very small number of demonstrations. It demonstrates promising empirical results of being able to match the performance of offline RL algorithms when given ground truth reward functions. The paper is well written and clearly presented. Therefore I vote weak accept.

[1]: Imitation Learning from Pixel Observations for Continuous Control (Cohen et. al)
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) Summary: Given a dataset of trajectories, a subset of which are known to be expert-generated, but none of which include reward information, this paper proposes an algorithm for assigning  to unlabelled trajectories with the goal of assigning higher rewards to trajectories that are closer to expert demonstrations.  Existing offline RL methods can then be used to learn a policy based on the estimated rewards.

(b) Strengths: All reviewers agreed that this is a conceptually simple approach, and the empirical performance appears to be quite good.

(c) Weaknesses: The reviewers had two main concerns.  First, whether this is an incremental contribution over existing work.  Second, the choice of cost function appeared to be somewhat arbitrary; a more thorough discussion of this design decision would have made for a much stronger paper.


justification_for_why_not_higher_score: With a deeper discussion of appropriate costs, and without the concerns that multiple reviewers had about whether the algorithm was sufficiently novel, I would have recommended accept.


justification_for_why_not_lower_score: I think a talk that highlights this paper would be a valuable part of the conference; I think a spotlight would be enough to get the main ideas across.
Decision: Accept: notable-top-25%