Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper deals with offline LfO with imperfect demonstrations. The main contribution of this paper includes solving a distribution matching problem without  requiring an inverse dynamics model. The reduction of the problem is simple. But the algorithm and the theory heavily borrow from DemoDICE[15]. and the contribution of this paper is incremental.
strengths_and_weaknesses: - Strengths
  - The paper reduces the regular minimax problem in matching state distribution to a single convex minimization problem to make the proposed algorithms more stable than prior work.
  - There are sufficient baselines in the experiments to support the performance of LobsDICE.

- Weaknesses
  - The algorithm and the theory heavily borrow from DemoDICE[15].
questions: Figure 1 shows that the performance gap between LobsDICE and other baselines becomes larger as the level of stochasticity increases. And there is no gap when the environment is nearly deterministic. 

However, the Mujoco environments in Figure 2 are deterministic within my knowledge. So how to explain the performance of LobsDICE in figure 2?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper propose an offline learning from observation (LfO) algorithms, which imitates the expert policy by stationary distribution matching without interacting with the environment. The authors presented how to transfer the original complex minimax optimization problem to a simple convex minimization problem in detail, which avoids the instability issue in the original DICE based algorithms. 
Empirical experiments show the effectiveness of the proposed algorithms and shows competitive performance compared with baseline algorithms.
strengths_and_weaknesses: - **Originality**.  
The proposed algorithm is based on previous DICE techniques including DualDICE, OptiDICE for offline policy evaluation or improvement, and the authors extended the idea to the offline learning from observations settings. As far as I know, the techniques are not new but there are very few prior works that consider extending the idea to the offline LfO setting. 

-  **Quality**.  
The paper is technically sound and the empirical experiments are conducted thoroughly. One concern I have is that, as the authors mentioned, there are some concurrent works (SMODICE) related to this topic. In the appendix the authors discussed potential drawbacks of the concurrent work, while there are no empirical experiments to validate and support the claim, which I think the authors should make clear comparison and discussion in both the experiments and the related work sections (I checked with their paper, it seems that they already release the code?). 

- **Clarity**.  
Overall the paper is well presented and it is easy to follow. The derivation presented in the main content are easy to follow if the readers know the basic DICE techniques in the previous work.

- **Significance**.  
The presented techniques are a little bit incremental, but I think the proposed algorithm is a good addition to the offline LfO settings. 



questions: I have a long existing fundamental question for this line of works, which try to use DICE techniques to perform offline imitation learnings, ever since ValueDICE, and also for this work.

 If we consider Equation (1) or final Equation (5) in the paper for offline imitation learning, what is the distribution of $\bar{d}^{E}(s, s^\prime)$, $d^{I}(s,a)$ or $d^{E}(s,a)$? Are they the **discounted** stationary distributions (the distribution would involve initial state distribution and a discounted factor $\gamma < 1$) or just the **average** visitation stationary distribution ($\gamma =1$ or the distribution you just uniformly sample from the empirical dataset).

This question is important, because when we consider to use the **discounted** stationary distribution $\bar{d}(s, s^\prime)$ or $d(s, a)$ of the current learning policy to match the target distributions ($d^{I}(s, a)$ or $\bar{d}^{E}(s, s^\prime)$), the target distributions should also be the **discounted** ones. As a result, $d^{I}(s, a)$ or $\bar{d}^{E}(s, s^\prime)$ must be the **discounted distribution**, which are not the **same as the distributions we uniformly sample from the empirical dataset**. 

From what the authors describe in the paper, I think the authors just use the empirical data distribution as $d^{I}(s, a)$ or $\bar{d}^{E}(s, s^\prime)$, which are actually not right because the uniformly sampling procedure would give us the average visitation stationary distribution ($\gamma = 1$). 

There are two ways to fix the flaw:
1. Consider to define $\bar{d}(s, s^\prime)$ or $d(s, a)$  as the average visitation stationary distribution ($\gamma = 1$), thus Equation (7) is different, and the whole derivation of this paper need to change. 

2. Consider to change the sampling procedure such that $d^{I}(s, a)$ or $\bar{d}^{E}(s, s^\prime)$ are the discounted stationary distribution, which involves the initial distribution $p_{0}(s)$. One possible simple solution is to sample transition data according to $\gamma^t$, where $t$ is the time step. 

-----
Empirically I think there might not be significant impact of above concern, and please correct me if I misunderstand anything in the paper, and I will change my score. 








  
limitations: The authors adequately addressed the limitations and there are no negative societal impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper algorithmically extends recent advances to the DICE literature in the offline RL setting to the IFO setting. They present a novel extension to match state-transition distributions as well as a more stable minimization objective to achieve SOTA performance for Offline IFO on the Mujoco benchmark task. 
strengths_and_weaknesses: ### Strengths
1. Technical contribution: Extends much of the recent advancements in the DICE literature to the IFO setting to do (s, s') distribution matching. 
2. Definitely excited to see the stabilization of the DICE objective to be a single minimization than a min-max. On a side note, this makes me wonder if this sheds some light into scaling up to higher dimensional benchmarks?
3. Strong experimentation showing the effective state-transition minimization with state-of-the-art performance on the Mujoco benchmarks. I know D4RL does not provide datasets for Humanoid but would be interesting to fully finish up the benchmark suite for IFO.

### Weaknesses
1. One small nitpick would be compare against an Offline IL method such as MILO [1] slightly modified for IFO (train discriminator on (s, s') rather than (s, a)). This would be a comparison against a principled offline method for adversarial IL. This would provide more proof of the stability contribution of this work of having just a minimization objective rather than a min-max objective - a benefit the authors mention throughout the text.


[1] Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage, 2021 
questions: This is not a critique, but from the experiments it looked like you used 5 expert demonstrations. I'm curious about how low could you go? BC is able to achieve expert performance with as low as 1 trajectory in these environments (I understand BC also gets actions so has a definite advantage), but it would be really cool to see how LobsDICE does with less and less expert demonstrations. 
limitations: Yes the authors have sufficiently addressed limitations and societal impacts in their paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose an algorithm for offline learning from _observations_, i.e. for learning from observed expert trajectories where only the states, not the actions can be observed. Additionally, similar to some prior work, the authors assume access to a large set of data from suboptimal policies, but containing actions.

Overall, the paper follows the DemoDICE paper very closely, both in structure of the paper as well as in terms of the algorithm, but with the crucial difference than DemoDICE assumes access to expert actions.

Interestingly, the optimization target for both algorithms ends up looking exactly alike:

Equation (23) from LobsDice: 

$$
\widetilde{\mathcal{L}}(\widetilde{\nu})=(1-\gamma) \hat{\mathbf{E}}\_{s\_{0} \in D_{0}}\left[\widetilde{\nu}\left(s_{0}\right)\right]+(1+\alpha) \log \hat{\mathbf{E}}\_{x \in D^{I}}\left[\exp \left(\frac{1}{1+\alpha} \widehat{A}_{\widetilde{\nu}}\left(s, a, s^{\prime}\right)\right)\right]
$$

Equation (17) from DemoDice: 

$$
\widetilde{L}(\nu ; r):=(1-\gamma) \mathbb{E}\_{s \sim p\_{0}}[\nu(s)]+(1+\alpha) \log \mathbb{E}\_{(s, a) \sim d^{U}}\left[\exp \left(\frac{A\_{\nu}(s, a)}{1+\alpha}\right)\right]
$$

with, as far as I can tell, the only difference being whether the discriminator is learned on tuples $(s,s')$ or $(s,a)$.

However, as far as I can tell, it is not obvious a-priori that the solutions will end up looking the same, hence making the derivations in the paper necessary. 

Performance is evaluated against several baselines (BC, BCO, DemoDICE with inverse dynamics model, OPOLO) on both a tabular environment and several MuJoCo environments (Hopper, Walker2d, Ant and HalfCheetah), showing good performance.
strengths_and_weaknesses: ## Strenghts

* I found the writing accessible and good to follow, as well as the mathematical derivation well explained. I do think the paper might be a bit harder to follow if one is not familiar with similar algorithms (e.g. the *DICE family), so a short recap of their their core ideas in the background section would be helpful. Similarly, a recap of DemoDICE would be helpful and make the close connection clearer.
* The topic of offline IL is relevant, as is the extension to LfO
* I liked the experimental section and found both the figures and discussion easy to follow as well as insightful.

## Weaknesses

* I think the presentation of the derivation of the algorithm could be slightly improved. While each step was easy to follow, it took me a while (including going back and forth between LobsDICE and other *DICE papers) to get a decent high level motivation for the individual steps. It certainly does not require any major changes, but maybe a bit of additional signposting could help.
* I think the relationship to DemoDICE should be made clearer. The authors already mention that DemoDICE solves a related problem, but I would have found it helpful if also the similarities and differences in the solution would be discussed.
* One could argue that the "delta" in terms of research compared to DemoDICE is limited. However, I believe there is sufficient difference to warrant publication. 
questions: Please see my suggestions above regarding small changes to the presentation of the paper. 
I don't have any question and I think this is a solid paper which should be accepted.
I'm not rating it stronger than "accept", due to the similarity (in approach and solution) to DemoDICE. 
limitations: N/A.
review_title: null

Meta-review: metareview: Reviewers all agree that the paper has solid contributions on both theory and experiments. 
confidence: Certain
Decision: Accept