Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose an algorithm for imitation learning which rewards the agent for observing state-action pairs that are part of the demonstration-set. The main contribution of the paper is the theoretical analysis which shows that the algorithm, given sufficient expert data, matches the expert’s occupancy distribution and expected reward. The proof comes in three parts: first, the authors relate expert rollouts to the limiting distribution of the expert’s policy. Second, the authors put a probabilistic bound on the expected agreement with expert-data based on the mixing time of the agent. Finally, the authors use this bound on expected agreement to bound the difference in expected extrinsic reward. The authors also provide an empirical evaluation in standard control benchmarks.
main_review: The paper is generally well written and easy to follow; however, the structure is sometimes unusual. The proofs would be easier to follow if the authors moved Lemmas from Appendix C & D into the main body while moving some of the detailed proofs from the main body into the Appendix.

The derived bound is non-trivial, but it is unclear how significant it is. The main issue lies in the assumption on the amount of data that is required which is linear (with a factor > 1) in the squared size of the state-space (note that the bound remains large even if \delta is set to 1). Such a large amount of expert data is required since rewarding agreement is not a valid strategy when expert data is sparse: imagine a sub-sampled expert trajectory where the agent has the capability to return to previously seen states. In a more typical distribution-matching objective, the agent has to visit all the demonstration states and thus still has to complete the trajectory. In the objective proposed by the authors, the agent can maximize reward by returning to the same demonstration-state repeatedly. Naturally, if demonstrations are available in all states and the agent’s representational capacity is unlimited, then maximizing agreement is trivially optimal. The author’s bound does not require demonstrations to be available in all states and may thus be useful; however, the required amount of expert data is extremely large and no statement can be made if less data is available.

In the related works section, the authors claim that the method is minimizing the GAIL-objective. This is misleading; the GAIL objective is equivalent to minimizing the JS divergence to the expert histogram for all choices of N while the proposed method only minimizes TV distance and thus JS divergence if N is large enough. 

For the empirical evaluation, the authors use an L2-distance as a stand-in for the reward used in the analysis. This is necessary as the reward is undefined in continuous state-action spaces (similarly, it will be too sparse in large discrete state-action spaces), but the implications of that heuristic are not explored in the paper. It would be good if the authors could add motivation and discussion for this form of the reward. The results are very positive, but it is unclear whether they are due to the theoretical properties of the algorithm or due to the particular nature of the mujoco walkers. Since the reward is a part of the observation-space in these domains, using simple heuristics like squared distance can plausibly lead to high performance; however, this would not generalize to other domains. It is furthermore problematic that the hardest benchmark task, humanoid, is missing as the given baseline algorithms have no problem solving it. Finally, the reported numbers for behavioral cloning are significantly worse than what has previously been reported in the literature.

summary_of_the_review: The proposed bound is non-trivial and potentially useful; however, it would benefit from additional context to show whether it is truly significant. The empirical results are interesting, but rely on a heuristic that is not well-motivated and may be domain specific.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers imitation learning problem which aims at obtaining a policy that imitate expert behavior. Authors considers using reinforcement learning with a stationary reward constructed by expert datasets. Through theoretical analysis, the corresponding imitation policy is proved to achieve high expected per-step intrinsic reward and extrinsic reward. The difference between the expert policy and the imitation policy is bounded with a high probability. Some empirical experiments are performed on continuous control tasks. The results show this method is comparable with other algorithms while the algorithm is simpler. 
main_review: The idea of using reinforcement learning to imitate expert behavior is simple to execute compared to other imitation learning methods. I have a concern regarding the sparse reward function used in this paper. When training a reinforcement learning model, the sparse reward becomes problematic when you apply RL algorithm to solve it. Although the better intrinsic reward leads to better extrinsic reward, it is still troublesome to get a good policy by any reinforcement learning algorithm with this sparse reward function. This makes the result less useful than just applying other comparison methods. Could you give more explanation on this?
In term of computation time, the sparse reward can cause problem of bad convergence rate. It is much clearer if you can provide comparison of computation time of different algorithms in the experiment section. 
Overall, this paper is well written and very clear to readers. The result is promising and interesting from a theoretical point of view.

summary_of_the_review: This paper provides rigorous analysis for using reinforcement learning in imitation learning. The result is novel and shows the potential to accomplish imitation learning in a straightforward way. The intrinsic reward function reduced by the expert dataset is simple to construct. By setting a constant one reward for matching the demonstrated action in a demonstrated state, any reinforcement learning algorithm can provide policy for such a MDP problem. The result in this paper gives intuition of how this method performs in terms of expected return. And sufficient experimental results are shown to verify their theoretical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors show that, for deterministic experts, imitation learning can be done by reduction
to reinforcement learning with a stationary reward. theoretical analysis both
certifies the recovery of expert reward and bounds the total variation distance
between the expert and the imitation learner, showing a link to adversarial imitation
learning. Experiments are given to confirm that the reduction works well in
practice for continuous control tasks.

main_review: The paper includes potentially interesting results. Some comments that can improve the paper are given below. 
1) In this paper, average rewards are considered. Is there any reason to consider the average reward MDP instead of discounted MDP, which is more widely used?
2) It is not clear what the norm || ||_{TV} means.
3) It would be better to discuss what kinds of RLs can be used in Proposition 1. It is not clear in the current paper. 
4) It would be better if the definition of intrinsic reward and extrinsic reward is explained in this paper. 
5) It is not clear how the imitation learner can learn the expert policy from the intrinsic reward in section 3. 
It would be better if the process is explained in more details.
6) In the experiment, it is not clear how the comaprative analysis is done. Because the authors did not develop new algorithms, 
it would be better to explain details of the comparative analysis. 
 7) Overall, the organization of the paper can be improved further to more clearly deliver the ideas by adding more detailed backgrounds. 
summary_of_the_review: In sum, the paper seems to contain potentially interesting results. 
The overall organization and presentation can be further improved to improve the readability of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper contributes a theoretical analysis of imitation learning (IL) under deterministic experts. The paper shows that IL in this setting can be reduced to RL with a stationary reward, and the stationary reward minimizes the total variation distance between the expert and the learner. Several empirical experiments were presented to validate the theoretical analysis.
main_review: Strength:
1. The paper shows that imitation learning (IL) may be reduced to RL with stationary reward for deterministic experts. The analysis provides the theoretical grounding for several existing IL methods. The paper also contributes the connection between the stationary reward to minimizing f-divergence typically seen in adversarial IL, proving that the stationary reward minimizes total variation distance.

2. The proof is clear, well motivated and easy to follow.

3. The experiment studies an important aspect of IL, which is the relationship between the amount of expert data and the IL's performance.

Weakness:
1. Similar to previous works, the practical algorithm uses soft reward (1 - min L_2 distance) to avoid reward sparsity. While a very reasonable choice, the soft reward violates the critical assumption of deterministic expert. Could the authors explain the implications of the soft reward on the theoretical results?

2. The empirical evaluation is limited, with several related questions unaddressed. For instance, the SAC expert appears to be rather weak (in terms of its performance) across different environments. It is unclear whether similar patterns would be observed if the expert matches the level of the performance observed in previous works (e.g. https://github.com/berkeleydeeprlcourse/homework/tree/master/hw1/experts). Another question is how the expert is trained has any impact on the IL performance. For instance, SAC implementation typically directly constraint the range of the action, while a Gaussian expert only clips the action as a post-processing step. In my experiences, these details have quite significant impacts on IL's performance and I hope the authors could expand on the experiments to further validate their theoretical results.

Questions:
1. Comparison between Wang et al. 2019 vs the 1-min L_2 distance reward. Could the authors motivate why the latter reward is used and how it compares to Wang et al. 2019?

2. Could the authors further elaborate on the assumption of the deterministic expert? E.g. How much does it matter in the practical algorithm? Is it even possible to differentiate between stochastic vs deterministic one from limited training data?
summary_of_the_review: The paper shows that, for deterministic experts, imitation learning (IL) could be reduced to RL with a stationary reward. The paper thus provides theoretical grounding for several existing works. In addition, the paper also draws connection with f-divergence minimization commonly found in adversarial IL. On the other hand, the empirical results are consistent with those from existing works, but are limited in its scope. Despite the flaws, I think the work is still interesting to the community. It would be nice if the paper could include more experiments to further support the theoretical analysis.
review_title: null

Meta-review: The paper provides theoretical bounds for imitation learning with rewards (algorithm from Wang et al. (2019)). The bounds/proofs are highly novel and a very interesting contribution to the community, even though they are a lot more conservative than what is observed in practice. All reviewers agree on this point.
It is laudable that the authors also additionally provide an experimental evaluation. After the revision and the discussion, quite a few of the reviewers are still not 100% convinced about them, on the one hand as they would have liked to see more tasks, and on the other hand due to concerns about the reward relaxation (i.e., doesn't match the assumptions in the theorems any longer) which is required for experiments on standard benchmarks.
In the final answer the authors provide evidence that there is no big discrepancy, which is good enough (given that there don't seem to be any alternatives to get around this issue, except removing the experimental section altogether, which would be undesirable). Please clearly point out those limitations of the experiments in the paper and also incorporate this evidence.
Decision: Accept (Poster)