Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the problem of imitation learning, building on the recent IQ-learn framework. Instead of an adversarial reward-policy loss like GAIL, IQ-learn instead parameterizes the Q-function so that the policy can be directly extracted. While IQ-learn works fine, the paper notes that some of the practical tricks don't match the analysis, e.g. regularizing both the expert and the imitator (which also should prevent direct extraction of policy in theory).

In this paper, the authors present Least Squares Inverse Q Learning (LS-IQ). LS-IQ patches some of the gap described above with IQ-learn. 

- First, LS-IQ shows that the mixture regularizer is, naturally, regularizing a chi-squared divergence between the expert and the mixture of expert/policy occupancies. In practice this is good because the mixture ensures that the divergence is bounded.
- Second, they use a least-squares RL perspective to figure out how to properly treat absorbing states.
- Third, they propose using a regularization critic, to account for the additional regularizer term in the objective.
- A few more tips and tricks, e.g., replacing bootstrapping target with fixed target, learning from observations (with no action information) by training an IDM.

strength_and_weaknesses: Strengths
- the biggest strength of the paper is that it is very practical: it identifies many tips and tricks for making IQ-learn better. This is a great contribution, given the already strong results of the base IQ-learn algorithm.
- some of these tricks are somewhat theoretically grounded. Though (as the paper notes itself) some parts are not super convincing, e.g. fixed Q target, it is still useful for future work.
- the experiments have ablations of using different subsets of the proposed tricks.

Weaknesses
The first few pages were find to read, but the rest of the paper (esp section 3) was poorly structured in my opinion. It reads like a laundry list A, B, C, and the reader has no idea what to anticipate next. Some of the insights feel like offhand remarks that have little relevance to the rest of the paper, and are hard to distinguish from the actual important insights. I would try to restructure the paper so that the most important parts are emphasized (even repeated), and move more minor details to the appendix.
clarity,_quality,_novelty_and_reproducibility: Clarity is subpar. Quality, novelty, and reproducibility are great.
summary_of_the_review: In summary the authors improve on IQ-learn, the current SOTA imitation learning algorithm on many tasks.
The paper presents a laundry list of tips and tricks, which was hard to read, but shows good practical improvements that should be very valuable to the community.

--------

Update: I have read the author response, and am keeping my evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: An online imitation learning algorithm with implicit reward is proposed, where the learning objective is to minimize $\chi^2$-divergence between the expert distribution and the mixture of expert and policy distributions. The idea of this work is motivated by the practical implementation of IQ-Learn that violates their theoretical derivation from introducing reward regularizer in practice. Authors argue that the proposed learning objective with the mixture-based divergence is more stable since it properly bounds the range of Q (whereas the range is  unbounded when the mixture is not considered). Additional details on absorbing states, a shared critic for entropy and reward regularization, and LfO with Inverse Dynamics Model are considered to improve the performance of the proposed idea. 
strength_and_weaknesses: Strengths
- Authors look into the gap between practical algorithm and theoretical derivation of the baseline and try to minimize the gap.
- Algorithm is shown to applicable to various IL scenarios (online IL, LfO) and seems stable for all MuJoCo tasks.
- Algorithm used a lot existing techniques (dealing with absorbing states, IDM) to improve the performance. 

Weaknesses
- The readability should be improved.
- Whether the proposed learning objective is new or not is unclear. I'd like to argue that the objective (12) has a strong relation with SQIL's objective. 
- The baseline performance is much poorer than the performance reported in the paper. 
clarity,_quality,_novelty_and_reproducibility: For each category, I added my questions below:

### Clarity
- In introduction section, when implicit reward approach is introduced, Kostrikov's DAC is referred to be one of implicit reward approaches. Since DAC uses the discriminator, I think we should move it to explicit reward methods.
- In introduction, authors mentioned more robust extrapolation is possible, but I don't think any extrapolation was tested in the paper. 
- In introduction, the "less variance" cannot be guaranteed through the experiments (I agree that the proposed method may stabilize the algorithm). Similarly, how can we see the exploding Q-functions targets?
- For IDM, there should be the references.
- In Introduction>Related Work> "aforementioned imitation learning approaches" need to be more clearly stated. 
- In Section 2 in Eq (3), the definition of $\mathcal{J}_\rho$ is not defined.
- In Section 3.1., above Eq (7), it may be better to put a short definition of the variation form of $\chi^2$ divergence. 
- In Section 3.1., below Eq (8), what is $r^*$?
- In Section 3.2., I think the authors argue that there are RL perspectives since we minimize Bellman errors, but it seems quite awkward to me. I think this is simply the form of the learning objective and not strongly relevant to RL.
- I couldn't understand the intuition in Section 3.3. The explanation between Eq (13) and Eq (14) needs to be more elaborated. 
- In Section 3.3., Eq (13) and (14) uses subscripts iq and lsiq, but I would use non-italic capital letters. 
- In Section 3.4., it is unclear why we have to introduce hard-Q for explanation. I think what authors have to emphasize is simply about using a single critic for entropy and reward regularizer, which is not relevant to hard-Q.
- Below Eq (17), I think the description should be elaborated. 
- In the sentence "Finally, ~" below Eq (20). Why do we care about data imbalance and do we need to discuss this?


### Quality
- The partition function $Z_s$ below Eq (2) uses integral, but I think it should be defined by using summation since I believe the discrete action space is assumed in Notation section. 
- In Eq (4), $r$ should be replace to the Bellman error in the last equation. 
- In Eq (9), $\pi\in\Pi$ should be replaced to $\pi\in\Omega$.
- In the paragraph above (17), we need to remove "When now".


### Novelty
- Authors argue that the proposed objective is different from SQIL's objective in Eq (12). However, I believe if we use Q+c (constant) instead of Q in Eq (11), we can shift $r_{max}$ and $r_{min}$ by constant and find that the objective becomes exactly the same as that of SQIL.


### Reproducibility
- IQ-Learn is the crucial baseline of this work, and it seems to me that it performs far worse than what was reported from the IQ-Learn's paper. For example, when a single trajectory is used for Hopper in IQ-Learn paper, IQ-Learn successfully imitates the expert's performance. However, we can see that IQ-Learn performs poorly on Hopper in Figure 3 and 4. I wonder why such difference happened. 
summary_of_the_review: Although the motivation of this work is interesting, the paper should become clearer to be accepted. With the current version, the authors' intuitions behind are not straightforward. There's also a baseline reproducibility issue.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides a theoretical analysis of Inverse soft Q-Learn (IQ-Learn) and proposes a novel algorithm named Least Squares Inverse Q-learning (LS-IQ), which outperforms state-of-the-art algorithms such as GAIL, VAIL, IQ, and SQIL. At first, the authors show that the maximum entropy IR objective with the regularizer proposed by Garg et al. (2021) is identical to a minimization of a $\chi^2$ divergence between the expert state-action distribution and a mixture of expert and learner. Then, the authors formulate the IRL objective using the regularizer as the entropy-regularized least-squares problem, which is related to LSGAN. To derive the practical algorithm, the authors modify the loss function for the Q-function to incorporate the term added by the regularizer. 
strength_and_weaknesses: Strength
1. The paper is well-written, and I found the main contributions of this study to be novel.
2. The proposed method provides the correct way to deal with absorbing states in inverse reinforcement learning.
3. Show the relation between the proposed method, IQ-Learn, and SQIL. 

Weakness
1. The proposed method is evaluated on the MuJoCo benchmarks, but they are relatively easy for modern IRL methods. It is unclear whether the proposed method is robust to distribution shifts between the expert and agent distributions. 
2. As described below, some technical details are not shown in the manuscript. In particular, $r_{\min}(s)$ in Equation (20) should be explained. 

clarity,_quality,_novelty_and_reproducibility: 1. The original regularizer $\psi_{\pi_E}(r)$ is recovered when $\alpha = 1$. In this case, $r_{\min}$ defined in Proposition 3.1 goes to negative infinity. Is it the main reason the original regularizer causes instabilities in continuous action spaces? 

2. The difference between Equations (13) and (14) is interesting. However, I am unsure whether $\tilde{Q}(s, a)$ converges or not when the agent policy converges to the expert policy. Although Figure 1 shows that the proposed method deals with absorbing states correctly, it would be better to discuss the convergence of $\tilde{Q}(s, a)$ at the absorbing states. 

3. In practice, sampling from $d_{\pi_E}$ or $d_{\pi}$ is infeasible. Would you explain how the training data, especially $\mathcal{D}_\pi$ is collected?

4. I do not fully understand the discussions given in Section 3.5. For example, if the minimization of Equation (13) is challenging, minimizing $E_{d_\pi}[(r_Q(s, a) – r_{\min})^2]$ is also challenging. In addition, $r_{\min}(s)$ is introduced in Equation (20), but it is not defined before. Is it a parameterized function of the state? 

summary_of_the_review: I think this paper studies an important problem and has some interesting theoretical results. Moreover, this paper is well-written, and the contribution is quite impressive.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focus on addressing the reward bias issue of a SOTA imitation learning method called IQ-Learn that directly uses an implicit reward function. Specifically, this paper proposed an implicit reward regularization approach called Least Square Inverse Q-Learning (LS-IQ), which uses a modified inverse Bellman operator to address the reward bias issue of IQ-Learn. LS-IQ achieves this goal by using a squared L2 penalty on the mixture distribution of expert and policy. Empirically, LS-IQ outperforms some baseline methods on standard continuous control tasks.  
strength_and_weaknesses: ## Strength

+ This paper noticed that the squared norm regularization on the implicit reward function is effective in imitation learning, but lack of theoretical analysis. Interestingly, this paper uses the regularizer under a mixture distribution of the policy and the expert, and understand the learning procedure in a illuminating perspective. The original objective was understood as a least-squared Bellman error minimization, resulting an minimization of $\chi^2$-divergence between the expert and the mixture distribution.   

+ Given theoretical analysis of the regularizer, this paper indicates some sources of instabilities of the IQ-Learn approach: the arbitrariness of the Q-function scale, exploding Q-function targets and reward bias. Then, Least Square Inverse Q-Learning was proposed to addressed these issues. 


## Weakness 

+ The proposed method are only evaluated on online imitation learning tasks. To fairly and fullly compare the proposed method to IQ-Learn, I suggest the authors to also evaluate the methods on offline settings as in the IQ-Learn paper.

+ The proposed method, LS-IQ, stabilize training by providing fixed targets for the Q-function on expert states, and properly treat absorbing states. This work also introduces an entropy-regulariation critic. However, there is no ablation to study the true effects of each component. 

clarity,_quality,_novelty_and_reproducibility: ## Clarity

Most part of the paper are well written, but the clarity can be improved. Some parts are not clear. Give some suggesions bellow:
+ In Section 3.6, why not introducing your method using Fig. 2. How can a IDM training online with the collected data can infer action for expert observations? how to guarantee the correctness of the predicted expert actions?


## Quality
Motivated by theoretical analysis and derive the novel objective of LS-IQ, I can believe the quality of the proposed method is good. 

## Novelty
The proposed method is novel.

## Reproducibility
Need more details to reproduce the results.

summary_of_the_review: Overall, the proposed method is well motivated by theoretical analysis. I expect the authors to further improve the clarity of the paper and address my concerns above. 


======After rebuttal===
Thanks the authors for addressing my concerns. I agree to raise my rating to 6. Hope to see more details and open source code for better reproducibility.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper identifies a few issues of a SOTA imitation learning---IQ-learn (such as reward bias), and propose a new algorithms LS-IQ which addresses these issues. Empirically, LS-IQ outperforms strong baseline methods on standard continuous control tasks.

All reviewers agree on that the problem studied is interesting, and the proposed algorithm is practical and achieve great performance, thus we are convinced with the contribution of this paper. The author response and the additional experiments added during the discussion phase also well addressed a majority of the reviewers concerns. We thus recommend acceptance.
justification_for_why_not_higher_score: There remain some clarification concerns among all reviewers.
justification_for_why_not_lower_score: The problem identified is interesting, and the result is convincing. We believe the results worth publishing.
Decision: Accept: poster