Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This work considers a regularized IRL setup, where instead of the entropy regularization used in maximum entropy IRL, an arbitrary convex regularizer $\Omega$ is used. The work presents a number of theoretical results for this general setting, and it is shown that when $\Omega$ is Tsallis entropy, the $RL \cdot IRL$ is equivalent to minimizing a Bregman divergence defined based on the Tsallis entropy and the expert state-action distribution. A practical algorithm is presented for IRL with the Tsallis entropy. A number of experiments are performed to obtain understanding of various components.

I hope that the following questions can be resolved during the discussion period as some things are a bit unclear to me which is preventing me from providing a more detailed analysis of the work.

* Section 3
  * Section 3.1, paragraph after equation 6: Can you clarify (for example with concrete examples or equations) what you mean by terms like "functional-form" and "intractable"?
* Section 4
  * __Could you clarify your structured discriminator? What exactly are you using for $t(s,a,\pi)$? How does this relate to the Tsallis entropy baseline equations derived earlier?__
  * Equation 13: Should this be argmax? I think maybe this should instead be just a small gradient update using this objective since $\hat{t}(s,a;\pi^{(i)})$ is the correct objective locally around $\pi^{(i)}$?
* Section 5
  * __Based on the description in H.2 I don't understand what the Density-Based Model is. Please clarify.__
  * Section 5.1: Can you clarify what you mean by "acquires the ground truth rewards"? Figure 2 is showing different reward curves for each method, so how can they all be the ground truth?
  * Section 5.2: Can you explain how you are computing the divergences between the joint state-action distributions $p(s,a)$ of the expert and trained policy? Also how do you compute these divergences for the Mujoco experiments in section 5.3?
  * Section 5.3: Could you elaborate what you mean by "Unfortunately, RAIRL fails to acquire a reward function that effectively minimizes the target divergence in continuous controls."?

You may also want to cite:
@article{ke2019imitation,
  title={Imitation Learning as $ f $-Divergence Minimization},
  author={Ke, Liyiming and Barnes, Matt and Sun, Wen and Lee, Gilwoo and Choudhury, Sanjiban and Srinivasa, Siddhartha},
  journal={arXiv preprint arXiv:1905.12888},
  year={2019}
}
review_title: Requesting clarifications on methodology and some other components

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper shows a formulation of regularized Markov Decision Processes (MDPs), which is slightly different from that of Geist et al. (2019). Then, the authors propose a novel inverse reinforcement learning under regularized MDPs. One of the contributions is that policy regularization considered here is more general than that of Yang et al. (2019). 

This paper is written very well and is of publishing quality. I think it is sufficiently significant to be accepted. Still, I have the following questions. 

1. The proposed method is based on the relationship between imitation learning and statistical divergence minimization. If my understanding is correct, Bregman divergence plays a role in generalizing generalized adversarial imitation learning. However, as the authors mentioned in Section 6, Bregman divergence does not include f-divergence, which is also studied in imitation learning. Would you discuss the connection to the formulation using f-divergence in more detail?

2. I am interested in the relationship between the proposed method and Lee et al. (NeurIPS2018). Is the proposed method nearly the same as Lee et al. (2018) when Tsallis entropy is selected as regularization? If not, does the proposed method outperform Lee et al. (2018) in the MuJoCo control tasks? 

3. The authors claim that the solutions provided by Geist et al. (2019) are intractable in the Introduction. However, it is shown that the reward baseline term in Corollary 1 is intractable except for some well-studied setups. Does it imply that the proposed method faces the same difficulty when applied with arbitrary policy regularization?

4. The experimental results shown in Figure 3 is interesting, but I have a few concerns. In some cases, the averaged Bregman divergence of RAIRL-NSM (\lambda = 1) was larger than that of Random. Would you show the example of the learned policy for the readers’ understanding? Besides, is the same policy regularization used in Behavior Cloning? Finally, are exp, cos, and sin the meaningful regularizer? 

5. To derive the practical algorithms, the authors consider the same form of the policy regularization used by Yang et al. (2019), which is given by - \lambda E[\phi(\pi(a))]. Is it possible to derive the algorithm in which the regularizer is given by \Omega(\pi)?

review_title: Good paper that generalizes policy regularization in regularized MDPs

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros:
1. this paper studies an interesting problem called regularized inverse reinforcement learning, which is novel to me and brings me some knowledge.
2. this paper first proposes a general solution for regularized IRL; then, TSALLIS entropy is proposed with IRL, as well as an adversary learning-based training strategy. The methodology part seems reasonable and the contribution is good.
3. the paper is well-organized. The experiments are convincing, trying to illustrate the performance under different scenarios, including continuous and discrete reinforcement learning settings.

Cons:
1. in the experiments, much comparison is internal comparison of proposed methods. In the beginning, the authors mention Shannon-entropy regularizers' limitations. The authors should conduct more experiments to prove the statement. Now, only Experiment 3 mentions it. 
2. I don't see many comparisons with other baselines. Adding more baselines is better.


In summary, this paper studies a novel problem, regularized inverse reinforcement learning. The paper proposes several techniques to solve the regularization in different aspects. The experiments are conducted under different settings, but some cons are needed to revise.
review_title: Regularized Inverse Reinforcement Learning

final_score: rating: 7: Good paper, accept
review_content: =====POST-REBUTTAL COMMENTS======== 

I thank the authors for the response and the efforts in the updated draft. I think the paper is stronger and should be accepted.


########################################################################## 
Summary: 
This paper examines the problem of regularized IRL. As opposed to standard IRL, which can have degenerate solutions, regularized IRL has a unique optimal solution. The authors examine different forms of regularization and derive an efficient IRL algorithm which generalizes AIRL and is applicable to continuous control tasks.

########################################################################## 
Reasons for score: 
The  idea of regularized IRL is a nice contribution to the field. It ties in nicely with recent work on regularized RL and overcomes some of the challenges of classical IRL. The mathematical foundation is rigorous and the experimental results are promising.

########################################################################## 
Pros: 1. Nice general mathematical framework that generalizes prior work. 2. Tractable algorithms that work in continuous state and action domains. 3. The empirical analysis of the actual divergence is interesting. 

########################################################################## 
Cons: 1. The paper is a bit notation heavy and that makes it hard to follow. 2. Some undefined notation such as \mathcal{D} on page 2.  3. Lacking in justification for the choice of entropy regularizers. Why Tsallis? Why follow Yang 2019 in choice of regularizer? 4. In general a lot of the theoretical results are given without any intuitive explanation for what they mean or how to interpret them. 5. It's unclear how this is different from the theory for GAIL which is also in terms of a general regularizer.

##########################################################################
Other comments:

The sentences before the discussion mention that RAIRL is good for safe imitation learning. This is unclear since safety is undefined. Also in the same sentences the authors mention that RAIRL fails to minimize divergence, but it seems to work in the figure for some values of q'. What do the authors mean by fail?

Figure 1 seems to transition from having high divergence in the top to low divergence (for  small log \sigma). Why?

Rather than using a single dash for parentheticals, it is usually better to use latex em-dash for parentheticals by using three dashes --- with no space before and after 



review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new method for regularized inverse RL. The paper builds upon work by Geist et al. who studied regularized MDPs with convex policy regularizers. The Shannon entropy is a special case of such a policy regularizer. The paper extends the analysis of Geist et al. for regularized IRL and devises tractable solutions to regularized IRL that only depend on the analytic knowledge of the regularizer. The paper further proposes regularized adversarial IRL (RAIRL), an extension of AIRL by Fu et al., as an algorithm for IRL in regularized MDPs. The algorithm is validated on a number of domains.

The paper is generally clearly written. I believe that the paper is technically correct (and I appreciated that the derivations are well explained in the appendix). The paper is novel to the best of my knowledge and I think better forms of regularization could improve the state of the art in inverse RL and will be of interest to the ICLR community. 

I think it's cool that this paper allows us to solve inverse RL problems in Tsallis entropy regularized MDPs and a great technical contribution. However I think what is less clear to me is why we would want to. As it is I'm leaning towards acceptance because I enjoyed the paper and appreciate the technical contribution but I think the paper would be substantially stronger if it made a better case for why regularizers other than the normal Shannon entropy are relevant for practical IRL problems (perhaps even with an example application demonstrating a clear benefit over AIRL as well as other imitation learning baselines).

Minor comments/questions/typos:
- Would RAIRL-NSM eventually reach expert performance on the ant-v2 task?
- I find the results in figure 4 a little surprising. I would have expected the divergence to be minimal for q=q'. Could you comment further on this empirical result?
- I'm wondering why the exponential/cosine/sine regularizers were chosen in experiment 1. Are these just meant to demonstrate the flexibility of the framework or is there a motivation for using them in practice?
- I appreciate the need to stay within the page limit but would encourage the authors to decompress the paper a bit for a potential camera-ready version. 
- I think it would be nice to include an algorithm box as a summary of the proposed algorithm. 
- Figures 3 and 4 are a bit small and hard to parse. 
- The paper might benefit from some further proofreading for typos and small grammatical errors such as missing articles

review_title: Nice theoretical contribution but the paper would be stronger if it did more to motivate why we would want regularizers other than Shannon entropy.

Meta-review: This paper studies inverse reinforcement learning through the prism of regularized Markov decision processes, by generalizing MaxEntIRL from the negative entropy to any strongly convex regularizer (as a side note, strict convexity might be enough for many results).
The reviewers appreciated the clarity, the mathematical rigor and the empirical evaluation of this paper. They asked some questions and raised some concerns, that were mostly addressed in the rebuttal and the revision provided by the authors.
This is a strong paper, for which the AC recommends acceptance.

Decision: Accept (Spotlight)