Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper cast the imitation learning problem as an iterative RL-IRL process and proposes a new adversarial imitation learning (AIL) algorithm based on mirror descent. The authors prove the convergence of the learning policy to an optimal policy in $\mathcal{O}(\frac{1}{T})$. The experiments show that the proposed method, MD-AIRL, consistently outperforms RAIRL.
strengths_and_weaknesses: **Strengths**:

Motivated by some practical observations, e.g., the global solution in AIL is hard to obtain by a monolithic estimation process, the authors convert the original AIL formulation to a sequence of policies and associated reward functions, and derive an MD update rule that enjoys a regret bound of $\mathcal{O}(\frac{1}{T})$. This approach is appealing and seems sound, which also provides a linear convergence rate to the optimal policy. The experiments demonstrate MD-AIRL consistently outperforms RAIRL and is more robust to the noise and the number of trajectories. 

Overall, this paper is clearly written and easy to follow.

**Weaknesses**:

While the analysis part contributes a lot, the compared baseline only consists of RAIRL and BC. Some recent work, e.g., f-IRL (in terms of IRL) [1] and ValueDice (in terms of imitation learning) [2] can also achieve a more robust learning process than AIL and AIRL, which are missing in the literature or as baselines. Besides, MD-AIRL extends the AIRL framework, thus the learned reward function may be used in the downstream task as done in f-IRL. However, the functionality of the learned reward function has not been revealed. 

[1] Tianwei Ni, Harshit S. Sikchi, Yufei Wang, et al. "f-IRL: Inverse Reinforcement Learning via State Marginal Matching." CoRL 2020.
[2] Kostrikov, Ilya, Ofir Nachum, and Jonathan Tompson. "Imitation learning via off-policy distribution matching." ICLR 2020.

questions: 
(1) The current theoretic analyses show the convergence to the optimal policy, which seems to come from the nature of MD. Is it possible to establish some results on the learned reward function? Or is it possible to give some analysis (experimental or theoretic) on the learned reward function?

(2) I noticed the BC results in the MuJoCo tasks are very low, while BC can be a very strong baseline on MuJoCo tasks with only 1 demonstration (please see Figure 4 in [1]). Can you clarify the implementation details of BC?


**Minor:**
In Thm. 2 (Line 220), what is the difference between $A_t$ and $A_T$?


[1] Ziniu, et al., "Rethinking ValueDice - Does It Really Improve Performance?", ICLR Blog Track, 2022.
limitations: The authors proposed an appealing approach to make the AIL process more robust. The theoretical result on the convergence rate is of great importance and the experiments show MD-AIRL is consistently better than RAIRL. MD-AIRL learns the reward function, which is also important for the IRL problem, while the authors dig little into the learned reward. Besides, comparisons with other recent IL and IRL algorithms will be helpful.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Inspired by mirror descent, the paper proposes MD-IRL, which considers the reward function as an iterative sequence in a proximal method. The paper proves that such a method ensures robust minimization of a Bregman divergence under the mirror descent framework. 

strengths_and_weaknesses: Strengths:
 - It is novel to consider using MD in the reward update. Rigorous regret upper bound is also provided to show that the Bregman divergence is minimized to a local optimum. In the meanwhile, stepsize considerations are also provided, which is good. 
 - Thorough empirical studies are provided (though limited, see weaknesses). 

Weaknesses:
 - The paper is closely related to RAIRL. I think the authors should make a thorough comparison with RAIRL. 
 - In the empirical study, the paper only compares with RAIRL, which is slightly limited. 

questions: See weaknesses. 
limitations: Yes.
review_title: null

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: The main idea of this paper is to modify adversarial inverse RL to perform mirror descent policy updates. In practice, this means that the regular RL actor objective is modified by minimizing the divergence between policy iterates. The paper proves that this method converges under reasonable assumptions (e.g., Robbins Monro learning rate), and shows that the method does work roughly on par with one prior methods (RAIRL), for a wide range of Bregman divergences.
strengths_and_weaknesses: Strengths
* The proposed method is applicable to a wide range of Bregman divergences.
* Experiments study robustness to varying noise in the dataset and choices of hyperparameters.

Weaknesses
* Poor writing made the paper very difficult to understand. For example, when introducing the theoretical results, it'd be good to explain what question these results are answering before stating the results themselves. Similarly, when introducing the proposed method, it'd be good to explain the problem before introducing the solution (e.g., the dual discriminator is introduced without motivation).
* I'm unsure about some of the claims in the introduction and related works section (see below).
* Limited baselines for the continuous control experiments. E.g., for Fig 7/8/9, I'd recommend comparing to a method like DAC [1], or a more recent+competitive method.

**Summary**: Overall, it seems likely that mirror-descent is the right way to do imitation learning, and that it can outperform vanilla gradient descent in many settings. However, I'm not convinced that this paper effectively makes this point, partially because the writing is very hard to follow, and partially because the experiments do not convincingly show that other Bregman divergences consistently outperform L2 distance (which is effectively what gradient descent uses). So, I think the paper is on the right track, and will eventually make for a strong resubmission after significant revisions to the writing and experiments.


Additional, less important questions/comments:
* I'd recommend running a spelling + grammar checker on the paper.
* L23 -- L30 -- It seems that this paragraph is really about imitation learning, not inverse RL. Indeed, adversarial imitation learning methods do not actually recover the expert reward function, only the expert policy. So, I'm not sure that the claim that these methods "estimate the ... reward function directly" is true.
* Table 1 -- The column about "rewards" seems misleading, for two reasons. First, it's unclear of learning the reward function is actually useful, if the user only cares about acquiring the optimal agent. Second, methods like GAIL do not actually learn the expert's reward function. The column about Bregman divergences seems misleading, too, because BC can be implemented with different Bregman divergences (e.g., the standard BC corresponds to a forward KL divergence). BC also enjoys iterative solutions, and also inherits convergence guarantees from standard supervised learning theory.
* L73 "Pythagorean theorem" -- Is it true that all Bregman divergences obey the pythagoream theorem? I don't think it's true for the KL divergence. The citation [13] doesn't mention Pythagorean theorem.
* Fig 2 is very hard to read. I'd recommend making all subplots a consistent size (e.g., remove the tiny subplots in B) and making the font sizes all larger.
* Eq 14 -- I'd recommend adding a paragraph of explanation before this, explaining the intuition before stating the final objective.
* Eq 15 -- Why is the $\lambda$ parameter needed? From a probabilistic perspective, it seems like $\lambda = 1$ should be optimal
* L263 -- L265 -- What is the motivation for introducing these different Bregman divergences? The proposed method is general, in that it can use any Bregman divergence. But making the claim that this generality is useful requires evidence that different Bregman divergences are useful in different situations.
* Sec 7.1 -- I didn't understand the motivation for this experiment. It seems like the optimal policy is given by the empirical action distribution, which is trivial to compute with gradient descent or iterative updates.
* Fig 5 -- Does the RAIL baseline also decay the learning rate? If the proposed method uses a learning rate schedule, it'd be good to equip the baselines with this schedule, too.
* Fig 6, "goal entropy" -- Why is this a reasonable metric? It seems like some measure of divergence from the expert policy would be more meaningful.

[1] https://arxiv.org/abs/1809.02925
questions: 1. To what extent are the results limited to inverse RL, versus any algorithm that performs policy updates? It seems like the idea of doing regularized policy improvement could (and has been?) studied in a wide range of contexts, beyond imitation learning. To what extent are prior results *already* applicable to this setting, and to what extent do the proposed results extend to other settings (e.g., are they stronger than the results in [1]?)?
2. While it seems like the theoretical results are correct, I'm unsure if they motivate the use of mirror descent. What would similar convergence results looks like for (say) standard/Euclidean gradient descent?



[1] https://proceedings.mlr.press/v97/geist19a.html
limitations: The limitations section is OK. The discussion of extending the analysis to f-divergences is a good point, but not really addressing the limitations of the proposed method. The discussion about "unsafe behavior" and "automation of labor" seem overly broad. I would recommend making more specific limitations about the work (e.g., arbitrary Bregman divergences might not always be useful, and the choice of divergence measure adds an additional hyperparameter for the user to tune).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper derives MD-AIRL, a new algorithm for inverse reinforcement learning as an instance of online mirror descent and proves convergence and bounded regret.
The derived algorithm is closely related to existing methods in the field of adversarial imitation learning, by using a structured discriminator akin to AIRL to estimate the expert densities (but additionally training a state-only discriminatoris to more directy match the state marginals). Furthermore, similar to RAIRL, the algorithm is tested for different Bregman divergences and policy regularizers and uses a density based model for the discriminator. The main algorithmical difference compared to prior methods, is that the discriminator does not directly specify the reward function for the given iteration, but instead is used to estimate the expert policy (which changes in every iteration, motivating the online MD formulation). The update in Eq. 14, hence, corresponds to an additional step that computes the next reward based on the expert policy estimate.
MD-AIRL is compared to RAIRL on a multi-armed bandit problem, a continous point-mass environment and Mujoco environments, and shows slightly improved performance across all settings.
strengths_and_weaknesses: 
## Originality
__Strength__
- The mirror descent formulation seems novel and leads to a slightly different algorithm that can make use of the broad theory around mirror descent.

__Weakness__
- Algorithmically, the changes compared to prior work are not overwhelming.  

## Quality
__Strength__
+ The derivations seem sound and the main claims are proven with sufficient rigor.

__Weakness__
- Proofs on convergence do not consider estimation errors (for estimating log-ratios and for policy optimization) 
- I think some aspect of the algorithm are not fully consistent with theory and not sufficiently well motivated:
1. The second (state-only) discriminator is motivated based on [45, 46], where the GAN setting is considered. I don't see how these paper motivate the particular use of a state-discriminator. 
2. As far as I understood, the regularized reward function (Eq. 15) deviates from the theory. While I understand the motivation behind it (biasing the agent to vist expert states), it is not clear how this additional term affect the theoretical results on convergence and regret bound. I think the algorithm should also be evaluated without the state-reward

## Clarity
__Strength__
- Overall, the paper is well-structured and well-written.

__Weakness__
- Some parts are a bit hard to understand:
1. In Eq.14, it is not clear how $\pi_{\phi}$ is computed from $\psi_{\phi}$. I assume the optimization is actually performed w.r.t. a parameterized policy and then mapped to the reward function (or the policy is directly expressed in terms of $\phi$), however this should be made more explicit. Also, it is not clear how the reinforcement learning relate to the derivations; the MD update in Eq. 7 does not consider time series data. 
2. I don't understand how the main change to prior work (Eq. 14) follows from the MD formulation. After the estimation of the expert densities, couldn't we directly perform the MD update by optimizing the policy with respect to Eq. 7 (using expectations over trajectories)?

## Significance
__Strength__
- The work is a solid contribution to the field by providing a mirror descent formulation for inverse reinforcement learning, along with the corresponding algorithm, which preforms reasonably well.

questions: I only have two questions, but it would be important to answer them thoroughly in the rebuttal:

1. How crucial is the state reward (Eq. 15) for the performance of MD-IRL, how does it relate to the theory? 
2. In  Eq. 14, do I understand it correctly, that the objective is minimized with respect to $\pi_{\phi}$, and the optimal policy is then mapped to a reward function using the regularized reward operator? How does this relate to the paths in Fig. 1? If we know $\pi_{t+1}$ already (because it is the policy that we used to compute $\psi$), why do we need to perform reinforcement learning (cf. Eq.4 and the equation after Eq. 15) if we know the (valid) distribution for the current reward? I understand that we need to mitigate covariance shift and take into account the auxiliary state-reward, but how does this relate to the theory?  
limitations: The paper only discussed the limitation that some divergences are not covered by this formulation.

The paper would be much stronger, if it would thoroughly discuss the *mismatch between theory and algorithm* and clearly state that the *effects of estimation errors* have substantial impact in practice, but are not considered in the current work.
review_title: null

Meta-review: metareview: This work proposes imitation learning via the route of mirror descent inverse-rl. Mirror descent is a well-understood optimization algorithm and framing IRL via it is a good theoretical exercise. Using an expert to help schedule learning is a novel theoretical contribution in the context of adversarial imitation learning. This is directly guiding how to design the approach. 
 
The current concern is that the experimental results are not statistically significant and even though the nice theoretical properties of mirror descent are nice to potentially leverage, it is not coming through strongly yet.  One suggestion is to drastically increase the number of random seeds (say 25) and report 2*standard error instead of standard deviation, especially comparing to RAIRL. The promising innovation is the idea of multiple discriminators which can better account for distribution shift. The authors are encouraged to bolster the experiments with this in mind and frame this as the central point of the work with the theory of MD as supporting evidence. 

The writing of the paper can also be improved and the idea of estimating experts for curriculum can be highlighted better as this is a significant contribution and is currently a bit buried in text. A significant refactoring of Section 4 and a running example that connects Fig 2 and 3 to the algorithm in Section 5 will greatly help. Lines 130-151 are not the main contribution and can be moved to appendix or cut. 

This is also mainly an imitation learning paper and not an IRL paper as the reviewers have noted. While the naming follows the convention of other IL papers like GAIL, RAIRL, it can be a bit misleading. Perhaps the authors can reconsider the name. 
  
confidence: Certain
Decision: Accept