Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a Bayesian IRL algorithm based on variational inference by learning a policy similar to the one by the demonstrator and inferring the reward function at the same time without sampling. This is achieved by maximizing the ELBO of the posterior of rewards given the demonstrations assuming a softmax probability over Q-values. The contribution is to parameterize the policy and the reward function separately and connecting them through a constraint, which can be reformulated to lead to a single objective.

The paper is well written. Generally, I was particularly impressed by the authors’ explanations of related literature in sections 2 and 3. With one exception. The authors should consider citing Rothkopf & Dimitrakakis, Preference elicitation and inverse reinforcement learning, (2011), particularly, if they cite Ramachandran & Amir, (2007) as well as citing Dimitrakakis & Rothkopf, Bayesian multitask inverse reinforcement learning, (2011) if they cite the hierarchical Bayesian extensions Choi & Kim (2011) and Choi & Kim (2012).

The evaluations are not totally convincing, though. For the final version of the paper, I would encourage the authors to provide simulations on standard IRL problems used in the literature and quantifying performance e.g. with the cumulative loss of the Q-values wrt. the optimal or demonstrator policies. Also, the current manuscript does not provide probabilistic evaluations of the inferred policies or reward functions nor an assessment of how well the variational approximation with GP prior fairs in reward function inference compared to previous algorithms.

Very little details are provided about the implementations, which is a weak point. 

Overall, it would be very helpful if the authors can provide some more details about the implementation and provide evaluations that quantify the inferred reward functions in the rebuttal. With the additional information, this could be a very strong paper that advances BIRL.

The authors are encouraged to release the code to foster open science and reproducibility.  Only the publicly accessible repositories used in this research are mentioned but not whether the authors intend to release their code upon acceptance of their manuscript.

Typo: page 3: “offline setting setting”.
review_title: Potentially great contribution to BIRL, some details missing

final_score: rating: 7: Good paper, accept
review_content: This paper presents a method, called Auto-encoded Variational Reward Imitation Learning (VARIL). The presented idea is to learn a probabilistic (Gaussian) vector representation of the implicit reward function, so that the policy (Q) function can be trained in scale. Authors demonstrated that the presented methods can outperform (recently proposed) existing methods over several benchmarks in the offline RL.

Strong points
- Literature in inverse/offline RL is well illustrated. Especially, limitations of existing methods (impractical for modern, complicated, and model-free task environments) are clearly explained.
- The proposed framework/algorithm outperform existing state-of-the-art in several benchmarks.
- Contributions and presented models are easy to follow. 

Weak points
- The engineering details are not well explained. I agree with authors that an arbitrary function approximator can be used for encoder. However, descriptions as in Figure 1 are too abstract.
- There are some typos (e.g., the the, figure ...)

It would be good to explain more the training procedure in the paper. As an example, it is not sure why the presented encoder is called as auto-encoding reward when the input is demonstration and the output is the policy. 
review_title: A reasonable contributions on Bayesian Inverse RL

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a Bayesian inverse reinforcement learning algorithm that learns a distribution over reward functions from offline demonstrations. The key idea is to avoid solving an MDP in the inner loop by maximizing the likelihood of the demonstrations with respect to a learned Q function, and to fit a distribution over reward functions to minimize the squared Bellman error with respect to the learned Q function. Experiments on an action prediction task in the healthcare domain as well simulated locomotion and video game tasks suggest that the proposed method is competitive with prior methods for imitation learning.

Overall, I think this paper explores an interesting and impactful idea. There are similarities to the inverse soft Q-learning method [1] for learning the demonstrator's internal dynamics model, which involves simultaneously fitting a Q function to maximize the likelihood of demonstrated actions and to minimize the squared Bellman error with respect to a parametric model of the dynamics. The key difference between ISQL and the proposed method is that the proposed method learns the demonstrator's reward function instead of their internal dynamics model, and that the proposed method learns a distribution over reward functions instead of a point estimate. 

While I think this paper is a promising initial step, there are a few issues:

The main issue is that the action prediction experiments show small improvements over the AUC of prior methods on the healthcare dataset, and that AVRIL performs slightly worse than prior methods on several of the control benchmark tasks. One explanation for this relatively poor performance on imitation learning benchmarks could be that standard deep Q-learning does not perform well in the offline RL setting [2], and that AVRIL might share some of these issues because it minimizes the squared Bellman error with respect to learned Q values.

In general, the evaluations could do a much better job of illustrating the benefits of Bayesian IRL vs. standard imitation learning or IRL methods. The current paper evaluates AVRIL's ability to predict actions or imitate demonstrations in the demonstrator's MDP. Instead, one could test AVRIL's ability to learn a reward function that transfers to a new environment with a different initial state distribution or different dynamics -- something that IRL methods can do much better than standard imitation methods. One could also evaluate AVRIL's ability to compute high-confidence bounds on imitation policy performance, as in the evaluations for other Bayesian IRL methods like B-REX [3]. Another possibility is to use AVRIL to learn a distribution over reward functions, then perform risk-averse imitation, as in the experiments for inverse reward design [4]. Similarly, one could potentially use AVRIL as a component in an active learning method for learning rewards, like active inverse reward design [5]. 

The interpretability analysis on page 8 does not highlight the benefits of learning a distribution over reward functions. The confidence bounds in the top of Figure 4 don't vary substantially, and the bottom of the figure doesn't illustrate AVRIL's uncertainty over the learned weights. It would be helpful to see if, for example, AVRIL learned tight confidence bounds for state-action pairs that are close to those seen in the demonstrations, and wider bounds for out-of-distribution pairs. Furthermore, it would be nice to see if different samples of reward weights from the learned distribution yield distinct, but equally plausible, hypotheses to a human supervisor.

Typos:
 - manor -> manner (paragraphs 2 and 3)
 - In the "Apprenticeship through rewards" paragraph on page 2, "Herman et al. (2016)" should use \citep instead of \citet
 - "it's" -> its (last line on page 5)

1. https://arxiv.org/pdf/1805.08010.pdf
2. https://arxiv.org/pdf/2005.01643.pdf 
3. https://arxiv.org/pdf/1912.04472.pdf
4. https://arxiv.org/pdf/1711.02827.pdf 
5. https://arxiv.org/pdf/1809.03060.pdf 

Update after rebuttal
-----
Thank you to the authors for addressing my concerns. I have updated my score.

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: Sequential decision-making is investigated in this paper. A number of demonstrations (i.e. trajectories) is provided to the algorithm that has to lean a policy using only data. Any exploration using sampling from the environment is not possible. To allow for a better understanding of the domain, the algorithm also has to recover the reward that is optimized by the agent that provides demonstrations. In this paper, the authors propose two neural networks that are expected to resemble an autoencoder. One of the neural networks learns a policy that can recover the decisions made in the demonstrations, and the second neural network learns the reward of the policy that is learned by the first neural network. The empirical results show that the new method outperforms alternative methods with respect to the "action matching". The reward learned by the algorithm is not compared with any other approaches.

The overall quality of writing in this paper is at a high, professional level. The authors know related literature, and citations are relevant and sufficient. I noticed very few types that are below.

The approach proposed in the paper is very interesting, and this work could be significant because the medical application requires this kind of method. I am not in position to give a strong recommendation to accept this paper because I feel that the core algorithm is not explained sufficiently in the current version of the paper, and a few important experiments are missing. This position is justified by my major concerns below.

Major concerns:

The biggest problem that I have is that I cannot easily link Fig. 1 with Eq. (11) because the current explanations in the paper leave many details implied. In particular, Fig. 1, shows two neural networks, one for $q_\phi$ and one for $Q_\theta$, but the paper does not explain what are the inputs and what are the outputs in both networks, and which components of Eq. (11) are used to optimize every network. It seems that the learning objective in $Q_\theta$ uses terms 1 and 3 in Eq. (11); this is possible when $\phi$ are frozen at the time when $\theta$ are updated. This must be a coordinate ascent algorithm then. So, the reward predicted by $q_\phi$ is not the input to $Q_\theta$; it is used only as a penalty term in $Q_\theta$. This is confusing because Fig. 1 indicates that R is the input.

Next, let's consider $q_\phi$. It takes (s,a) tuples as input, and it predicts $(\mu,\sigma)$ that define the normal distribution over R for every (s,a). I guess that terms 2 and 3 in Eq. (11) are used as objectives. The KL divergence wants the reward to be close to the prior, and the 3rd term, wants the predictions made by $q_\phi$ to be close to the real differences encoded by $Q_\theta$. So, again, when $\theta$s are fixed, the $\phi$ can be updated. The parameters of one of the networks can be updated, when the parameters of the other network are frozen.

Thus, this summary of the two networks indicates that this procedure is not an autoencoder. There are two independent NNs here, that are trained using coordinate ascent. In particular, if $Q_\theta$ is the decoder, then its error should be backpropagated to the second network, i.e., to $q_\phi$, or at least the decoder should be decoding using the low-dimensional embeddings instead of receiving entire input. This is definitely not the case here. There are two networks that are trained independently, i.e. they don't share the error information during their own backpropagations. I think that these nuances of the algorithms should be made clear; it took me some time to identify these details.

I would further claim that $q_\phi$ is not even required to learn $Q_\theta$ with good predictive accuracy. It would be sufficient to use $Q_\theta$ with the first term in Eq. (11) and then penalize the differences in Q(s,a) that are in the second term without the use of $q_\phi$ at all. $Q_\theta$ would be to learn the same policy, I believe, using simpler regularization. The paper should show the result of such a simplification of the method presented in the paper. This simplification would still use reward to regularize $Q_\theta$, but without a use of a probability distribution over rewards, i.e., without the use of a second NN.

The addition of $q_\phi$ is sensible overall if one wants to have a (normal) distribution over rewards for every (s,a). But still, the impact of having $q_\phi$ should be demonstrated empirically as I described in the paragraph above.

There is something wrong about this sentence: "where given the graphical model in figure 2 the reward can be seen as a latent representation of the policy." In RL, the reward is myopic and instantaneous, and the Q-values have to learned to represent a policy. The Q-values take into account long-term consequences of actions, whereas R(s,a) does have this information in MDPs.

The goal of this paper is to learn reward, but on p. 7 the authors said that in their experiments the performance is measured with respect to action matching", and one of the baselines mentioned on the same page can recover the reward function as well. I feel that the rewards extracted by those methods should be compared with the rewards learned by the new method. This is a missing exploration.

Minor problems:

Introduction is of high quality, but this sentence: "This redirection to obtain policies in a manor we shall refer to as apprenticeship learning (AL) introduces its own challenges, particularly in the offline setting." does not explain apprenticeship learning. Please provide a link with IL and IRL that precede this sentence.

One page 4, "they still" -> "there still"

In bibliography "bayes" -> "Bayes" or "gans" -> "GANs"

Summary:

The authors learn the network $Q_\theta$, and they use another network $q_\phi$ to regularize it. It seems that the same empirical results could be achieved without the second network because the first network could be regularized directly. The experiments showing the results of the simpler method would be highly desirable. The link with the autoencoder is not clear because the two networks that are used in this paper are leaned separately and the original inputs are given to the decoder, i.e. the decoder does not decode using a low-dimensional representation. It is not clear why this approach is equated with an autoencoder. The decoder does not decode using the low-dimensional embeddings, and it uses the low dimensional embeddings for regularization only. The low dimensional embeddings are not even given as input to the decoder.

This is interesting work with a good potential, but the above reasons don't allow me to suggest acceptance of the current version of the paper.

------
Added after the discussion period: Thank you for answering my questions and the lively online interaction. You explained a few things to me, and I could see that you understood my point about VAEs. That was a great outcome. I hope that that you will address the  points that we discussed, where showing an honest link with VAEs will be highly desirable for your future readers. 

review_title: The authors propose two neural networks to solve the inverse RL tasks. These are expected to resemble an autoencoder. One of the neural networks learns a policy that can recover the decisions made in the demonstrations, and the second neural network learns the reward of the policy that is learned by the first neural network. The empirical results are promissing with respect to the "action matching". The reward learned by the algorithm is not compared with any other approaches.

Meta-review: The reviewers agree that the submitted paper is of high quality and provides a promising approach/framework for Bayesian IRL. Certain concerns regarding details of the implementation and evaluation have already been addressed by the authors during the rebuttal phase, and also the title of the paper was adjusted in line with discussions with the reviewers. For the final paper, the authors should make sure to clearly highlight the advances of inferring a distribution over rewards (this is already partly done by the added grid world experiments) and discuss relations to VAEs as the initially had in mind and even in the paper title. Beyond that, the should of course also address other reviewers’ comments.
Decision: Accept (Poster)