Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To recover the parameters of an unknown reward function, this work introduces the notion of the Bellman score - corresponding to the (vector-valued) gradients of the log probabilities of the true optimal policy with respect to the true unknown reward. Hence, it proposes an algorithm to estimate this quantity via approximate dynamic programming procedures. The authors show improvements over prior literature performing inverse reinforcement learning and transfer learning in a continuous control benchmark. They also show their algorithm is effective for reward estimation and predicting changes in optimal behavior in an econometrics-based simulation environment.
strength_and_weaknesses: **Strengths**

1. The novelty and contribution of this work are significant.
2. The reward identification and IRL problems are very relevant and have several downstream implications.
3. The evaluation is performed for different problems, highlighting the generality of the procedure.

**Weaknesses/Areas of improvement**

1. I believe the paper does not appropriately address the limitations of the proposed algorithm in continuous/high-dimensional environments. I would imagine challenges related to the deadly triad [1] would provide even greater instabilities when applied to an estimator of a high-dimensional vector. Moreover, using a backup operator with a buffer of stored behavior will incur the 'double-sampling' issue in stochastic environments (e.g. [1, 2]). Currently, there only is a brief mention of the limitations in the Appendix, solely listing minor and non-specific computation-time considerations.

2. I have some concerns about the empirical IRL results. In particular, the authors mention in the Appendix they perform a hyper-parameter search for their method and run it until convergence. In contrast, the authors mention that the baselines use off-the-shelf implementations that were already tuned for the considered environments. However, since they mainly consider evaluating given unusually low amounts of data, I believe that also the IRL baselines' hyper-parameters should be re-tuned for this specific experimental setting and be allowed to run until convergence (of the reward/agent behavior).

3. Some of the stated contributions sound a bit exaggerated without proper context. For instance, the authors claim "[We] derive, for the first time, the gradient of the Maximum Entropy IRL (Ziebart et al., 2008; Finn et al., 2016b; Fu et al., 2017) objective in the fully general setting with stochastic dynamics case and non-linear reward models." I believe this result (Theorem 2) already follows quite intuitively from the work of Ziebart et al. 2010 [3], which would be important to explicitly specify.

4. The authors state "Hyperparameter ablation studies as well as compute time comparisons are in Appendix C." However, Appendix C does not contain any table/performance curve showing empirical results, but only two paragraphs with some general considerations and recommendations. Such missing information would be very important to get an intuition for the sensitivity and generality of the proposed algorithm. Since the authors state "More detailed ablation results will be added shortly.", do they plan to add this data for the rebuttal revision?

4. I believe this work provides far too little information to allow the reproduction of the results. I find this concern of particular relevance for correctly assessing the contribution, given its significant novelty. As the authors mention that "The implementation code will be released shortly.", I would appreciate it if this code could be anonymously shared with the reviewers for the rebuttal phase.

5. I think there is significant room to improve the paper's writing. For instance, I encountered repetitions of sentences even within the same paragraph, e.g. "The theory in this section will be derived in the Maximum Entropy RL (MaxEntRL) setting (Haarnoja et al., 2017; Kostrikov et al., 2020) but similar results for the vanilla RL setting without entropy regularization can be found in the Appendix [...] The theory in the remaining sections will be derived in the MaxEntRL setting for readability. Similar results for the standard RL setting without entropy regularization can be found in the Appendix." Moreover, I found several typos, e.g.: "i.e gradient of", "Gradients of the objective in Eq. 6 is highly", "the jacobian", "The key challenges with this approach are the difficulty of",... I would encourage the authors to perform additional proofreading of the text to improve clarity.

*References*

[1] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.

[2] Zhu, Yuhua, and Lexing Ying. "Borrowing from the future: An attempt to address double sampling." Mathematical and scientific machine learning. PMLR, 2020.

[3] Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.

**Additional questions**

1. The underlying assumption of the proposed method is that the observed behavior was generated by a MaxEnt policy. What would the authors empirically expect to observe if this assumption was broken? Would there be any way to verify if this were the case?

2. Given we might be interested in imitating some demonstrations observing an external agent, without access to the agent's actions, can we leverage the same procedure to recover the score of a hypothetical reward function only dependent on the visited states?

clarity,_quality,_novelty_and_reproducibility: The work has considerable novelty and much potential to impact the RL field. The quality of the analysis (W1, W3) and empirical comparison (W2, W4) could be improved. Moreover, the text could benefit from additional proofreading (W5). Finally, the experiments are not reproducible with the provided resources and information (W4). 
summary_of_the_review: I found the contribution of this work to be novel and relevant, with the potential to have significant implications. However, in its current form, I believe there are several flaws in the paper, undermining its quality, clarity, and potential fairness. Moreover, given the nature of the contribution, I think that providing a way to reproduce and verify the experiments is of particular importance. Hence, I believe this paper is currently borderline, but I am very open to updating my score based on the rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a dynamic programming algorithm ("score iteration") which can efficiently compute Q-gradients, and thereafter, "Bellman scores". This Bellman score is useful because it directly gives information about how changes in reward affects policy. The authors then demonstrate its utility for doing behaviour imitation, policy transfer, and counterfactual predictions. 
strength_and_weaknesses: Strengths: this paper demonstrates strong benefits of using the Bellman score, in particular for maximum entropy IRL and counterfactual predictions. And most crucially, they discover a dynamic programming algorithm that enables its efficient computation. The algorithm is conceptually solid and reasonable, and well motivated by classic ideas in deep RL (esp DQN, bootstrapping, etc). The demonstrations on IRL and counterfactuals were strong. Relevant baselines were compared, and a good variety of tasks were used. 

Overall, for its purpose, it is pretty solid. Develops theory and shows relevant experiments to make a central coherent point about the benefits of efficiently computing Bellman score. 


Weaknesses: More details on exact implementation would be helpful, so that readers can have a chance to replicate and further study their phenomena. 
clarity,_quality,_novelty_and_reproducibility: The paper was pretty easy to read. As discussed above, the results were clearminded and novel. 
summary_of_the_review: A clearminded paper that would be of interest to the community and should hopefully attract further investigation into the benefits of Bellman score in RL, and even better ways to efficiently compute it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper contributes to the study of gradient-based inverse reinforcement learning, in particular, learning a reward function by gradient ascent, under which an observed policy is optimal. The main problem of interest is the estimation of the gradient of the policy return with respect to the parameters of the reward function. It is shown that this gradient follows a dynamic-programming Bellman-type equation similar to the actual policy return, which can then be exploited to derive gradient estimation algorithms based existing RL algorithms. Substantially improved performance is demonstrated on benchmarks.
strength_and_weaknesses: This paper has a strong basic insight, which is utilized well. There are no obvious flaws that struck me while reading.
clarity,_quality,_novelty_and_reproducibility: The paper is quite clearly written and the work seems to be of high quality. I have not complaints about writing and exposition. The underlying idea of gradient-based IRL is not new, but this paper makes basic yet fundamental observations about the structure of the gradient and as a result is able to formulate substantially more efficient algorithms for its estimation. 
summary_of_the_review: This paper makes an important, well-argued contribution to a relevant problem. Novelty and significance are relatively high.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the problems of understanding and learning optimal policies from demonstrations, including reward identification, counterfactual analysis, behavior imitation and behavior transfer. Under the  framework of MaxEntRL, they first propose the concept of Bellman score which captures how changes in the reward functions influence the optimal policies, and further show that Bellman score can be calculated as a difference of the gradients of optimal Q-values w.r.t the reward parameters. Then they introduce the Bellman score operator and prove that applying this operator repeatedly yields the true Q-gradients. Based on this operator, they develop a model-free algorithm Score-learning to learn Bellman score with finite samples. Next, they apply the Score-learning method on tasks of IRL and counterfactual predictions. In experiments, they evaluate that applying score-learning can achieve SOTA performance on the mentioned four tasks.
strength_and_weaknesses: 
Strength:

1. The authors propose the concepts of Bellman score and Q-gradient, which are novel to my knowledge. The proposed Bellman score and Q-gradient have an important role on the problems of understanding and learning optimal policies from demonstrations, including inverse reinforcement learning (IRL) and counterfactual analysis.





Weakness:

1. In section 4.1, they derive a new formulation for the gradient of the MaxEntIRL (i.e., Eq. (7)) and propose a new MaxEntIRL algorithm GAC based on Eq. (7) and Score-learning. However, I cannot see the benefits of GAC over existing IRL methods like GAIL, DAC. 

   The authors claim that existing IRL methods utilize the formulation of Eq. (8) and thus suffer the issues of data reusing and high variance. First, for the data reusing issue, DAC has already addressed this issue by using data in replay buffer to update rewards. 

   Second, I think there is no high variance issue in performing SGD updates based on Eq. (8) and the analogy between Eq.(8) and MC-PG method made by the authors does not make sense. In Eq. (8), we use the sample average of reward gradients to estimate the expectation. Unlike the policy value (a multi-step value with a scale of 0 to T) estimated in MC-PG method, the reward gradient is a one-step value and thus using MC estimates does not have a high variance. Instead, GAC based on Eq. (8) could have a high variance since it only uses initial state-action pairs for estimating.             

2. The computational cost of GAC is large. Compared with existing IRL methods like GAIL, AIRL and DAC, GAC additionally maintains the Q-gradient network, target Q-gradient network and score network. As pointed in Appendix C, GAC requires roughly 2+ more compute time than the baselines.   

3. The authors claim that score learning can be applied in behavior transfer. In experiments, they show that learned rewards from GAC can transfer behaviors to perturbed environments. As far as I am concerned, unlike Adversarial IRL, GAC does not contain special algorithmic designs for handling transfer. It remains a question why GAC can transfer rewards to new environments.    

4. The theory in this paper is derived within the scope of MaxEntRL. The authors claim that similar results for the standard RL can be found in the Appendix. However, I do not see these results in the Appendix. 

5. This paper contains the review of algorithm progress in IRL and IL. However, some basic works on the theory of IRL and IL are missing, see e.g., [1, 2, 3].

References:

[1] Y. Zhang, Q. Cai, Z. Yang, and Z. Wang. Generative adversarial imitation learning with neural network parameterization: Global optimality and convergence rate. ICML, 2020.

[2] N. Rajaraman, L. F. Yang, J. Jiao, and K. Ramchandran. Toward the fundamental limits of imitation learning. NeurIPS, 2020.

[3] T. Xu, Z. Li, and Y. Yu. Error bounds of imitating policies and environments. NeurIPS, 2020.

Minor issues:

1. The first line in page 3: "the the soft Q-values"
2. In definition 3, the shape of the Bellman score should be d \times |X \times A|? 

3. In the line of "update Q-gradient" in Algorithm I, g (x, a) should be g (x', a') under the transition probability of P (x', a' |x, a).

clarity,_quality,_novelty_and_reproducibility: 
Clarity: Overall, this paper is well-writen and is easy to follow.

Novelty: The authors propose the concepts of Bellman score and Q-gradient, which are novel to my knowledge. The algorithm for learning Q-gradient is similar to DQN and thus is not very novel.   
summary_of_the_review: In this work, the authors propose the concept of Bellman score, which is novel. To learn Bellman score with finite samples, they develop the algorithm score learning based on DQN. The Score-learning method has a broad applicability on tasks of IRL and counterfactual predictions. However, the benefits of the developed IRL method GAC over existing IRL methods are not clear. Besides, GAC suffers the computational issue. Some claims in this paper are not well supported. In conclusion, I recommend the score of 5.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces the Bellman score, which is the gradient of the log-probabilities of the optimal policy with respect to the reward parameters, and a corresponding operator to learn the gradient of the Q-function (which is related to this quantity) via fixed-point iteration.
The authors show that score-learning is a versatile approach and derive an actor-critic algorithm for IRL (GAC).  
They empirically study several applications: imitation learning, inverse RL, reward transfer, counterfactual estimation.
strength_and_weaknesses: ### Strengths

This is as far as I can tell a novel, neat idea that is quite versatile.

No knowledge of the environment dynamics is required.

The paper is overall well-written, clear, and to the point. It is a bit dense though, and might benefit from additional figures.

Empirical results on various tasks (IRL, imitation learning, reward transfer) are good.

### Weaknesses

I would advocate for using optimal (or near-optimal) instead of the term rational that I find confusing, e.g. one could think that it means sensible according to human evaluation. Unless this is related to the fact of using demonstrations, in which case it should be better explained.

Nit: Algorithm 1 misses the initialization of the Bellman score

It is not clear to me how the gradient of the reward appears in the Bellman score operator (definition 2). Could authors expand on that and add a clarification in the text?

Algorithm 2 leverages the gradient of the reward. Does that mean that score learning only works in tasks where a reward parameterization exists and is known? In that case, I think the current state of the paper is not clear enough about this. I do get that this is not a limitation for IRL and IL tasks since a reward is to be learned anyway.

I encourage the authors to open-source their code to maximize the impact of their contribution.

clarity,_quality,_novelty_and_reproducibility: Clarity: a few details are missing (see comments), the paper is quite dense, otherwise clear 

Quality: high 

Novelty: novel and very interesting idea

Reproducibility: code is missing, the appendix contains the hyperparameters for GAC

summary_of_the_review: This is an exciting paper with a lot of potential impact.

I have made a small number of questions/comments that could be addressed by the authors.

I recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors of this study make the crucial point that by understanding how variations in the underlying rewards impact optimal behavior, several discussed issues on imitation learning and inverse reinforcement learning may be resolved. This amount can be approximated locally by the Bellman score, which is the gradient of the log probability of the best strategy with respect to the reward. The authors present such a Bellman score operator, which can be used to estimate the score directly and is shown to converge to the gradient of the infinite-horizon optimal Q-values with respect to the reward. Using their theory as a guide, the authors construct an effective score-learning technique that can be applied for score estimation in high-dimensional state-actions spaces. The authors demonstrated that score-learning might be used to transfer policies between environments, accomplish state-of-the-art behavior imitation, make counterfactual predictions, and accurately identify rewards.
strength_and_weaknesses: **Strengths:**
- The paper is well written, and the flow is excellent.
- The motivation is described well.
- Empirical studies are promising and seem to be representative of a successful comparative evaluation. For example, the proposed method achieves near-optimal performance for the IRL experiments with a single demonstration, and the use of 10 random seeds follows the widely-studied deep RL benchmarking standards.
- The approach has some novelty in it. 
- The adaptations of the introduced algorithm to the considered downstream tasks seem credible.


**Weaknesses:**
- The proposed method heavily relies on the _Maximum Entropy RL (MaxEntRL)_ algorithm and the [work](https://arxiv.org/abs/1702.08165) by Haarnoja et al. (regarding notation, background, theory, etc.). While the authors claim to advance the prior art, the additional novelties are unclear and decrease the soundness of the presented work due to such an issue.
- The authors constantly discuss "model-free settings with high-dimensional continuous state-action spaces." However, the experiments are performed on a set of discrete action spaces. I couldn't understand the connection until I looked at the supplementary material in which the SAC algorithm is used for a part of the RL learner, which remains unclear in the main text. 
- Can the proposed method be extended to standard continuous control (e.g., OpenAI Gym benchmarks)? I know it can't (mostly), yet the authors should address that. 
- The preparation of the paper is untidy. This reduces the quality and demonstrativeness of the proposed method. See my detailed comments below. 
- There are many references (maybe some belong to the supplementary text); consider reducing them. For example, the ones in the introduction section, i.e., references to the studies on economics. In addition, most of the references, e.g., the SAC algorithm, point to the pre-print versions. Please replace them with the published versions.
- I suggest the authors define the concepts of reward identification, counterfactual analysis, behavior imitation, and behavior
transfer, rather than only giving examples, as many other readers not be familiar with them.
- Although I believe that the proofs of the theorems are correct, not giving them in the main text prevents the persuasiveness of the math in the paper, which is the backbone of the proposed dynamic programming algorithm. I suggest authors provide at least a single proof in the main text that they believe is the most important.
- Although an extensive set of experiments are performed to evaluate the proposed method and compare it against the baselines, the discussion essentially translates the results in the Figures or Tables to words. Please discuss your comparative evaluations in-depth, such as why your approach outperforms baselines, what they lack, what are the limitations of your approach and, how they can be overcome, etc.

**(Very) Detailed Comments:**
- Last paragraph of section 3: There is a sentence "(We omit the finite-horizon version for brevity, but it is very similar to the infinite-horizon case)". Why didn't the authors show this (or give a reference) instead of briefly mentioning it?
- First paragraph of the introduction: The reward in an MDP may not always capture the goal. Rewards can sometimes represent the intermediate goals that are mandatory to obtain the final goal (or aim of the task). Consider changing the corresponding sentence with something like: "The reward covers the information about the goal..."
- Second paragraph of the introduction: "an RL algorithm..." not "a RL algorithm".
- Preliminaries: An MDP does not necessarily consist of a discrete state and action space. If such a case is considered, please explain why.
- First line on page 3: Duplicated "the". 
- Second paragraph of Preliminaries: Consider referring to Bellman, Dynamic Programming (Dover publications) while mentioning dynamic programming. 
- Line 5 of the second paragraph on page 4: Put a comma after (x, a).
- Second sentence of the first paragraph in Section 4.1: Remove "from" before the dot.
- Put floats on the same page or later when they are first mentioned. For instance, Algorithm 3 is first mentioned on page 6, but the pseudocode is given on page 5. 
- Give a cross-reference to Table 1; what does it tell (e.g., "Table 1 shows the results for...")?
- I suggest the authors not highlight the highest scores in Table 2; make them bold instead.
- Put punctuation after equations if they are used in conjunction with sentences.
- Algorithm 2: "if \psi is not None" should be "if \phi is not None"

clarity,_quality,_novelty_and_reproducibility: - **Clarity:** The paper is well-written, and the flow is excellent.
- **Quality:** The quality of the paper is a bit below the standard since the background, notation, and theory are primarily based on the paper the [work](https://arxiv.org/abs/1702.08165) by Haarnoja et al. See my previous comments in the _Weaknesses_ section.
- **Novelty:** Novelty is also a bit below the standard but can still be acceptable. Nevertheless, I couldn't say this work is a high-impact study. See my previous comments in the _Weaknesses_ section.
- **Reproducibility:** The paper lacks critical information regarding reproducibility since there is no code submitted, and the experimental setup is only provided in the supplementary material. I suggest the authors provide an anonymous repository link (with an anonymous GitHub profile so that no one can identify the authors through the actual GitHub page) and briefly describe the experimental setup (e.g., hyper-parameters, implementation, etc.)

summary_of_the_review: Given my previous detailed comments, I believe this work is consistent in terms of theory, the presentation of the introduced approach, and empirical studies. However, there are still issues with the paper. Specifically, I wouldn't say that this paper is reproducible, as no detailed experimental setup or code is given. Furthermore, the quality and novelty are slightly below the standards for a venue like ICLR. In particular, the theory and background are heavily based on prior work. Only a set of trivial calculus extends the previous approaches. Overall, I couldn't say this work is acceptable unless my concerns are addressed through straightforward comments or references.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces an algorithm that computes two utile quantities, that they call Q-gradient (g(x, a)) and Bellman-score (s(x, a)), given transitions from a replay buffer, .
The Q-gradient can be used to update a reward function in IRL, and the Bellman-score to predict the adaptation of a policy to a change of the reward function.
In both cases, the experiments shows that both applications provides strong results, outperforming the baselines used for comparison.

strength_and_weaknesses: As far as I understand the maths are Ok (I haven't been through the appendix), and the approach definitely makes sense. 
The results are impressive, especially for the IRL experiments with a single demonstration. The experimental protocol is correct (they used 10 random seeds for each baseline and averaged scores of policies over 20 sampled trajectories).
clarity,_quality,_novelty_and_reproducibility: My main critics are regarding the structure of the paper, that makes difficult to guess how and why the different quantities are computed.
For ex, I had to go up to the IRL algorithm to understand why the Q-gradient is needed, and I had to go up to Sec. 4.2 to understand why the Bellman-score is needed. 
Also, I still don't understand how the gradient of the reward is computed for the Q-gradient updates in Eq. 4
summary_of_the_review: Overall, although the paper could be more impactful and would benefit some writing clarifications, I think it already desserve communications and will benefit the RL community. I rate for a marginal accept, but may increase my score.

Minor comments:

- Preliminaries, in the unconditional occupancy measure two lines after Eq1, I think Pi_T should be replaced by Pi_0.

- At the end of page 2, I am pretty sure that the entropy as used in the MaxEntRL setting is only a function of the state, and is the entropy of the policy at a given state. Maybe a comment is missing here.

- Page 3 Line 1: "the the"

- Algo 2: "if \psi is not None" --> "if \phi is not None"

- A related work for IRL that also have strong results with single demonstrations: Primal Wasserstein Imitation Learning by Dadashi and Al.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers agreed that the paper proposes a novel gradient-based algorithm for learning an unknown reward function using new insights about the underlying gradient structure. However, the reviewers also raised several concerns and questions in their initial reviews. We want to thank the authors for their responses and active engagement during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. As a result, the reviewers have an overall positive assessment, and there is a consensus for acceptance. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.
justification_for_why_not_higher_score: The paper could possibly be given a score of "Accept (oral)" based on score calibration with other accepted papers.
justification_for_why_not_lower_score: "Accept (spotlight)" is justified as the reviewers agreed that the paper has high novelty and the results would be of broad interest to the community.
Decision: Accept: notable-top-25%