Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper studies the reward identifiability problem in inverse reinforcement learning, under a special family of MDPs -- entropy-regularized MDPs. The authors provide some mathematical deductions to show the identifiability is still an issue given entropy-regularization and side information from another policy with a different transition probability or a different discount factor but a consistent reward function may be sufficient to resolve it. They then construct a simple linear-quadratic Gaussian MDP to support their theorems and conduct numerical simulations. The empirical results validate their claims. 
main_review: Originality: This paper makes some conceptual contributions in theorializing some intuitions of reward identifiability in IRL. I like the way the authors employ simple but appropriate mathematical language and examples to explain the idea. However, the idea they convey seems not novel to me, given the prior work from Ng et al. 1999. It is true that the entropy-regularization might be something not covered by Ng et al. 1999, but I wonder how the authors would compare this work with Kim et al., 2021, who study the same reward identifiability problem and deduce some structural properties for MDPs to obtain such identifiability. 

Quality: The problem this paper studies is interesting -- to what degree the reward function is identifiable in IRL. The 

Clarity: This paper is overall well-written. I like the way the authors build up everything step by step. 

Significance: This paper may be interesting in terms of how the authors conceptualize the problem. But in my opinion, it is at least not as significant as Kim et al., 2021, who go beyond a trivial solution of introducing side information from another optimal policy. I would like to know my fellow reviewers' thoughts on this.  

Kim et al. 2021, Reward Identification in Inverse Reinforcement Learning
limitations_and_societal_impact: The authors did not make extrapolative claims.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors consider the question of recovering the exact reward function from trajectories in inverse reinforcement learning. To this end, the authors consider the formalism of IRL with a maximum-entropy regularized objective which can be seen as a generalization of the MaxEntIRL objective. Using this view, the authors show that the degrees of freedom of the reward correspond to an arbitrary, desired value function. The authors show that the reward function can be fully recovered if the policy is known in two distinct MDPs with sufficiently different transition-dynamics and/or discount factors and give conditions for which this is the case. Similar theorems are given for finite MDPs as well as the linear quadratic regulator. 

main_review: I found the writing to be exceptionally clear and the proofs were easy to follow. One welcome change would be to clarify the contributions early on as they differ from what is typical in the IRL literature. Usually, identifiability is treated as a concern for stability and empirical analysis (of MaxEntIRL, for example) concentrates on the apprenticeship learning problem. In this paper, apprenticeship learning is not a concern as the authors explicitly assume that the policy can be fully recovered from demonstrated data. It is thus also unclear how well the algorithm would fare if expert data is limited and if there are scenarios where the insights from this paper could help recover optimal policies where behavioral cloning is insufficient.

While the given theorem is clear and, to the best of my knowledge, novel, it is also likely to be of limited use in direct applications. Not only do the authors assume access to the full expert policy, they furthermore assume access to the policy in two different MDPs with identical reward functions. Such MDPs can be constructed numerically, but are difficult to construct in practice.  In the background section, the authors claim that the transition dynamics can be recovered from expert trajectories; however, this is inconsistent with the proposed procedure where the dynamics have to be constructed. Furthermore, if the expert is a human demonstrator or a differently specified system, the discount factor is likely not known and should be treated as a modelling assumption. (In fact, human expert demonstrators are assumed to follow hyperbolic discounting, rather than exponential). Asking a demonstrator to provide demonstrations for two specific discount factors is therefore highly limiting. Discussion on how the proposed methods may generalize in the approximate case would be appreciated.

Numerical results are minimal and serve only as sanity checks. No application of the method is proposed. Overall, the contribution of this paper is clear, novel and well supported, but also limited.

limitations_and_societal_impact: -
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper considers the problem of reward identifiability in a maximum entropy inverse optimal control setting when access to the expert policy is available. It is shown that the reward function can only be recovered up to a state-dependent shaping term (which relates to the temporal credit assignment). If access to the optimal policies for two variants (different discount factor or different transition model) of a given MDP is available, it is shown that the reward function can be recovered (up to a constant).
The results are validated on numerical experiments on a finite MDP, with known transition model policies.
main_review: Significance
----------------
Learning reward functions that generalize across different dynamics is one of the main open problems in inverse reinforcement learning. New insights under which assumptions such reward functions are recoverable could help to tackle this problem and would constitute an important contribution.

Originality
--------------
Unfortunately, I don't think that the paper presents any significant new insights. Fu et al. [2017] already noted that the log-policy corresponds to the soft advantage and thus a shaped reward function. They also noted that any reshaping with a potential function $\phi(s)$ results in the same optimal policy, and that this potential function corresponds to the value function of a reward function that can be computed as $r(s,a) = A(s,a) - \gamma E[\phi(s')] + \phi(s)$. Hence, I argue that its neither news that the expert policy alone only allows to recover the reward up to a temporal reshaping (relating to the Value function), nor is it news that the reward function can be recovered when in addition to the policy also the value function is known.

The paper also presents proof that the reward function can be recovered when having access to two optimal policies for different discount factors. While such proof seems to be novel, it is not surprising that the ill-posedness regarding the temporal credit assignment can be resolved by observing policies with different temporal weighting. Furthermore, I think that the derivations are straightforward and the insight little useful in practice.

Quality
-----------
The quality is good. The derivations seem correct, the numerical evaluation (or confirmation) make sense and shows the expected results.


Clarity
----------
The paper is well written. Eq.1 is missing parentheses around reward and entropy (discounting should also affect the entropy). 

limitations_and_societal_impact: Sufficiently addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work presents theory and experiments on the problem of reward function non-identifiability in IRL. The key result is Theorem 1, which entails that the reward function (in the entropy-regularized MDP setting) cannot be uniquely identified given an unknown value function. Following this line of reasoning, the authors argue that it is possible to identify the reward function (up to a constant) provided we can disentangle immediate rewards from preferences over future states (e.g., with data from two agents with different discount factors in the LQG setting). This result is also demonstrated empirically using a small example. 
main_review: This paper investigates a key problem in IRL, which is relevant for important issues, e.g., value alignment. The paper is generally well-written and the theoretical results are novel (to my knowledge) and elucidate why non-identifiability occurs, i.e., the unknown value function, and discusses when the true reward function can be retrieved. This theoretical study could potentially lead to novel downstream IRL methods. On the downside, the paper has issues that I hope the authors can address: 
- The paper is not well-placed wrt the existing literature surrounding identifiability in IRL and the related works section misses out on closely-related prior work, e.g.,  (Kim et al, 2012) discusses conditions under which the reward function identifiable, and recent methods (e.g., Amin et al, 2017; Balakrishnan et al, 2020) have been proposed with non-identifiability in mind. See below for additional references. 
- Can the authors provide some justification for the claim that the function class in Theorem 1 is unique? This is important as other reward function structures may also lead to the optimal policy, potentially giving rise to other forms of unidentifiability? Theorem 2 also relies on the uniqueness of the reward function. Also, can the authors clarify Remark 2? From my understanding, the results in Ng et al (1999) does not imply that non-identifiability solely arises from potential-based reward shaping functions. 
- The experiments serve to support the claims, albeit in a very simple setting, and could be better presented. I suggest the authors reorganize the results; it would be helpful if all the key results were in the main paper rather than split off into the supplementary (e.g., fig 6/7 is in the supp). Fig 2 and 3 take up unnecessary space and can be summarized. 

Overall, the work tackles an interesting problem, but due to the issues above, I remain uncertain and unconvinced about the significance of the work. I hope the authors can address my queries above.

References:
- “Reward Identification in Inverse Reinforcement Learning”, Kim et.al, ICML 2021
- "Repeated Inverse Reinforcement Learning”, Amin et.al., Neurips 2017
- “Towards Resolving Unidentifiability in Inverse Reinforcement Learning”, Amin et.al., 2016
- “Inverse Optimal Control with Linearly-Solvable MDPs”, Dvijotham et. al, ICML 2010
- “Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization”, Balakrishnan et.al,

### Post Response
Thank you for your response and the detailed comparison to other works. I agree that it was not reasonable to compare to Kim et al 2021 in the submitted version, but it would be good to include these comparisons in an updated version. While I think the contributions are of limited practical value at this time, there is some value to the findings and I have raised my score to a 6. 
limitations_and_societal_impact: Yes, the limitations and societal impact are discussed.
review_title: null

Meta-review: Two reviewers recommended rejection of the paper (1x reject, 1x weak reject) and two reviewers recommended (weak) acceptance of the paper. Initially the reviewers raised concerns regarding novelty, significance, and relation to existing work. This was acknowledged by reviewers and some of them increased their score for the paper. I discounted the concern regarding comparison with Kim et al. as this paper was not available by the deadline of this conference (as also indicated by the authors). Overall the discussion of the authors with the reviewers and my own reading of the paper led me to the conclusion there are some interesting contributions in the paper and I am therefore recommending acceptance of the paper.
Nevertheless, the paper in its current form does not place the paper clearly in the context of existing results (as evidenced by a very short related work section and the initial confusion of the reviewers) and I strongly encourage the authors to make the relation to existing results more clear in final paper (e.g., by incorporating the responses they gave to reviewers during the discussion period).
Decision: Accept (Poster)