Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies an inverse (linear) contextual bandits (ICB) problem, where, given a $T$-round realization of a bandit policy’s actions and observed rewards, the goal is to design an algorithm to estimate the underlying environment parameter, along with the “belief trajectory” of the bandit policy. A particular emphasis is placed on the belief trajectory being “interpretable” and capturing changes in the policy’s “knowledge of the world” over time.

The paper’s main contributions are (i) formalizing the inverse contextual bandits problem, (ii) designing two algorithms for this problem based on two different ways of modelling beliefs of the bandit policy, and (iii) providing empirical illustrations of how their algorithm can be used to investigate and explain changes in medical decision-making over time

main_review: **Problem definition and setup**:

I found much of the content in section 2 to be unnecessarily general. Indeed, the preliminaries describe a reinforcement learning framework, but only the contextual bandits setting is considered in the paper. I found this additional notation to be distracting to the paper, and would suggest the authors introduce only background that is used in the paper. Additionally, this unneeded generality makes explanations in various parts of the paper more complicated than needed -- in particular, Remark 1 would not be needed if the RL framework were not introduced.

Additionally, I found the objectives of the paper to be rather vague. Specifically, the authors place significant emphasis on the interpretability of the output of an ICB algorithm, but do not seem to describe what interpretability means. I think it would be useful to describe some desired or defining characteristics of an interpretable output, and to discuss how the belief trajectory, as the authors have modelled it in their two algorithms, does or does not satisfy those characteristics.

**Relation to related work**:

I am a bit confused with how this work fits into/differs from some prior works mentioned in the paper. In particular, I was quite confused why one of the three main contributions of the paper was to formalize the ICB problem, but the IRL problem (which, as I understand it, is strictly more general) has already been formalized and studied. Could you elaborate more on how your problem setting and approaches differ from, e.g., [39] (Bayesian inverse reinforcement learning)? In what ways are you solving a different problem, or improving on previous works on IRL in your contextual bandits setting? Why should one expect the performance in Tables 3-4 of the algorithms you propose to be better than those from prior works (e.g., B-IRL)?

**Experimental results**:

While I find it interesting that, in the OPTN experiment, you are able to associate (external) policy decisions with changes in feature importances, I have a number of questions/comments about the interpretability of the algorithm output, and the applicability of the model considered in the paper.

Contextual bandits model: I think it would be useful to explain (i) what the rewards represent in the context of this experiment (I couldn’t find this explained anywhere, but I might have missed it) and (ii) why it is reasonable to assume that $\rho^*$ does not vary over time. I was confused on point (ii) for a while, since I would expect most bandits algorithms to converge to a fixed $\rho^*$. In the context of your experiments, this would correspond to the relative importance of the features converging to a single value. Could you explain why one should not expect this to happen in your experiments?

Explainability of the results: I think that the paper could benefit from more explicitly outlining the desired properties of explainable results, how their method achieves this, and various ways that the method could be used. Indeed, in order for the results of Figures 3-4 to be “explainable” or “interpretable,” it seems that external context (such as that given in Figure 4) is crucial. Even then, only general trends can be seen from the figures displayed in this section. It is not clear to me how one might interpret such trends in this way when the number of features is large. Do the authors have more general ideas for interpreting these results? How might interpreting these results change when using Algorithm 1 vs Algorithm 2 in your paper? It may be useful to discuss these in the paper.

summary_of_the_review: While the problem setting seems interesting, I think that there are a number of issues indicating that this work is not yet ready for publication. Indeed, the problem setting should be more concisely described, and the relation of this work to prior works on IRL should be discussed more thoroughly. Additionally, I think that the authors should more clearly outline the goals of the interpretability of their method, and outline concrete solutions others could use to interpret the belief trajectories in practice.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of Inverse Contextual Bandit.  They raise an important question: given demonstrated behavior
from an agent, how has the agent’s knowledge been evolving over time? Formally, Given a contextual bandit problem
$(X, A, R, T )$,  where $R, T$ are unknown to the agent. Given an observational dataset $D$, and a family of reward parameterizations $P$ and belief parameterizations $B$, the inverse contextual bandits problem is to determine the true environment parameter $\rho^*$
and the belief parameters $\beta_{1:T}$. 

They propose two algorithms to learn these parameters. The first uses the agent’s knowledge in terms of Bayesian update. The second uses the Gaussian process. They demonstrate their algorithm through simulated and real-world data for liver transplantations.

main_review: Strengths: the considered problem is significant. Their model gains many important properties compared to related works. As they mentioned,  this is the first formal attempt at learning interpretable representations of nonstationary behavior. Using Bayesian methods to compute posteriors is valid. In addition, the experiments for liver transplantations are interesting.

Weaknesses: The presentation is hard to follow. It should be improved.  There is no new technique in this paper and there are no theoretical guarantees for their proposed algorithms. It would be interesting if the authors can upper bound the error $|\rho^* - \hat{\rho}^*|$  to show that their algorithms converge.   
summary_of_the_review: Based on strengths and weaknesses as I mentioned above, I think this paper is on borderline. Now I am leaning towards acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present the ICB an offline method for providing an interpretable description of observed decision-making whilst capturing the agent's non-stationary understanding of the world.
main_review: # Abstract

- Style: you don't need to quote the ICB abbreviation; you introduced it for the first time so not only do you not need to quote it that's also not how abbreviations are defined.

# Introduction

- Style: pick quoting or emphasis, don't use both w.r.t. "descriptive modeling".
- I find your core application interesting. I wonder w.r.t. introduced policies if you attach a particular reward to a particular organ allocation policy? Indeed one may find that a policy is not that good at all so then one would have to revert back to the previous observation regime and in so doing, have introduced an unnecessary non-stationary regime.
- Style: please put your desiderata in bullets, it makes it easier to read.
- What do you mean that the environment needs to be learned? I.e. which part are you referring to of the 3-tuple that is a contextual bandit: $\langle A,S,R \rangle$? Please be precise. It is currently not clear what you mean (I think you mean the action-indexed reward-distributions, but pretend that not everyone reading this is a bandit pro).
- Figure 1: I cannot help but to think that this is an overly complex way of depicting non-stationarity. You may as well just show a Markov chain of rewards and simply say that the directed edges propagate through different regimes and ergo the model as stands is non-stationary. Effectively just an HMM wherein each state your agent takes actions. But alas, I may have misunderstood your intentions with that figure.
- Isn't an ICB simply a structural (in time) bandit conditional on a previous bandit which acted in the same domain (though crucially not in the same environment)? Consider bandit one ($t=1$): $\langle A_1,S_1,R_2 \rangle$ and bandit two ($t=2$): $\langle A_1,S_2,R_2 \rangle$ it sounds as if you're suggesting updating the reward distribution at time two as: $p(R_2 \mid A_1, A_2, R_1)$. One bandit is conditional on a past bandit which also took actions in the same system.
- What does "evolves smoothly" mean? Do you mean that you're able to take derivatives and that the environment is Lipschitz continuous (the former is strictly a subset of the latter) or do you mean that there are no discontinuities?

# Inverse contextual bandits

- Notation: perhaps you should stick to convention w.r.t. to the state-space and denote it with $\mathcal{S}$ as is common.
- $\mathbb{D}$ sounds awfully a lot like an MDP - what's the difference?
- "In this work, we consider state transitions that occur independently of past states and actions" - that's an interesting assumption. What's the logic there? We assume that the organ allocation problem operates on the same dynamical system (your 'environment') no? If so, then tracking the non-stationary behaviour, without taking past allocations into account, is akin to saying that your bandits are all independent. But if that is the case why consider contextual bandits at all (I.e. why include $\mathcal{S}$)? Your assumption alone allows you to simply stack up 'normal' bandits $\langle A,R \rangle$ Which it to say you are prescribing more complexity than you are actually using - why?
- It is **not** ok for you to cite research nr [23] (by Y. Qin, F. Imrie etc) which is not yet publically available, not even on arXiv, for us (reviewers) to corroborate your claims.
- You say "Distributions of newly arriving organs are largely independent of prior allocation decisions; [...] [23, 32, 33]" -- figure 2 and section 3 of [33] very much seem to refute your statement. It would be helpful for you to have a similar diagram and point to the d-separation that you claim impose the conditional independence. At present, it sounds implausible.
- For definition 1, one can you please make use of equation environments to show your maths -- you have superscripts to subscripts and it is all becoming a bit too small.
- You have now defined ICB about four times, once is enough.
- Can you please clarify this statement: "In this first work on modeling non-stationary agents," -- non-stationary MABs have been studied for decades so I do not think that's what you mean, hence, please clarify.
- Table 1 is great, good job, very informative.

# Illustrative examples

- Figure 3 is great.
- Figure 4 is great. I think it would be better if you had a similar figure or that figure closer to the top; it really captures the problem you are trying to solve. So placing it closer to the beginning of the paper may help the reader get the gist of your paper early on.
summary_of_the_review: This is a good and interesting paper. But it is questionable if this could not be achieved with non-stationary MABs or indeed; if past organ allocations have no impact on present allocation, simply running one MAB after another (though it is not clear why we cannot do this). Simply, their novel question in the second paragraph of the introduction is not novel and has been asked many times before.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper formulates and studies the problem named “Inverse Contextual Bandits (ICB)”, which asks “Given demonstrated behavior from a decision-making agent, how has the agent’s knowledge been evolving over time?”  Then, the paper proposes two concrete learning algorithms, imposing different specifications regarding the agent’s behavioral strategy. Finally, some simulations using both simulated and real-world data are provided, showing how ICB can be applied as an investigative device for recovering and explaining the evolution of organ allocation practices over the years. 
main_review: There are many Strengths in the paper:
- the paper tackles and formulates the important and relevant problem of ICB
- the paper is very well-written and the motivation was easy to follow
- related work is covered rigorously
- simulation setups are clearly stated and seem comprehensive 

A weakness might be that the implementations used in the experiments are not shared at this moment. I would love to see an effort to ensure reproducibility (sharing the implementations as supplementary material). 

Note here that I’m not the expert in this sub-field, so I might miss some critical merits or weaknesses of the paper. 

summary_of_the_review: Overall, the paper is well-written; the motivation, related work, and the experimental part are all easy to follow. I do not find critical flaw in the paper at this moment, but it is very likely that I might miss something. I would recommend weak accept, but I will pay attention to the other reviewer's opinion.
review_title: null

Meta-review: Summary: This paper studies an inverse (linear) contextual bandits (ICB) problem, where, given a $T$-round realization of a bandit policy’s actions and observed rewards, the goal is to design an algorithm to estimate the underlying environment parameter, along with the “belief trajectory” of the bandit policy. A particular emphasis is placed on the belief trajectory being “interpretable” and capturing changes in the policy’s “knowledge of the world” over time.

The paper’s main contributions are (i) formalizing the inverse contextual bandits problem, (ii) designing two algorithms for this problem based on two different ways of modelling beliefs of the bandit policy, and (iii) providing empirical illustrations of how their algorithm can be used to investigate and explain changes in medical decision-making over time

Discussion: This paper has received high quality, long and detailed reviews that highlighted some flaws, in particular in the well-posedness of the problem and the clarity of the writing. The authors' response was long and detailed as well, and its quality was recognized by the committee. 
However, the consensus is that this work would require a full pass allowing to include most of the feedback received in the main text rather than in appendices, to discuss related problems in the literature in more depth and perhaps to refocus the exposition on the problem considered.

Recommendation: Reject.
Decision: Reject