Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
This work proposes an approach for understanding and explaining decision-making behavior. The authors aim to make the method 1) transparent, 2) able to handle partial observability, and 3) work with offline data. To do this, they develop INTERPOLE, which uses Bayesian techniques to estimate decision dynamics as well as decision boundaries. Results on simulated and real-world domains show that their method explains the decisions in behavior data while still maintaining accuracy and focuses on explaining decision dynamics rather than the “true” dynamics of the world.

Strengths:
- This work tackles an interesting problem. The proposed setting that considers interpretability + partial observability + offline data is important and reflective of many real-world problems so an approach that works in this setting is very meaningful.
- The paper is well-written and clear. The authors do a good job clearly stating the motivation for the work and differences from prior work.
- The authors consider both simulated and real-world data. The application to healthcare is useful and interesting. Overall, the evaluation helps support the claims, except for the interpretability component described below.

Weaknesses:
- My biggest concern with the work is that it argues for interpretability but the best way to evaluate interpretability is through human evaluation. I appreciate that the authors include the visualizations and talk through what each point means, but it wasn’t 100% clear whether it was truly interpretable. The best way to judge this would be to compare non-interpretable and interpretable techniques and have humans blindly rate which made more sense.

Other comments:
- In Section 3 under Learning Objective, is it reasonable to assume access to the state space?
- In Section 5, you talk about counterfactual updates. Can you describe this in more detail? This wasn’t super clear to me.
- Adding to the weakness point above, the visualization in Figure 3 does not fully make sense. I understand the points and the arrows, but it’s not very clear whether the point’s exact location in the belief simplex is meaningful (e.g., in the middle vs on the line between MCI and dementia).
- It looks like the action matching metric is used to evaluate the ability to imitate and the belief/policy mismatch is used for evaluating explainability. But belief/policy mismatch cannot be used for ADNI, so for this domain, there’s no proper evaluation of explainability other than the visualizations. I think this is a key piece that needs to be evaluated and shown.

Recommendation: 
Overall, the work poses an interesting problem and solution, but the key motivation is to develop an interpretable approach, which I don’t think is sufficiently and correctly evaluated. Thus, I’m on the fence and would like to hear from the authors about this point.

=========================

Response after author rebuttal: 
The authors answered my concern about evaluating the interpretability of the approach. They evaluated the method with a few clinicians, and I'm glad to see that they preferred the authors' method.

Adding these results + clarifying the points I included will definitely make the paper stronger. I increase my score as a result and recommend acceptance.
review_title: Official Review #3

final_score: rating: 7: Good paper, accept
review_content: The paper proposes an algorithm for learning policies and internal models ("decision dynamics") from demonstrations. The key idea is to fit a distribution over policies, observation models, and transition models using an EM-like method. Offline experiments on a healthcare dataset show that the method learns interpretable decision dynamics, recovers biased internal models, and accurately predicts actions relative to prior methods.

Overall, the paper is well-written, the experiments are convincing, and the proposed method, Interpole, makes a meaningful contribution to the literature on modeling decision-making.

It would be nice to evaluate Interpole on tasks in which the demonstrator's decision dynamics operate on high-dimensional latent belief states, to illustrate how Interpole scales with the complexity of the demonstrator's internal models. In such settings, would the mean-vector representation in Equation 2 become problematic due to the curse of dimensionality?

There is some missing related work on learning from demonstrations that also satisfies the three criteria of transparency by design, partial observability, and offline learning: assistive state estimation [1], learning sensor models from demonstrations in the LQG setting [2], learning from a demonstrator with false beliefs [3], and inverse rational control [4].

1. https://arxiv.org/pdf/2008.02840.pdf 
2. https://fias.uni-frankfurt.de/~rothkopf/docs/Schmitt_et_al_2017.pdf
3. https://arxiv.org/pdf/1512.05832.pdf 
4. https://arxiv.org/pdf/2009.12576.pdf 

Typos:
 - "log-like-lihood" -> log-likelihood (page 5)

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a method for obtaining an interpretable representation of some behavioral policy based on partial observations and in an offline learning setting. The paper is well written and the approach is clearly explained. I just have few comments/questions:

In terms of alternative approaches to modeling agent/decision maker behavior, I can think of at least one other alternative and I wonder how your method can be compared to it: ‘Agent Markov Model’ introduced by Unhelkar and Shah (Learning Models of Sequential Decision-Making with Partial Specification of Agent Behavior AAAI 2019) is also modeling agent behavior using a state space model with partial observations. Similar to your approach, they are also interested in modeling the agent and not the actual mechanics of the world. As far as I understand AMM also satisfies three key criteria presented in the paper. AMM also uses a simpler model for the policy; they assume agent follows a stationary Markov policy: \pi(a|s, z). Can you discuss advantages of your model over theirs? 

Another advantage of AMM over INTERPOLE is that it can infer the number of latent states via a nonparametric Bayesian approach. In INTERPOLE, do you have any recommendation on how S can be chosen in a more general setting? How sensitive are the results w.r.t model misspecification (specifically w.r.t. the number of latent state)? 
review_title: Simple and effective idea but missing a key related work

Meta-review: Explaining by Imitating: Understanding Decisions by Interpretable
Policy Learning

The topic is maximally timely and important: Understanding human
decision-making behaviour based on observational data. Any tangible
steps towards this challenging goal are bound to be significant, and
those this paper makes.

A Bayesian policy-learning method is introduced for this task, and
validated on both simulated data and user exeperiments in a real
decision-making task. The novel contribution is on learning
interpretable decision dynamics

The paper is written clearly enough..

The updated paper clarified most major concerns the reviewers had. In
particular, they added a user study.

The biggest remaining weaknesses are that

- relationship to the AMM model did not become completely clear yet

- the real user study has been carried out with only a small set of
users. But a large-cohort study would be too much work to ask for a
paper which has also a strong methodological contribution.

Decision: Accept (Poster)