Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper motivates an "inverse online learning" framework to learn non-stationary policy online. The generative assumption is a deep state-space model and it is assumed that the agent acts optimally according to the current belief in potential outcomes and updates its model based on the true outcomes observed. The critical assumption seems to be that of non-stationarity over the policy function as it responds/interacts with the environment based on updated estimates of its belief over the response function. The parameters of the deep state space model are estimated using the ELBO over the log likelihood of the observed trajectories. Synthetic and real-world validation demonstrates some benefit in terms of "identifying the optimal treatment" based on the true model when acting based off the estimates. 
main_review: 1. The setting and description is quite confusing. The goal as stated in Sec 2 seems to suggest the problem is about "inverse reinforcement learning" while also stating the goal is to learn a non-stationary policy from trajectories as in imitation learning under a causal framework. 

2. The paper overall sounds like that of learning non-stationary policies under "misspecification" of the counterfactual response functions (linear in covariates/confounders), which is almost inevitable if we go by a parametric way of solving these problems, and that I believe is a much more transparent setup to motivate the problem. But the whole narrative about "inverse learning" is extremely confusing and in my opinion redundant. If I am missing something, I would ask authors to clarify. The narrative also has the danger of now confusing IRL as a task where the explicit goal is to summarize beliefs over the rewards and policy learning itself.

3. The optimization and inference is not super well motivated. Why does maximizing likelihood the only and appropriate goal here? It makes sense, because the assumption on how the agent is acting based on the current estimates of response functions. In that sense I don't disagree but I believe the presentation needs to be significantly improved and the cost well motivated. Right now the motivation section and inference seems very disconnected.

4. Also validation using synthetic example seems insufficient. Comparison to a stationary assumption is only one baseline. Isn't it important to also focus on gradient updates being noisy due to the model mis-specification of the response functions? Also the more interesting evaluation seems to be critical to show to over time. 

5. Table 2 results are confusing especially comparison to CIRL, what are the "unrealistic" assumptions CIRL is making that is being relaxed in this work? In fact it seems a bit weird to me that CIRL and IOL are not performing comparably modulo non-stationarity. If that is the main issue, (and I am not sure because i am not familiar with CIRL), then it doesn't seem to be about causal assumptions. Can authors please clarify their empirical evaluation more clearly?
summary_of_the_review: Overall I believe the presentation of the technical problem is confusing, could be significantly more clear, cost function/inference better connected to the problem and validation improved. I strongly believe the validation should include some form of real misspecification and then evaluate how the learned policy improves over time. 


Post-discussion update - Thank you to the authors for their clarifications. I have updated my score based on their response.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 
This paper proposes an approach for understanding the decision making of a black-box online learning agent from the a dataset of observed trajectories. Given a dataset of trajectories / decisions made by an agent along with initial context and resulting outcome, the paper proposes a method to perform inference over the generative model of the given dataset using variations inference. The results are used to interpret the decision making of an agent that is learning online. The paper demonstrates its usefulness by highlighting insights into factors that may govern decision on a organ donation acceptance decision dataset.
main_review: This is an interesting paper that presents a new and interesting problem. I think the use of generative modeling and inference to gain insight into decision making of an online learning agent is quite innovative. Overall, I think it is an interesting paper. However, I felt some part of the paper could be clearer. 
Example: CATE: conditional average treatment effect. It is not clear if the paper is introducing this terms or borrowing it from existing literature because there is no citation. 
Also, it seems weird that the perceived utilities of an agent’s decision is given what seems to be a close to a medical term? 

Assumption 2 is not clear at all. The first part of assumption2 is clear but the second part 
“Given a stochastic policy,  t(x) is a monotonic increasing function of the agent’s belief over the
treatment effect ~  (x; t).
”
does not seem to be complete statement. What is monotonic? 

I felt the experiments in the paper were quite weak. The paper shows that the proposed algorithm outperforms existing imitation learning methods in terms of predictive power. However, what would also have been interesting is to see success/failure of existing imitation learning methods for gaining insight into decision making process of organ donation acceptance decision. Is that possible? If no, why not and if yes, what is the motivation behind the new method. 

One of the claims of the paper is to formulate the problem of inverse online learning. While after reading the paper I can guess what the formulation might be. However, I dont think that the paper states the formulation explicitly. As in given the data, we would like to find X. 


	

  
summary_of_the_review: Overall, I like the paper but it can improve on clarity and experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The objective of the paper is an interpretable descriptive model of online learning from trajectories of the form $(x_t,a_t,y_t)$, where $x_t$ is an i.i.d. context, $a_t$ is the action taken that depends on the context and history $(x,a,y)$ up to time $t-1$, and $y_t$ is the outcome resulting from the application of $a_t$ to $x_t$.

The authors' motivating example is liver donation acceptance decisions: a potential donor approaches with an organ they are offering to a particular recipient. The treatment center makes a binary decision to accept or reject the donation based on features of the donor and the recipient. The outcome is the survival time of the patient.

The structure of the authors' model is based on three components:
1. A memory summarization network that maps from the history up to time $t-1$ to the parameters of a multivariate normal distribution over memory vectors. The specific structure is that a new memory vector is produced from the previous memory vector and the observation from the previous time step.
2. An outcome estimation network. This network takes the memory vector as input and produces weights for each action. A dot product of these weights plus the (exogenous) action features yields the agent's estimate of the outcome of each action.
3. The treatment rule is just a logistic function applied to the predicted outcomes.

The model is trained using stochastic variation inference with the ELBO using an LSTM.

Empirically, the authors find that their approach outperforms action prediction baselines such as behavioral cloning, reward-regularised classification for apprenticeship learning and counterfactual inverse reinforcement learning. Additionally, they use their model to analyze shifts in the transplant policy over time, e.g., the declining importance of MELD score and how offers that are rejected make it less likely than a subsequent offer will be rejected.
main_review: Strengths:
1. The framework is a principled approach to the analysis the authors wish to perform in the organ exchange setting (where the linear restriction seems to fit well—maybe? see discussion).
2. The performance of the model "looks good" in the sense that it produces plausible explanations and has a better AUC than the baselines.
3. The authors seem to be the first to identify this problem as of interest (see discussion about novelty in weaknesses).
4. The authors' linear decision-making approach with linear weights might make their approach compatible with linear statistical approaches that are compatible with the medical community.

Weaknesses:
1. In my opinion, the authors' claims about novelty are somewhat overstated. For example, for IRL:
- Many IRL methods can produce an imitative policy as well as a reward estimate (e.g., Ho and Ermon (which the authors cite) and Fu, Luo and Levine). Particularly relevant to the authors' response is section 7.3 of Fu, Luo and Levine where they evaluate on pure imitation learning tasks, finding performance that is comparable with pure imitation learning methods.
- IRL can produce non-stationary policies if the state includes the appropriate observations that would induce non-stationarity. In the authors' case, if the history of (X,A,Y) up to time t is included in the state at time T, the policy produced can be non-stationary in the way the authors desire. This is not that different from what the authors end up doing.
- If the requirement is that a linear decision policy is learned, this can be accomplished by having the neural network produce the action weights as output and then using them as input to the dot product. This is essentially what the authors end up doing.
2. There are many different design decisions about how this procedure could be carried out—from more macro-level decisions like whether the memory should be stochastic or deterministic, whether to use an RNN-type structure for memory (as the authors have done) vs. a GRU or LSTM, to micro-decisions like hyperparameter selection, whether to use regularization, etc. There isn't a lot of justification that appears in the paper, which makes it hard for subsequent authors to know whether the authors tried other ideas and they failed or if this was just the first pass. The authors perform no ablation testing on model features.
3. I am somewhat concerned about the limitations of the linear model. There are many simple decision rules that cannot be captured in a linear model—for example, rejecting a patient if a particular feature is too high/low. I think the authors are a little glib about what linear models can cover.

Post-review: I have lowered my review slightly after reading the other reviews. I thank the authors for their response. 
summary_of_the_review: The paper identifies an interesting problem that takes a different perspective than past work. It covers a lot of related literature but is a bit tunnel-visioned on the problem's perceived newness. The algorithmic approach introduced is a pretty basic tweak on existing methods, but the performance is promising.
review_title: null

Meta-review: All three reviewers suggest acceptance of the paper. The authors study an interesting problem (understanding non-stationary and reactionary policies) and propose a solution to the problem which compares favorably to baselines in experiments. However, some of the reviewers also criticize unclarities in the presentation of the paper and the made assumptions. The authors clarified those points quite well in their rebuttal. Further concerns regarded design decisions and the comparison to failure cases of baselines. The authors addressed those in their rebuttal and promised to include corresponding material in their updated paper. Hence I am suggesting acceptance of the paper. Nevertheless, I would like to urge the authors to carefuly revise their problem presentation in the paper in order to improve clarity and add the promised additional insights to the final version of the paper.
Decision: Accept (Poster)