Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers the problem of reinforcement learning in the presence of unmeasured confounders. In particular, the authors consider a setting where the data is collected according to a behavioral policy $\pi^b: S \times U \rightarrow \Delta(A)$ but the variables in $U$ are unobserved. The goal is to learn a "super" policy which might be history dependent and has return higher than the optimal policy that uses only the observed states in S.

In order to solve the super policy learning problem, the authors use techniques from proximal causal learning. In particular, they assume that there are proxies for both actions and rewards at each time step. If the proxies satisfy certain conditional independence assumption the expected return from a super policy can be expressed through a set of $Q$-bridge functions. These $Q$-bridge functions only depend on observable variables (states in S, and proxies W, Z) and they are estimated as the solution of a set of linear integral equations.

The authors finally provide sample complexity guarantees for learning optimal policies under confounded contextual bandits and confounded RL setting, but with memoryless unmeasured confounding. The bounds are derived assuming bounded complexity for the class of $Q$-bridge functions and their projections.
strength_and_weaknesses: Strengths:
- I think the application of proximal causal learning in confounded RL setting is quite interesting. 
- As far as I can tell, the identification results look complete if one assumes the existence of proxy variables with desired properties.

Weaknesses:
- The paper is quite hard to follow at times. For example, the authors do not motivate the existence of proxy variables with desired properties. There is also no real-world example regarding the setting considered in section 4 (confounded RL with proxy variables for rewards and actions).
- Perhaps the major drawback of the paper is that the proposed method is not validated on any real-world dataset. So it is hard to tell where such methods can be applied.
- The final section on finite sample guarantees requires many assumptions. For example, it requires a bound on $p_\max$ which is completely a property of the behavioral policy. Second, for RL setting it requires memoryless confounding. I am not sure if confounding is memoryless one still requires such complicated method for estimation. Wouldn't it be possible to just perform posterior inference on hidden variables from the observation? 
- The authors also do not discuss how to solve the sequence of linear integral equation. This problem should be hard in general and it should have been discussed. Moreover, for finite sample guarantees, one needs to solve this set of equations from finite samples and needs to argue about estimation error. However, as far as I can tell, no such error bound (in terms of samples n) was provided.



Questions for the authors:
1. Why do you need two different types of proxy variables? Can you obtain similar results if there is a single proxy variable depending on the unobserved hidden state? 
2. I am not sure if the assumptions of multiple proxy variables and bridge functions with appropriate conditions are realistic. Can the authors provide some real-world examples where such multiple proxies are available and bridge functions satisfy the desired properties. I understand that it might not be feasible to verify some of the properties in practice, but does there exist any real-world datasets which suggest existence of such $Q$-bridge functions at every time-step?
3. Why does the finite sample guarantee depend on the constant $p_\max$? In general, one should expect a constant related to the overlap between the behavioral policy $\pi^b$ and optimal policy to show up in the bound but the constant $p_\max$ as well as $p^\omega_{t,\max}$ just depend on behavioral policy and can be arbitrarily large.
4. The role of the class $\mathcal{G}$ is not clear in theorem 5.1. Since this is just obtained from projecting the class of $q$-functions, why isn't bounding the complexity of bridge functions sufficient? 
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is hard to follow at times. Although the motivating example was interesting, many assumptions are introduced without proper context and should be motivated with examples.

Quality: The main drawback of the paper is that the methods are not applied on any real-world dataset. It is also not clear how much of the identification results depend on prior work on proximal causal learning, and how much of it is new. Finally, the last section on finite sample guarantees makes several strong assumptions in order to obtain meaningful bounds.
summary_of_the_review: I am leaning towards rejecting the paper mainly because the presentation of the paper is not clear, and the author makes many assumptions without giving proper example. The methods are also not validated on any real-world datasets, and not clear how they compare with prior methods.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers offline reinforcement learning in the presence of unobserved confounding. There are two observations. First, including the "expert recommended" action (from the behavioral policy) as part of the state is helpful, because this action was informed by the unobserved confounder. Second, if there are proxies for the unobserved confounder available, these can be leveraged to learn the optimal policy under the (better) expert-recommendations available setting. This later result is a direct application of recent proxy identification methods in causal inference.

------------

I read the rebuttal comments and the revised paper. I think the point about past/future state serving as proxies is a good one (though I haven't given much thought to its correctness). The authors have also added some experimental demonstration. Unfortunately, these are substantial revisions and I don't have the capacity to do more than a cursory evaluation of them.  

Overall, I remain weakly positive on this paper. It's not earth shattering, but it seems to be a solid execution of a reasonably interesting idea.  I've updated my score to an 8, but with the intention that this should be parsed literally ("I think it's a good paper worth accepting") and not as a very strong endorsement. 

(FWIW though, I find the brain stimulation example pretty unconvincing---if patient actions are available at runtime, then we'd presumably just ignore whatever the learned policy is and defer to the patient)

-----------
A second update:

Following the reviewer discussion, I'm downgrading my score to a weak reject. The crux of this is:
1. we all agree that proxy identification results from causal inference translate to results for policy learning for unobserved confounding. This paper seems to do the translation correctly, and I'm willing to accept that it's likely technically correct.*
2. however, a main claim of the paper is that expert recommended actions can serve / are especially good choices for the required proxies. None of us were able to articulate clearly how this part of the argument goes. (AFAICT, the real world healthcare examples don't use expert recommendations in this role)
3. thus, it seems that there's at least a clarity problem, and potentially a technical correctness problem. Based on the discussion, this is severe enough that the paper needs a major revision. If no reviewer is able to explain the gist of the main idea, then the paper is not communicating its development clearly enough.

*Although, here, I wonder if the presentation could be improved substantially simply by taking the building block as the identification result rather than the tool used for identification. That is, explain directly how to translate from such-and-such is causally identified, so the policy learning can now be achieved as such-and-such. The technical details of how proxy identification results work don't seem obviously different than how it works for identification, but they're involved enough to be distracting


strength_and_weaknesses: Strengths: this paper has a clear problem statement and exposition. I have no checked the proofs in detail, but they seem to be correct. There may be situations where the proposed scheme is useful. 

Weaknesses: the motivation is not totally clear to me. Particularly, the proxies for the unobserved confounder are introduced as an assumption to get around the non-identifiability of the target, expert-recommendation enhanced, policy. How realistic is this assumption? Are there particular motivating problems in mind? It would also be nice to have some experiments showing the scope of the benefits.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: This paper is clear and (apparently) correct. While I am not fully convinced of its importance, I think it's a nice demo of the use of proxy methods in reinforcement learning, and may have some real-world applicability.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes offline policy learning in a confounded setting for contextual bandits and sequential decision-making. The paper introduces the notion of super policy learning in the setup where expert-recommended actions are available at each step in addition to the past historical data. Under this setup, the author(s) propose two algorithms for the contextual bandit and sequential decision-making (RL) settings and derive corresponding regret bounds.

strength_and_weaknesses: Strength:

Important problem finding an optimal policy in confounded offline reinforcement learning.
Overall well-written, and the assumptions are stated in the context of the theorem.
The practical derivation of the proposed algorithms helps leverage them in a real-world task.

Weakness:

Real-world evaluation of the proposed practical algorithms is missing.
While assumptions are stated, it is unclear if all those are held in a real-world setup.
Missing empirical evaluation and discussion.


clarity,_quality,_novelty_and_reproducibility: Overall the paper is well-written and easy to follow. However, I would appreciate a proper paper ending with a conclusion or discussion section. In its current form, the paper seems to end abruptly on page 9.

The paper is well-motivated by real-world applications, and it proposed a relatively comprehensive theoretical analysis of the problem of finding an optimal policy in the presence of the confounder. The practical implementable algorithms are helpful in evaluating them in real-world environments (even in simulation). However, I am rather surprised that no such empirical evaluation is presented in the paper. The use of expert recommendations in decision-making is important and should improve existing policy learning. However, the practicality would depend on the availability of the expert recommendation.

My understanding is that the algorithm takes input the expert-recommended action at every timestep and then makes a decision based on both history and expert demonstration. However, I am a bit confused about the offline data generation process (history data). What are the assumptions on offline data generation? How is the offline data collected? Is it from a human agent or an optimally learned policy that uses confounded observation data? That raises the question of how much expert-recommended actions are needed to learn an optimal policy in the presence of a confounder. 

Moreover, having an expert action at each timestep might be infeasible in many RL tasks. For example, in learning robotic locomotion tasks (e.g., humanoid walk, quadruped run), access to expert recommendations would be infeasible. It is unclear how the proposed method can be leveraged in those settings. Are the confounded environments restricted where such demonstrations are feasible only? Clarifying these points in the abstract and intro would be helpful for the reader to understand. One suggestion would be to specify expert recommendations in the title. Alternatively, a justification would be helpful for how the proposed method is generalized for all the confounded environments, including environments where direct access to expert action at each timestep is not feasible (robotic locomotion).


summary_of_the_review: The paper tackles the important problem of finding an optimal policy in confounded offline reinforcement learning. The proposed practical algorithms have the potential to be implemented for real-world tasks. However, the justification of the assumptions on the real-world task is missing. Adding such empirical evaluation would eventually improve the justification of the paper's claims.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers a novel policy learning problem. One has access to past data generated by experts actions. The dataset logs covariates (or observed states), chosen actions, and rewards, but omits counfounders U which influence both potential outcomes and expert's actions. The goal is to learn a super policy which continues to observe some covariates and to observe expert recommendations, but can overrule the expert and pick any action it wishes. The question is scientifically interesting, though the immediate real world relevance is a bit unclear to me. (Usually, today, humans overrule algorithms. It is not clear also that humans would make recommendations the same way if they expected to be overruled).

*Strengths*:  The paper seems to propose a novel problem. They adapt the recent literature on proximal causal inference to formalize conditions under which their super RL agent succeeds, despite the presence of unobserved confounders. The paper initially contained no examples or empirical evaluation, but during the rebuttal period the authors ran some experiments (mostly on reusing past application of proximal causal inference) and placed those in the appendix. 

*Weaknesses*: This problem studied here is quite delicate. If unobserved confounders do not heavily influence expert actions, then there is little additional value in super RL, relative to standard off policy evaluation. If unobserved confounders heavily influence both expert actions and potential outcomes, unbiased evaluation of counterfacutal actions requires delicate assumptions. 

During the reviewer meeting, it became clear that *none of us* could articulate what problem features make this possible. The assumptions in the paper are quite abstract. I spent substantial time reading the proximal causal inference literature to try to understand them.  But I ultimately could not formulate a sensible sounding problem where the paper's various causal assumptions make sense AND observing expert recommendations in the future is quite valuable. The reviewers also carefully read the paper, looked at the updated examples in the appendix, and also cannot articulate the main idea.  It is hard to accept a paper when this is the outcome of peer review.

This may reflect a failing on the part of the review team. I'm sorry if that is the case. At the same time, the reviewers all work on related problems and read the paper with interest. Given this, it seems likely that the paper's lack of clarity will limit its impact and that it needs a substantial rewrite. 
justification_for_why_not_higher_score: The general concern is a lack of clarity and motivation for abstract assumptions. See above.
Decision: Reject