Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers regret-minimizing algorithms for selecting an optimal treatment regime from a policy space characterized with mixed state-action scopes. 

More precisely, the authors focus on a structural causal model, motivated by medical applications, where they present two algorithms: a UCB-based one, which however requires intractable planning, and Thompson sampling relaxation, which appears to be “less intractable” (i.e., the computational issue is about finding a near-optimal policy on a fixed MDP).

They show improvements compared to a naive baseline as well as theoretically over naive UCB.
strengths_and_weaknesses: I am not an expert of the model assumed here, so I can only give high-level comments.
The authors use some standard technology (e.g., UCB, Thompson sampling) in the structural causal model. To be clear, both UCB and Thompson sampling need to be adapted, so there is some novelty in the work. They provide also a limited empirical evaluation. I suppose that the empirical evaluation needs to be restricted to small works since it might be too expensive to run these algorithm large scale.
Overall, the work is in an interesting positions, towards a possible application, all the while doing some theoretical contribution.
However, I am not familiar enough with the the causal literature to strongly support the work here.

The authors should discuss the computational consideration of finding a near-optimal policy on a fixed MDP in a better fashion. As I understand, even the Thomspon sampling-based algorithm could be intractable. This is of course strictly a limitation of planning and not of the proposal here, but it should nonetheless be discussed in more depth.
questions: None of relevance, other than the computational tractability discussed above
limitations: Discusses above
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper provides an algorithm for online learning in causal systems with mixed policy scopes that has sub-linear regret.  Results are presented on (small) simulated domains.
strengths_and_weaknesses: (+)	The paper is motivated by real-world practical problem in treatment optimization.

(+)	The paper is well grounded in terms of theoretical framework and includes results on sub-linear learning rate for the proposed setting.

(+)	The results of sub-linear regret appear sound theoretically and are confirmed by simulations.

(-)	I wonder how useful & interesting is the proposed setting, grounded in causal learning, compared to more standard MDP settings.  I’m less familiar with this literature (but very familiar with MDP/RL literature), and it made it quite challenging to follow precisely the setting, algorithm and theoretical analysis.

(-)	The proposed method appears limited to very small settings (few variables). I do not find the results very convincing given their simplicity.  See specific questions below about Experiment 2.

(-)	The empirical results show an improvement for the proposed c-ts* method, compared to Thompson sampling alternatives. However the gap seems relatively small and both approaches seem sub-linear.  How do you know if the improvement of c-ts* is significant?  Is the results really meaningful?  

questions: - Can you provide more detailed analysis of the space/time analysis for the algorithms CAUSAL-UCB* and the MINCOLLECTs?    The simulation results suggest this is limited to very small number of state/action variables; is this due to learning efficiency (in terms of #samples) or computational efficiency (in terms of memory/#operations)?  Could you include simulation results as a function of different factors (delta, #states/actions)?
- Considering Experiment 2, only blood pressure is considered as a variable (not even age, gender and other well-known contributing factors).  Is this because the method is limited to such small domain?  Or because it necessitates strong assumptions?   

limitations: - The paper does not really discuss technical limitations of the work, except where it serves to motivate the proposed approach.
- There could be a deeper discussion of the ethical aspects of this work, given the proposed application to selecting optimal treatment regime, and the example on patients with cardiovascular disease.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper considers the problem of effective *online* learning of optimal policies with mixed scopes (e.g. different types of treatments) in an arbitrary causal system.
The authors propose an algorithm (CAUSAL-UCB) to solve the problem and prove that it achieves sublinear regret.
Furthermore, to improve computational efficiency, the authors introduce a canonical representation for structural casual models and consider minimal reduction of c-components, then propose CAUSAL-TS as an easy-to-implement alternative.
The TS algorithm is evaluated on simple causal graphs to demonstrate its advantage over simple baselines.
strengths_and_weaknesses: **Strengths**
- Originality
    - This paper introduces the novel problem of effective *online* learning of optimal policies with mixed scopes in an arbitrary causal system. The authors provide a UCB-based algorithm that solves this problem, as well as new ideas to improve computational efficiency.
    - Relevant works are cited, and the authors make the contribution clear.
- Clarity
    - The paper is overall well-structured, and I can get the main points quickly.
    - The paper is heavy in notations, but the authors make use of simple examples to explain them.
- Significance
    - I think the problem considered in this paper is very important and relevant in practice, and the current paper can motivate more research in online learning and causal inference.

**Weaknesses**
- Quality
    - Certain claims (e.g. Theorem 3) do not seem well-supported. Seem Questions session for details.
    - The instances in the experiments are somewhat over-simple.
- Clarity
    - This paper is VERY heavy in notations, and I find it hard to memorize and digest some of these notations when I first read the paper. A summary table is highly recommended.
- Significance
    - The algorithm heavily depends on the assumption that endogenous variables have finite supports. But in many practical settings, many endogenous variables are not discreet (especially for contexts), so I'm a bit doubtful about the practicality of the proposed methods.

questions: - For Theorem 3, I don't think we can convert UCB regret to TS regret for arbitrary problem instances. Please include exact conditions in the Theorem statement and rigorous proof.

- Is it possible to extend the current algorithm to endogenous variables with possibly infinite support? What would be the difficult part? Or, if this is too much, it is good to at least provide experimental results where the endogenous variables take more than two values.

- Please include a summary of notations in the appendix. It would be great if more intuitive explanations in words can be provided (e.g. what do they mean / why do they matter).
limitations: All my concerns are listed in the previous sessions. Please respond to questions and suggestions raised in "Questions".
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers online policy learning with different policy scopes in the context of causal inference. The paper develops online learning algorithms with sub-linear regrets and presents simulation results of the theoretical algorithms. 
strengths_and_weaknesses: First of all, I have to be frank that I'm only knowledgeable of causal inference as a practical RL person. So most of my comments are from a rough understanding of this draft. 

I feel the paper is clearly written with rigorous (possibly textbook-style) formulations and derivations. The motivation is clear, and all the proofs and introduced algorithms look correct. The simulation results also look reasonable as a theoretical paper. Although I cannot confidently judge the novelty of this work, I would still rate it positive for this work.
questions: My only question is regarding the statement that most of the works are based on the offline learning setting. However, I do have an impression that there are many causal learning algorithms (even on the practical RL side) that are online. Is the major difference here primarily about the policy scopes?
limitations: NA
review_title: null

Meta-review: metareview: The reviewers appreciated the direction of the work, but they all had rather low confidence despite each one being a foremost expert in RL, which may indicate a potential mismatch of interest with NeurIPS as a venue. They also found the potential applications limited, especially given the great amount of formalization. Nonetheless, they judged the developments on this new problem insightful and having the potential to inspire further work in the area and also found no reason to reject. In view of this, I recommend acceptance as I think the potential benefits outweigh the concerns of fit or practical relevance, and given there are no concerns regarding validity.
confidence: Less certain
Decision: Accept