Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers Bayesian optimization over objective functions defined via a causal graph, where the nodes correspond to functions defining the relationships between nodes in the graph (including the action variables). Within this framework, a UCB-based acquisition function is proposed. This acquisition function is shown to enjoy sublinear regret. Moreover, it is shown to perform competitively with respect to several state-of-the-art methods across several test problems.
strength_and_weaknesses: Strengths:
1. This work considers a relevant problem setting and proposes a technically sound method with good empirical performance and theoretical guarantees. 
2. This paper is very well written overall.
3. An implementation of the code used in the empirical evaluation is publicly available.

Weaknesses:
1. Novelty: My main concern is related to this work's novelty claims. I believe the problem setting considered by this work is the same as that of Astudillo and Frazier (2021) in the sense that a function network could be written as a causal graph and vice versa. Both papers consider a directed acyclic graph where nodes in the graph take as input an action variable and the outputs of their parent nodes. I would like the authors to clarify what they mean in the following statement: "Moreover, function networks are not causal models and do not model actions as interventions." Is it only the use of the causal model formalism that makes these two problem settings different? Assuming this is the case, the primary source of the novelty of this work becomes the proposed acquisition function. I believe this is enough for this work to warrant publication, but the authors should clarify this. The authors should also explain that Kusakawa et al. (2021) also developed a UCB-based acquisition function that comes with similar sublinear regret guarantees in the cascade setting. How does the acquisition function proposed in this work compare with that proposed by Kusakawa et a. (2021) for causal graphs with a cascade structure?
2. Lack of realistic experiments: As discussed by the authors, the problem setting considered in this work arises in many critical real-world applications. However, they only considered one realistic test problem in their empirical evaluation. Moreover, this problem does not seem to be built upon a high-fidelity simulator or real-world data.

Other minor comments: 
1. The authors should discuss how $\eta$ is modeled in more detail. I wonder how the choice of the neural network architecture affects performance.
2. This work falls within an existing line of research on grey-box Bayesian optimization (see, e.g., Astudillo and Frazier 2022). I recommend the authors discuss this literature.

Astudillo, R., & Frazier, P. I. (2022). Thinking inside the box: A tutorial on grey-box Bayesian optimization.
clarity,_quality,_novelty_and_reproducibility: This paper is very well written, and the code used in the empirical evaluation is publicly available. Thus, this work performs well in terms of clarity and reproducibility. The proposed method is technically sound, thus making this paper perform well in terms of quality. As stated above, my main concern is that the discussion regarding novelty over prior work is misleading. It is also somewhat disappointing that realistic experiments were not considered in the empirical evaluation.
summary_of_the_review: This paper proposes a novel and technically sound acquisition function for causal BO. This acquisition function performs competitively in numerical experiments. My main concerns are that (1) this work's discussion regarding novelty over prior work is somewhat misleading, and (2) realistic experiments were not considered. Were these two aspects of the paper improved, I would support accepting this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a version of the CBO paper from 2020. They introduce some new minor ideas of how to fuse BO with causal inference ideas.
strength_and_weaknesses: This review proceeds section by section per your paper.

# Abstract

- It is confusing when you say "downstream" variable of interest. Certainly BO is a sequential method but you are targeting static causal diagrams wherein which "downstream" makes it sound as if you mean something in the future. Looking at your graphs you are targeting the root node of each CD (as per causal inference literature, not decision tree root-nodes who flip the terminology -- though I notice you are calling the outcome variable the 'sink').
- CBO, as introduced by Aglietti et al., target a full SCM not just a set of SEMs as you infer. Unless you are introducing a new method with the same name, then you are not discussing the same model framework.
- The original CBO paper does not assume noisless measurements. Where does it say that? The implementation of the model CBO (see here for the complete graph from the paper: https://github.com/VirgiAgl/CausalBayesianOptimization/blob/master/graphs/CompleteGraph.py) clearly has an epsilon parameter (the noise parameter for the observational samples). Your statement is either wrong or you are referring to something different.
- BO does not come with guarantees either so I am not sure how yours is a valid point? BO is not guaranteed to converge and so by extension neither is CBO.

# Introduction

- You should mention that this idea was first proposed by Aglietti et al. and yours is not a new idea w.r.t. the second paragraph. Similar ideas were introduced by the Lattimore brothers in their 2016 paper on causal bandits and Bareinboim in their 2015 before them. Optimal decision making + causal knowledge is not a new idea.
- I am curious as to why you depart from standard ideas of representing observationa and interventions in graphical form w.r.t. figure 1. It is standard to make interventional nodes V as upper-case labels and then repsent interventions as removing any incoming arcs to that node which is relabelled with small-caps label v, Hence, why have you departed from this formalism? What is the reasoning?

# BACKGROUND AND PROBLEM STATEMENT

- You are assuming causal sufficiency (no UCs) but the original CBO papers does not. Is that not a bit of a limiting assumption? It certainly limits the type of problems that you can deal with.
- It would be clearer if you used more set-notation e.g. to refer to all variables bar the outcome variables write $\mathbf{X} \setminus \{Y\}$. It is clearer.
- The same thing for graph terminology: use standard family relationships to refer to e.g. the parents of X_i i.e. pa(X_i) if you are not including also the argument and Pa(X_i) if you are. Please stick to this and don't introduce new notation.
- Can you please provide the pagenumber for "soft intervention with unknown effect" in Pearl's book, I cannot seem to find it.
- I do not follow why graph mutilation (i.e. hard interventions) is an issue w.r.t. your method? Please explain. I do not follow the logic or rather reasoning for going with soft or shift interventions.
- Not intervening with a hard-intervention would simply be to observe the system which results in no grap mutilation which means the DAG is not modified at all. This seems to be a odds with your previous point.
- When we cannot intervene on a node we usually call it a "non-manipulative variable" - this is well studied. 
- As to _what_ we should intervene on, the intervention set as it is called, is also very well studied. See Lee and Bareinboim's 2018 paper on 'where to intervene' - they prove specifically that when there are no UCs present (like you are assuming) it is always best to intervene on _all_ the parents of the target variable.
- You ought to reference the metric that you are using as it is not novel and has been used for decades to evaluate bandits.
- Now in equation 6 you are using hard interventions but which in your setting will not mutilate graph is what you are saying? 
- I find it strange that you have not discussed _what_ you plan to intervene on given that you'll be exploring the whole powerset space of interventions $\mathcal{P}(\mathbf{x} \setminus \{Y\})$ and this seems like a detail worth exploring, if not this will be terribly inefficient.

# ALGORITHM

- Where does equation 9 come from?
- Please explain the last sentence of this paragraph where you say: "In the following, we set βi,t = βt for all i such that the confidence bounds are still valid."
- After equation 10 then it sounds as if you are indeed exploring the whole powerset over intervention variables. That is a _very_ large set to explore particularly as you do not have UCs so the best intervention is always the parents of Y. 
- What is $\mathcal{I}$?
- In the hard intervention paragraph you are saying that use the MIS/POMIS ideas of Lee and Bareinboim but you're not really; you are using one of their definitions (number 1) to find the minimal intervention sets because, again, you do not have UCs, their work is barely applicable to your setting. Their ideas come into their own when there are UCs but otherwise, with chain-structured graphs you just remove intervention sets which topologically yield the same outcome expression.

# THEORETICAL ANALYSIS

- This is a very nice section, nicely done.
- It would have been nice to have a few examples of the bound in theorem 1 (give it an equation number so it can be referenced) applied to a few of the examples in figure 2.
- Why are the a's in the figures no longer bold as in figure 1(b)?
- I think you are taking too many liberties with how you are drawing your graphs. Typically, in causal inference, we reserve dashed edges for items that regard unobserved confounders (mind you, Aglietti et al. also takes liberty with their graph drawing so mine is a general comment). 
- You should look into dynamic treatment regimes; a lot of your examples graphs have the same topological structure (and ordering). Could be interesting to see if your ideas are applicable there.

# EXPERIMENTS

- It would seem odd for you to call the original paper 'EICBO' and not 'CBO' - call it the latter, as they were first to publish this idea.
- It is very strange that you do not deploy all methods on all test cases in figure 3. 
clarity,_quality,_novelty_and_reproducibility: # Clarity

The paper is well-written and mathematically rigorous.

# Quality

A high-quality paper.

# Novelty

The novelty is low. There are sections that, if expanded, will be far more impactful and interesting.

# Reproducibility

The authors have included the code. I have not gone through it.

summary_of_the_review: This is a twist on an earlier idea. I do not think there is enough novelty to warrant inclusion as it stands.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper is about Bayesian Optimization (BO), and, in particular, Causal Bayesian Optimization (CBO) based on a novel model-based approach. Authors propose to extend the existing CBO using a model-based approach, along with proofs for bounding the cumulative regret and more realistic assumptions compared to current CBO approaches. The reference model is the Structural Causal Model (SCM) with known casual graph but unknown functional relations, which enables the representation of both observational and interventional distributions. Authors explore both soft and hard interventions, with and without constraints over actions. The objective is to select a sequence of actions to maximize the expected reward by minimizing the cumulative regret function with a finite horizon. Since the acquisition function cannot be evaluated in closed form, authors apply the "kernel trick" and leverage gradient-based optimizers. Finally, both theoretical proofs and empirical evaluations are provided.
strength_and_weaknesses: Strength points:
    - Model-based approach over structural causal models that inherits the theoretical framework for causal inference.
    - Guarantees in terms of bounded cumulative regret.
    - Assumptions make sense, allowing to generalize to a wide range of settings.

Weak points:
    - No unobserved confounding could be a naive assumption. 
    - Knowing the true causal graph combined with causal sufficiency means to know the causal mechanism, even if the functions are unknown.
clarity,_quality,_novelty_and_reproducibility: The paper is clear both in terms of theoretical proofs and empirical guarantees. 
Furthermore, the notation is consistent while necessarily complex. 
The assumptions on which the contributions are based are listed and discussed in a clear manner.
The same applies to the discussions about limitations and possible solutions to overcome them. 
Formulas, pseudocodes and images are of good quality and help the interested reader to go through the work. 
I vote for the experimental layout (setting) is bith correct and reproducible, in the light that the author/s make available the software code. However, I had not time to check it myself.
The examples are meaningful, in adequate quantity and increasing complexity. 
Finally, I think that the degree of novelty of the paper is good and what developed and describe is relevant for the target field.
summary_of_the_review: The paper is well structure, organized and written and I enjoyed reading it.
Theoretical statements are presented and supported by the associated proofs .
Furthermore, empirical achievements are well supported a rich set of numerical experiments and obviously the associated results which are presented and discussed in an effective manner. 
The presented approach is novel for the considered research area and its results are promising also when taking into consideration the assumptions made by the author/s. 
It is also worth to mentin that the topic of the paper is relevant and discussed in details, thus representing an added value for the venue.
In conclusion I think the paper deserves consideration for being presented to a restigious venue as ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new algorithm for causal Bayesian optimization for both hard and soft interventions with theoretical guarantees. The authors use a reparameterization technique to optimize the acquisition function which is a maximum over functions. They empirically show that the algorithm mostly outperforms relevant baselines.
strength_and_weaknesses: Strengths:
1. The problem of causal BO is motivated well and explained clearly with diagrams.
2. The algorithm's performance is empirically well-supported.

Weaknesses:
1. One concern comes from Assumption 3 not providing an explicit formula for the sequence $\beta_t$. The authors claim that there is an "overall sublinear regret for MCBO". However, $\beta_T^N$ appears in the regret bound, where $N$ is the maximum distance from a root node to the reward node. Without an explicit formula for $\beta_t$ (as is usual in this line of work), there is no reason to believe that  $\beta_T^N < \mathcal O(\sqrt{T})$. Even if we assume that the usual formula for $\beta_t$ (Chowdhury and Gopalan, 2017) is applicable, we still have that $\beta_t = \mathcal O(\sqrt{\gamma_{t-1}})$. For the Matern kernel, $\gamma_T = \mathcal O(T^{d(d+1)/(2\nu + d(d+1))} \log T)$ (Srinivas et. al., 2010). Thus, if the Matern kernel was used, there would be some value of $N$ beyond which the regret bound is not sublinear. Assumption 3 needs to be even stronger than it currently is for the sublinearity claim to be true. Ideally, the authors could provide an explicit formula for $\beta_t$ instead of assuming such a sequence exists, or at least discuss why constructing such a sequence would be hard.

2. Each $f_i$ is a vector-valued function modelled by a vector-valued GP. However, the vector-valued GP as presented in this paper does not match with the usual understanding of a vector-valued GP. From Sec. 3 in Alvarez et. al. (2012) (https://arxiv.org/pdf/1106.6251.pdf), vector-valued GPs have matrix-valued kernel functions $\mathbf K: \mathcal S \times \mathcal S \rightarrow \mathbb R^{d \times d}$. However, this paper's regularity assumptions use a vector-valued kernel function $k_i: \mathcal S \times \mathcal S \rightarrow \mathbb R^{d_i}$. The definition of GP posterior mean and variance Equs. 7 and 8 are not well-defined as they rely on $\mathbf K_t$ where $(\mathbf K_t)_{ij} \coloneqq k_i(\mathbf z_i, \mathbf a_i; \mathbf z_j, \mathbf a_j)$, but $k_i$ is vector-valued according to their definition, so $\mathbf K_t$ is not a matrix. Can the authors clear up this confusion?

3. Why is UCB not tested on ToyGraph and PSAGraph?

4. Other issues:
- For the definitions of $\mathcal X$ and $\mathcal Z_i$, did you mean to use Cartesian product $\times$ instead of union $\cup$?
- The legends of the plots say MBCO instead of MCBO.
clarity,_quality,_novelty_and_reproducibility: Quality:
Other than the technical issues raised in the previous section, the paper is generally well-written. The experiments section is good.

Clarity:
The paper suffers from a lack of clarity in some technical parts as raised in point 2 of the previous section. 

Originality:
The paper is reasonably novel as it provides theoretical guarantees for a setting in which previous work did not provide such guarantees.

summary_of_the_review: I believe this paper has potential. Theoretical guarantees are important and the experiments show that their algorithm performs well in comparison to baselines. However, unless I am mistaken, the paper suffers from non-trivial technical issues. Assumption 3 as presented is already very strong but actually needs to be even stronger to properly achieve a sublinear regret bound. The definition of the GP posterior mean and variance is not well-defined and hinders understanding of their method. Until these issues are addressed, I am hesitant to recommend acceptance.

POST-REBUTTAL UPDATE: The authors have satisfactorily answered the main technical concerns and have improved the paper. I have improved my score to recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies Causal Bayesian optimization (CBO) in a model that combines models in two lines of literature, one from Aglietti et al. (2020b) on causal BayesOpt that considered "hard interventions", and the other from Astudillo & Frazier (2021b) on BO for function networks that considered "soft interventions".  It proposes a novel acquisition function and proves an associated regret guarantee.  This is the first algorithm for CBO with a cumulative regret guarantee. The algorithm provides better average reward (a surrogate for cumulative regret) than benchmarks in numerical experiments.

Strengths

- Model is general and fills a gap in the literature between past work on CBO and function networks
- Non-trivial acquisition function and cumulative regret guarantee
- Sufficient novelty relative to Aglietti et al., Astudillo & Frazier, and Kusakawa et al.
- Well-written

Weaknesses

- Paper assumes no unobserved confounding, which is unrealistic in many causal inference applications

- The paper's novelty comes from filling a gap between Aglietti et al. 2020, Frazier & Astudillo 2021, and Kasakawa et al. 2021. There is enough novelty for publication, but the level of novelty is smaller than in some other accepted ICLR papers.

- I was surprised that the proposed method did not outperform the benchmarks by more, given that EIFN is designed to minimize simple regret, and UCB is unable to take advantage of problem structure. This may suggest that there is room to further improve the acquisition function.

Originally there were concerns about missing discussion of related literature in the submitted the manuscript including how Kasakawa et al. 2021 was discussed. These were addressed during the review process through conversations with the authors, who proposed edits to the manuscript responding to these concerns.
justification_for_why_not_higher_score: - Experiments are not "exciting" enough for an oral, as articulated by the reviewers
- Level of novelty is not high enough for an oral
justification_for_why_not_lower_score: - Solid analysis and algorithm
- Good improvement over existing methods
Decision: Accept: notable-top-25%