Causal Bandits with Unknown Graph Structure

1 Introduction . A multi-armed bandit ( MAB ) problem is one of the classic models of sequential decision making ( Auer et al. , 2002 ; Agrawal and Goyal , 2012 , 2013a ) . Statistical measures such as regret and sample complexity measure how fast learning algorithms achieve near optimal performance in bandit problems . However , both regret and sample complexity for MAB problems necessarily scale with the number of actions without further assumptions . To address problems with a large action set , researchers have studied various types of structured bandit problems where additional assumptions are made on the structure of the reward distributions of the various actions . Algorithms for structured bandit problems exploit the dependency among arms to reduce the regret or sample complexity . Examples of structured bandit problems include linear bandits ( Abbasi-Yadkori et al. , 2011 ; Agrawal and Goyal , 2013b ) , sparse linear bandits ( Abbasi-Yadkori et al. , 2012 ) , and combinatorial bandits ( Cesa-Bianchi and Lugosi , 2012 ; Combes et al. , 2015 ) . In this paper , we study a different kind of structured bandit problems : causal bandits . In this setting , actions are composed of interventions on variables of a causal graph . Many real world problems can be modeled via causal bandits . In healthcare applications , the physician adaptively adjusts the dosage of multiple drugs to achieve some desirable clinical outcome ( Liu et al. , 2020 ) . In email campaign problems , marketers adjust for features of commercial emails to attract more customers and convert them into loyal buyers ( Lu et al. , 2019 ; Nair et al. , 2021 ) . Genetic engineering also involves direct manipulation of one or more genes using biotechnology , such as changing the genetic makeup of cells to produce improved organisms ( Wikipedia contributors , 2021 ) . Recently there has been a flurry of works ( Lattimore et al. , 2016 ; Sen et al. , 2017 ; Lee and Bareinboim , 2018 ; Lu et al. , 2019 ; Lee and Bareinboim , 2019 ; Nair et al. , 2021 ) on causal bandits that show how to achieve simple regret or cumulative regret not scaling with the action set size . However , a major drawback of existing work is that they require significant prior knowledge . All existing works require that the underlying causal graph is given upfront . Some regret analysis works even assume knowing certain probabilities for the causal model . In practice , they are all strong assumptions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , our goal is to develop causal bandit algorithms that 1 ) do not require prior knowledge of the causal graph and 2 ) achieve stronger worst-case regret guarantees than non-causal algorithms such as Upper Confidence Bound ( UCB ) and Thompson Sampling ( TS ) whose regret often scales at least polynomially with the number of nodes n in the causal graph . Unfortunately , this goal can not be achieved for general causal graphs . Consider the causal graph consists of isolated variables and the reward directly depends on one of them . Then in the worst case there is no chance to do better than standard algorithms since no meaningful causal relations among variables can be exploited . In this paper , we study what classes of causal graphs on which we can achieve the goal . Our Contributions . We summarize our contributions below . 1 . We first study causal bandit problems where the unknown causal graph is a directed tree , or a causal forest . This setting has wide applications in biology and epidemiology ( Greenewald et al. , 2019 ; Burgos et al. , 2008 ; Kontou et al. , 2016 ; Pavlopoulos et al. , 2018 ) . We design a novel algorithm Central Node UCB ( CN-UCB ) that simultaneously exploits the reward signal and the tree structure to efficiently find the direct cause of the reward and then applies the UCB algorithm on a reduced intervention set corresponding to the direct cause . 2 . Theoretically , we show under certain identifiability assumptions , the regret of our algorithm only scales logarithmically with the number of nodes n in the causal graph . To our knowledge , this is the first regret guarantee for unknown causal graph that provably outperforms standard MAB algorithms . We complement our positive result with lower bounds showing the indentifiability assumptions are necessary . 3 . Furthermore , we generalize CN-UCB to a more general class of graphs that includes causal trees , causal forests , proper interval graphs , etc . Our algorithm first constructs undirected clique ( junction ) trees and again simultaneously exploits the reward signal and the junction-tree structure to efficiently find the direct cause of the reward . We also extend our regret guarantees to this class of graphs . In many scenarios , our algorithms do not recover the full underlying causal graph structure . Therefore , our results deliver the important conceptual message that exact causal graph recovery is not necessary in causal bandits since the main target is to maximize the reward . 2 Related work . The causal bandit framework was proposed by Lattimore et al . ( 2016 ) and has been studied in various settings since then . In the absence of confounders , Lattimore et al . ( 2016 ) and Sen et al . ( 2017 ) studied the best arm identification problem assuming that the exact causal graph and the way interventions influence the direct causes of the reward variable are given . Lu et al . ( 2019 ) and Nair et al . ( 2021 ) proposed efficient algorithms that minimize the cumulative regret under the same assumptions . When confounders exist , Lee and Bareinboim ( 2018 , 2019 ) developed effective ways to reduce the intervention set using the causal graph before applying any standard bandit algorithm , such as UCB . Even though the performance of above works improved upon that of standard bandit MAB algorithms , they all make the strong assumption of knowing the causal graph structure in advance . In our setting , the underlying causal graph is not known . Then a natural approach is to first learn the causal graph through interventions . There are many intervention design methods developed for causal graph learning under different assumptions ( He and Geng , 2008 ; Hyttinen et al. , 2013 ; Shanmugam et al. , 2015 ; Kocaoglu et al. , 2017 ; Lindgren et al. , 2018 ; Greenewald et al. , 2019 ; Squires et al. , 2020 ) . However , this approach is not sample efficient because it is not necessary to recover the full causal graph in order to maximize rewards . de Kroon et al . ( 2020 ) tackled this problem with unknown graph structure based on separating set ideas . However , this implicitly requires the existence of a set of non-intervenable variables that d-separate the interventions and the reward . Moreover , their regret bound does not improve upon non-causal algorithms such as UCB . In this paper , we take a different approach which uses the reward signal to efficiently learn the direct causes of the reward . Our approach is inspired by Greenewald et al . ( 2019 ) which proposed a set of central node algorithms that can recover the causal tree structure within O ( log n ) single-node interventions . Squires et al . ( 2020 ) also extended the central node idea to learn a general class of causal graphs that involves constructing junction trees and clique graphs . Following these works , our causal bandit algorithms also adaptively perform interventions on central nodes to learn the direct causes of the reward variable . However , our algorithms differ from theirs because we also take the reward signal into account . 3 Preliminaries . In this section , we follow the notation and terminology of Lattimore et al . ( 2016 ) ; Greenewald et al . ( 2019 ) for describing causal models and causal bandit problems . 3.1 Causal Models . A causal model consists of a directed acyclic graph ( DAG ) D over a set of random variables X = { X1 , . . . , Xn } and a joint distribution P that factorizes over D. The parents ( children ) of a variable Xi on graph D , denoted by PaD ( Xi ) ( or ChD ( Xi ) ) , are the set of variables Xj such that there is a directed edge from Xj to Xi ( or from Xi to Xj ) on graph D. The set of ancestors ( descendants ) of a variable Xi , denoted by AnD ( Xi ) ( or DeD ( Xi ) ) , are the set of variables Xj such that there is a path from Xj to Xi ( or from Xi to Xj ) on D. Without loss of generality , we assume the domain set for every Xi is Dom ( Xi ) = [ K ] : = { 1 , . . . , K } . For every Xi , we write the set of neighbors of Xi in graph D as ND ( Xi ) including variables Xj such that there is an edge between Xj and Xi regardless of the direction . The maximum degree of an undirected graph G is denoted by dmax ( G ) . Throughout , we denote the true causal graph by D , use V ( · ) as the set of vertices of a graph and define skeleton ( · ) as the undirected graph obtained by replacing the arrows in the directed graph with undirected edges . Definition 1 ( Directed Tree and Causal Tree ) . A directed tree is a DAG whose underlying undirected graph is a tree and all its edges point away from the root . A causal tree is a causal model whose underlying causal graph is a directed tree . For a node Xi on a directed or undirected tree D and its neighbor Y ∈ ND ( Xi ) , we write BXi : YD as the set of nodes that can be reached from Y through any path on the graph ( regardless of the directions of edges on the path ) , when the edge between Xi and Y is cut out from D. Note that the neighbor Y itself is always included in branch BXi : YD . Definition 2 ( Central Node ( Greenewald et al. , 2019 ) ) . A central node vc of an undirected tree T with respect to a distribution q over the nodes is one for which maxj∈NT ( vc ) q ( B vc : Xj T ) ≤ 1/2 . At least one such vc is guaranteed to exist for any distribution q ( Jordan , 1869 ; Greenewald et al. , 2019 ) . Informally , a central node vc guarantees that the weight of every branch around vc can not be larger than 1/2 according to the distribution q ( · ) . Definition 3 ( Essential Graph ) . The class of causal DAGs that encode the same set of conditional independences is called the Markov equivalence class . Denote the Markov equivalence class of a DAG D by [ D ] . The essential graph of D , denoted by E ( D ) , has the same skeleton as D , with directed edges Xi → Xj if such edge direction between Xi and Xj holds for all DAGs in [ D ] , and undirected edges otherwise . The chain components of E ( D ) , denoted by CC ( E ( D ) ) , are the connected components of E ( D ) after removing all directed edges . Every chain component of E ( D ) is a chordal graph ( Andersson et al. , 1997 ) . A DAG whose essential graph has a single chain component is called a moral DAG ( Greenewald et al. , 2019 ) . Definition 4 ( Causal Forest ( Greenewald et al. , 2019 ) ) . A causal graph is said to be a causal forest if each of the undirected components of the essential graph are trees . Many widely used causal DAGs including causal trees and bipartite causal graphs are examples of causal forest ( Greenewald et al. , 2019 ) . Bipartite graph applications can range from biological networks , biomedical networks , biomolecular networks to epidemiological networks ( Burgos et al. , 2008 ; Kontou et al. , 2016 ; Pavlopoulos et al. , 2018 ) .

Summary: Authors propose using the existing adaptive causal discovery algorithms to identify the reward variable in a causal graph efficiently and then run the existing bandit algorithms on this very small space of actions (interventions). They use the ideas from central node algorithm of Kristjan et al.: Identify a central node, either eliminate ~half the graph or find the subtree including the target node. They then have to check each child separately since either one of them can be the target node - which also serves as a method to check if central node itself is the target node. This takes ~dlogn interventions which can easily be converted to sample-complexity with the resorted assumptions.