Causal Bandits with Unknown Graph Structure

1 Introduction . A multi-armed bandit ( MAB ) problem is one of the classic models of sequential decision making ( Auer et al. , 2002 ; Agrawal and Goyal , 2012 , 2013a ) . Statistical measures such as regret and sample complexity measure how fast learning algorithms achieve near optimal performance in bandit problems . However , both regret and sample complexity for MAB problems necessarily scale with the number of actions without further assumptions . To address problems with a large action set , researchers have studied various types of structured bandit problems where additional assumptions are made on the structure of the reward distributions of the various actions . Algorithms for structured bandit problems exploit the dependency among arms to reduce the regret or sample complexity . Examples of structured bandit problems include linear bandits ( Abbasi-Yadkori et al. , 2011 ; Agrawal and Goyal , 2013b ) , sparse linear bandits ( Abbasi-Yadkori et al. , 2012 ) , and combinatorial bandits ( Cesa-Bianchi and Lugosi , 2012 ; Combes et al. , 2015 ) . In this paper , we study a different kind of structured bandit problems : causal bandits . In this setting , actions are composed of interventions on variables of a causal graph . Many real world problems can be modeled via causal bandits . In healthcare applications , the physician adaptively adjusts the dosage of multiple drugs to achieve some desirable clinical outcome ( Liu et al. , 2020 ) . In email campaign problems , marketers adjust for features of commercial emails to attract more customers and convert them into loyal buyers ( Lu et al. , 2019 ; Nair et al. , 2021 ) . Genetic engineering also involves direct manipulation of one or more genes using biotechnology , such as changing the genetic makeup of cells to produce improved organisms ( Wikipedia contributors , 2021 ) . Recently there has been a flurry of works ( Lattimore et al. , 2016 ; Sen et al. , 2017 ; Lee and Bareinboim , 2018 ; Lu et al. , 2019 ; Lee and Bareinboim , 2019 ; Nair et al. , 2021 ) on causal bandits that show how to achieve simple regret or cumulative regret not scaling with the action set size . However , a major drawback of existing work is that they require significant prior knowledge . All existing works require that the underlying causal graph is given upfront . Some regret analysis works even assume knowing certain probabilities for the causal model . In practice , they are all strong assumptions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , our goal is to develop causal bandit algorithms that 1 ) do not require prior knowledge of the causal graph and 2 ) achieve stronger worst-case regret guarantees than non-causal algorithms such as Upper Confidence Bound ( UCB ) and Thompson Sampling ( TS ) whose regret often scales at least polynomially with the number of nodes n in the causal graph . Unfortunately , this goal can not be achieved for general causal graphs . Consider the causal graph consists of isolated variables and the reward directly depends on one of them . Then in the worst case there is no chance to do better than standard algorithms since no meaningful causal relations among variables can be exploited . In this paper , we study what classes of causal graphs on which we can achieve the goal . Our Contributions . We summarize our contributions below . 1 . We first study causal bandit problems where the unknown causal graph is a directed tree , or a causal forest . This setting has wide applications in biology and epidemiology ( Greenewald et al. , 2019 ; Burgos et al. , 2008 ; Kontou et al. , 2016 ; Pavlopoulos et al. , 2018 ) . We design a novel algorithm Central Node UCB ( CN-UCB ) that simultaneously exploits the reward signal and the tree structure to efficiently find the direct cause of the reward and then applies the UCB algorithm on a reduced intervention set corresponding to the direct cause . 2 . Theoretically , we show under certain identifiability assumptions , the regret of our algorithm only scales logarithmically with the number of nodes n in the causal graph . To our knowledge , this is the first regret guarantee for unknown causal graph that provably outperforms standard MAB algorithms . We complement our positive result with lower bounds showing the indentifiability assumptions are necessary . 3 . Furthermore , we generalize CN-UCB to a more general class of graphs that includes causal trees , causal forests , proper interval graphs , etc . Our algorithm first constructs undirected clique ( junction ) trees and again simultaneously exploits the reward signal and the junction-tree structure to efficiently find the direct cause of the reward . We also extend our regret guarantees to this class of graphs . In many scenarios , our algorithms do not recover the full underlying causal graph structure . Therefore , our results deliver the important conceptual message that exact causal graph recovery is not necessary in causal bandits since the main target is to maximize the reward . 2 Related work . The causal bandit framework was proposed by Lattimore et al . ( 2016 ) and has been studied in various settings since then . In the absence of confounders , Lattimore et al . ( 2016 ) and Sen et al . ( 2017 ) studied the best arm identification problem assuming that the exact causal graph and the way interventions influence the direct causes of the reward variable are given . Lu et al . ( 2019 ) and Nair et al . ( 2021 ) proposed efficient algorithms that minimize the cumulative regret under the same assumptions . When confounders exist , Lee and Bareinboim ( 2018 , 2019 ) developed effective ways to reduce the intervention set using the causal graph before applying any standard bandit algorithm , such as UCB . Even though the performance of above works improved upon that of standard bandit MAB algorithms , they all make the strong assumption of knowing the causal graph structure in advance . In our setting , the underlying causal graph is not known . Then a natural approach is to first learn the causal graph through interventions . There are many intervention design methods developed for causal graph learning under different assumptions ( He and Geng , 2008 ; Hyttinen et al. , 2013 ; Shanmugam et al. , 2015 ; Kocaoglu et al. , 2017 ; Lindgren et al. , 2018 ; Greenewald et al. , 2019 ; Squires et al. , 2020 ) . However , this approach is not sample efficient because it is not necessary to recover the full causal graph in order to maximize rewards . de Kroon et al . ( 2020 ) tackled this problem with unknown graph structure based on separating set ideas . However , this implicitly requires the existence of a set of non-intervenable variables that d-separate the interventions and the reward . Moreover , their regret bound does not improve upon non-causal algorithms such as UCB . In this paper , we take a different approach which uses the reward signal to efficiently learn the direct causes of the reward . Our approach is inspired by Greenewald et al . ( 2019 ) which proposed a set of central node algorithms that can recover the causal tree structure within O ( log n ) single-node interventions . Squires et al . ( 2020 ) also extended the central node idea to learn a general class of causal graphs that involves constructing junction trees and clique graphs . Following these works , our causal bandit algorithms also adaptively perform interventions on central nodes to learn the direct causes of the reward variable . However , our algorithms differ from theirs because we also take the reward signal into account . 3 Preliminaries . In this section , we follow the notation and terminology of Lattimore et al . ( 2016 ) ; Greenewald et al . ( 2019 ) for describing causal models and causal bandit problems . 3.1 Causal Models . A causal model consists of a directed acyclic graph ( DAG ) D over a set of random variables X = { X1 , . . . , Xn } and a joint distribution P that factorizes over D. The parents ( children ) of a variable Xi on graph D , denoted by PaD ( Xi ) ( or ChD ( Xi ) ) , are the set of variables Xj such that there is a directed edge from Xj to Xi ( or from Xi to Xj ) on graph D. The set of ancestors ( descendants ) of a variable Xi , denoted by AnD ( Xi ) ( or DeD ( Xi ) ) , are the set of variables Xj such that there is a path from Xj to Xi ( or from Xi to Xj ) on D. Without loss of generality , we assume the domain set for every Xi is Dom ( Xi ) = [ K ] : = { 1 , . . . , K } . For every Xi , we write the set of neighbors of Xi in graph D as ND ( Xi ) including variables Xj such that there is an edge between Xj and Xi regardless of the direction . The maximum degree of an undirected graph G is denoted by dmax ( G ) . Throughout , we denote the true causal graph by D , use V ( · ) as the set of vertices of a graph and define skeleton ( · ) as the undirected graph obtained by replacing the arrows in the directed graph with undirected edges . Definition 1 ( Directed Tree and Causal Tree ) . A directed tree is a DAG whose underlying undirected graph is a tree and all its edges point away from the root . A causal tree is a causal model whose underlying causal graph is a directed tree . For a node Xi on a directed or undirected tree D and its neighbor Y ∈ ND ( Xi ) , we write BXi : YD as the set of nodes that can be reached from Y through any path on the graph ( regardless of the directions of edges on the path ) , when the edge between Xi and Y is cut out from D. Note that the neighbor Y itself is always included in branch BXi : YD . Definition 2 ( Central Node ( Greenewald et al. , 2019 ) ) . A central node vc of an undirected tree T with respect to a distribution q over the nodes is one for which maxj∈NT ( vc ) q ( B vc : Xj T ) ≤ 1/2 . At least one such vc is guaranteed to exist for any distribution q ( Jordan , 1869 ; Greenewald et al. , 2019 ) . Informally , a central node vc guarantees that the weight of every branch around vc can not be larger than 1/2 according to the distribution q ( · ) . Definition 3 ( Essential Graph ) . The class of causal DAGs that encode the same set of conditional independences is called the Markov equivalence class . Denote the Markov equivalence class of a DAG D by [ D ] . The essential graph of D , denoted by E ( D ) , has the same skeleton as D , with directed edges Xi → Xj if such edge direction between Xi and Xj holds for all DAGs in [ D ] , and undirected edges otherwise . The chain components of E ( D ) , denoted by CC ( E ( D ) ) , are the connected components of E ( D ) after removing all directed edges . Every chain component of E ( D ) is a chordal graph ( Andersson et al. , 1997 ) . A DAG whose essential graph has a single chain component is called a moral DAG ( Greenewald et al. , 2019 ) . Definition 4 ( Causal Forest ( Greenewald et al. , 2019 ) ) . A causal graph is said to be a causal forest if each of the undirected components of the essential graph are trees . Many widely used causal DAGs including causal trees and bipartite causal graphs are examples of causal forest ( Greenewald et al. , 2019 ) . Bipartite graph applications can range from biological networks , biomedical networks , biomolecular networks to epidemiological networks ( Burgos et al. , 2008 ; Kontou et al. , 2016 ; Pavlopoulos et al. , 2018 ) .

Summary: This paper gives low-regret algorithms for the causal bandits problem when the graph structure is unknown. The main contribution is in combining use of some recent results on efficient causal structure discovery with traditional UCB approaches. Some minimax results are given to demonstrate the necessity of the assumptions made.