Causal Contextual Bandits with Targeted Interventions

1 INTRODUCTION . Contextual bandits have been used as natural frameworks to model interactive decision making scenarios such as recommendation systems ( Liu et al. , 2018 ) , marketing campaign allocation ( Sawant et al. , 2018 ) and more ( Bouneffouf & Rish , 2019 ) . In this framework , the learning agent repeatedly interacts with an environment with the aim of learning a near-optimal decision policy that maps a context space to a set of actions ( also referred to as arms or interventions1 ) . In the standard stochastic variant , in each round of interaction , the agent observes a context from the environment and , once the agent chooses an action , the environment returns a sampled reward that is a function of the current context and chosen action ( Lattimore & Szepesvári , 2020 ) . The agent ’ s objective is to minimize some meaningful measure of closeness of the policy to optimality ( see Lattimore & Szepesvári ( 2020 ) for some standard definitions ) . One of the key issues in the wide application of contextual bandits ( and reinforcement learning , in general ( Dulac-Arnold et al. , 2021 ) ) is their need for a large number of samples that are costly to actualize in practice . For example , each arm might correspond to performing a product experiment on users or to conducting a specific medical intervention . However , we show that there are nuances in real-world situations which – though not fully captured by the standard formulations of contextual bandits – if modeled and leveraged , allow us to build methods that can improve the rate at which good policies can be identified . A motivating example Consider the example of a sales-assistance software agent that is learning to suggest a campaign that is optimal for a given sales lead ( defined by a set of features ) . With software products , there is often an opportunity to conduct experiments with variants of the software on defined subsets of the population of users ( each specified by a set of characteristics , or context variables ) to learn about resulting metrics ; see Google ( 2021 ) for an example . We can think of these experiments as constituting a training phase . More specifically , the agent , for example , can conduct a campaign ( which is the intervention ) on a specific subset of users ( for example , the context value os=iOS could define a subset ) and observe the outcome . Thus , instead of necessarily performing an intervention on a randomly sampled user coming from the natural population distribution , the agent 1We use the term intervention here because actions or arms in a ( contextual ) bandit setting can be interpreted as Pearl do ( ) interventions on a causal model ( Zhang & Bareinboim , 2017 ; Lattimore et al. , 2016 ) . See Pearl ( 2009 ; 2019 ) for more discussion on the do ( ) operation . can instead choose to target the intervention on a randomly selected user with specific characteristics ( as defined by an assignment of context variable values ) . We call this special type of interventions targeted interventions , and they fundamentally change the set of options that the agent has in every training round . In addition , the training phase often happens in a more lenient environment where the agent might have access to auxiliary context variables ( such as IT-spend ) that are unavailable in the evaluation phase . Further , there is also causal side-information sometimes available . For instance , we might know that emailsubject causes openemail , and not the other way around . Encoding this qualitative side-information as a causal graph can help the agent make use of this structure . Lattimore et al . ( 2016 ) and Yabe et al . ( 2018 ) demonstrate this in the best-arm identification case . After training , there is an evaluation phase ( e.g. , when the agent is deployed ) , where the agent observes the main context variables and decides an action ; its regret performance is measured at this point . Our framework Our framework captures the above intricacies ( which are not restricted to software product experimentation ) . The formal treatment is in Section 2 ; we present an overview here . The agent has T rounds of interaction that act as the training phase , followed by a ( T + 1 ) ’ th round on which it is evaluated . In each of the T training rounds , it has the choice to perform either a targeted intervention or a standard interaction . Further , in addition to the main context variables , the agent can observe a ( possibly empty ) set of additional auxiliary context variables during training ; these auxiliary context variables are not observable during evaluation . All variables are made available to the agent at the end of every training round ( this is similar to Lattimore et al . ( 2016 ) ) . The agent also has access to a causal graph that models the qualitative causal relationships between the variables ; there are very few assumptions made on this graph ( discussed in Section 2.1 ) . The causal graph allows a factorized representation of the joint distribution of the variables , and as a result , enables information leakage – i.e. , updating beliefs about several interventions after every single intervention . Also , importantly , we allow context variables to be categorical , and therefore the usual assumptions that enable generalization of the learned policy across contexts , such as linearity ( e.g. , in Dimakopoulou et al . ( 2019 ) ) , become invalid . The agent ’ s problem is one of sample allocation – how to allocate samples across the T training rounds so as to learn a policy that minimizes regret in the ( T + 1 ) ’ th round that represents the evaluation phase . In the evaluation phase , the agent observes the main context variables from the environment , against which it chooses an action and receives a reward – much like in a standard contextual bandit setting . Since targeted interventions are restricted to the training phase , it introduces a difference between the training and evaluation phases . The agent ’ s challenge , therefore , is to learn policies that minimize regret in the evaluation phase using samples it collects in the training phase ( from a combination of standard interactions and targeted interventions ) . The algorithm we propose utilizes a novel entropy-like measure called Unc that guides this sample allocation in a way that also exploits the information leakage . 1.1 CONTRIBUTIONS . In this paper , we formalize this modified setting , which we call “ causal contextual bandits with targeted interventions ” , provide a novel algorithm and show both theoretical and experimental results . Specifically , our contributions are : • We formalize the more nuanced contextual bandit setting described above ( Sec . 2 ) . This is the first work that we know of that formulates a contextual bandit setting with causal side-information . This is also the first paper we are aware of that introduces targeted interventions in a contextual bandit setting . • We propose a new algorithm based on minimizing a novel entropy-like measure ( Sec . 3.1 ) • We prove a bound on its regret , providing a theoretical guard on its performance ( Sec . 3.2 ) • We show results of experiments that use purely synthetic data ( Sec . 4.1 ) . The results demonstrate that our algorithm performs better than baselines . • We also run experiments that are inspired by proprietary data from a CRM software com- pany ( Sec . 4.2 ) .2 This provides evidence of our algorithm ’ s performance in more realistic scenarios as well . 2Identity of the company withheld here to ensure anonymity of submission . Our motivation comes from real world settings where experiments are costly ; therefore , we are interested in empirical behavior when the training budget T is relatively small . 1.2 RELATED WORK . Causal bandits have been studied in literature recently ( Lattimore et al. , 2016 ; Sen et al. , 2017 ; Yabe et al. , 2018 ; Lu et al. , 2020 ) and they leverage causal side-information to transfer knowledge across interventions . They , however , have been studied only in a best arm identification setting , with each arm modeled as an intervention on a set of variables ; the objective is to learn a single best arm . Our setting differs significantly since the agent attempts to learn a policy ( by which we mean a mapping from contexts to arms ) instead of a single best arm . Therefore , while the agent can perform targeted interventions that specify both the context and action , it is still attempting to learn an optimal context-action mapping . The contextual bandit literature is well-studied ( see Zhou ( 2016 ) and Lattimore & Szepesvári ( 2020 ) for surveys ) , and has taken various approaches to enable knowledge transfer across contexts . For example , Dimakopoulou et al . ( 2019 ) assume expected reward is a linear function of context , while Agarwal et al . ( 2014 ) make assumptions about the existence of a certain oracle . However , there has not been work that has looked at contextual bandits which utilize causal side-information as a way to transfer knowledge across contexts , or considered targeted interventions in the space of options for the agent . Our method utilizes the information leakage that the causal graph provides to not just learn about other interventions after every intervention , but also to be smarter about the choice of targeted interventions to conduct . Another related area is active learning ( Settles , 2009 ) . In active learning , a supervised learning agent gets the option to query an oracle actively to obtain the true labels for certain data points . However , it is in the supervised learning setting where the agent receives true label feedback , whereas in our setting the agent only receives bandit feedback ( that is , only for the action that was taken ) . Nevertheless , our work can be thought of as infusing some elements of the active learning problem into the contextual bandits setting by providing the agent the ability to perform targeted interventions . There has been some work investigating the intersection of causality and bandits with the aim of transfer learning . Zhang & Bareinboim ( 2017 ) study transfer of knowledge from offline data in the presence of unobserved confounders , in a non-contextual setting . As can be seen , this differs significantly from our setting . 2 FORMALISM . We assume that the underlying environment is modeled as a causal model M , which is defined by a directed acyclic graph G over all variables – i.e. , the variable to be intervened ( X ) , the reward variable ( Y ) , and the set of context variables ( C ) – and a joint probability distribution P that factorizes over G ( Pearl , 2009 ; Koller & Friedman , 2009 ) . C is partitioned into the set of main context variables ( Ctar ) and the set of auxiliary context variables ( Cother ) . G is sometimes called the causal graph or causal diagram ofM . Each variable takes on a finite , known set of values ; note that this is quite generic , and allows for categorical variables . An intervention do ( X = x ) onM involves removing all arrows from the parents ofX intoX , and settingX = x ( Pearl , 2009 ) . The agent has access only to G and not toM ; therefore , the agent is not given any knowledge a priori about the underlying conditional probability distributions ( CPDs ) of the variables . We specify a targeted intervention X = x conditioned on context Ctar = ctar succinctly by ( x , ctar ) . While standard interventions ( Pearl , 2009 ) result in a distribution of the form P ( . | do ( x ) ) , targeted interventions result in P ( . | do ( x ) , Ctar = ctar ) . Table 1 summarizes the key notation used in this paper . The agent is allowed T training rounds of interaction with the environment , at the end of which it aims to have learned a policy ϕ̂ : val ( Ctar ) → val ( X ) . Specifically , in each of the T training rounds , the agent can choose to either • ( Standard interaction ) Observe context ctar ∼ P ( Ctar ) , choose intervention x , and observe ( y , cother ) ∼ P ( Y , Cother | do ( x ) , ctar ) , ( or ) where , when there is no ambiguity , we use P interchangeably to mean either the joint distribution or a marginal distribution . Note that in both modes of interaction above , the intervention is only on X ; they differ , however , in whether the agent observes the context values from the environment or whether it chooses the context values on which to condition the intervention . After training , the agent is evaluated in the ( T + 1 ) ’ th round . Here , the agent is presented a query context ctar ∼ P ( Ctar ) to which it responds with an action x = ϕ̂ ( ctar ) using the learned policy , and receives a reward y ∼ P ( Y | do ( x ) , ctar ) from the environment . The objective of the agent is to learn a policy ϕ̂ that minimizes simple-regret , which is defined as : Regret ≜ ∑ ctar [ µ∗ctar − µ̂ctar ] · P ( ctar ) = ∑ ctar Regret ( ctar ) · P ( ctar ) where ϕ∗ is an optimal policy , µ∗ctar ≜ E [ Y |do ( ϕ∗ ( ctar ) ) , ctar ] and µ̂ctar ≜ E [ Y |do ( ϕ̂ ( ctar ) ) , ctar ] .

Summary: The paper studies a contextual bandit setting with two unique features: (1) the learning agent has ability to perform targeted interventions during the learning phase (ability to select target sub-populations or context) and (2) it also has access to and integrates casual information in the setting. The key motivation is that this setting captures real-world scenarios such as software product experimentation. One key contribution of the work is a new algorithm that achieves better sample efficiency. The algorithm is evaluated using synthetic experiments and real-world-inspired synthetic experiments.