SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alexis Bellot, Alan Malek, Silvia Chiappa
abellot@google.com
A unifying theme in the design of intelligent agents is to efficiently optimize a policy based on what prior knowledge of the problem is available and what actions can be taken to learn more about it. Bandits are a canonical instance of this task that has been intensely studied in the literature. Most methods, however, typically rely solely on an agent's experimentation in a single environment (or multiple closely related environments). In this paper, we relax this assumption and consider the design of bandit algorithms from a combination of batch data and qualitative assumptions about the relatedness across different environments, represented in the form of causal models. In particular, we show that it is possible to exploit invariances across environments, wherever they may occur in the underlying causal model, to consistently improve learning. The resulting bandit algorithm has a sub-linear regret bound with an explicit dependency on a term that captures how informative related environments are for the task at hand; and may have substantially lower regret than experimentation-only bandit instances.
Multi-armed bandits (MABs) constitute one of the most widely used frameworks for modeling decision-making under uncertainty. In this framework, an agent repeatedly takes actions in an environment with the goal of optimizing a desired objective, such as efficiently inferring the action with highest reward or maximizing cumulative rewards in the long run [34]. As in most reinforcement learning problems, there is a substantial amount of exploration involved while the agent learns about reward distributions under different available actions. This process can be costly in many applications; from an ethical perspective, for example, physicians may not risk compromising their patient's health with unknown treatments. It is therefore important to be efficient with experimentation while learning an optimal policy. In the literature, structured bandit instances can help navigate the exploration-exploitation trade-off effectively, for example with assumptions on the functional association between action and reward that facilitate estimation such as linear bandits [1, 15, 19] and causal bandits [23, 36, 28, 27, 24, 26, 12, 29, 7].
An alternative approach to alleviate the cost of active experimentation is to consider leveraging prior data or prior experimentation in related environments to inform an agent's decision-making, which leads to the hybrid learning paradigm. The expectation (or rather hope) is that informative prior data or prior experimentation can serve to narrow down reward distributions and warm start the MAB so as to converge to optimal actions faster and ultimately achieve higher cumulative reward. Current methods can be categorized into multi-task learning [41, 40, 13] and meta-learning [10, 42, 4, 20, 33, 30]. The former aims to solve a prescribed set of related bandit tasks with shared structure, e.g. multiple player scenarios with similar reward distributions. The latter considers an arbitrary number of bandit problems whose parameters are sampled i.i.d. from a meta-prior that
can be inferred as the agent experiments across the different related tasks<sup>1</sup>. However, both families of methods assume a relatively restricted class of potential changes across environments and rely explicitly on agent experimentation across all environments for learning. If discrepancies across environments are more general, naively leveraging prior data does not necessarily lead to more informative reward distributions or efficiency improvements in a new environment.
As a concrete example, consider a learning scenario in which historical data is available for the design of a clinical trial<sup>2</sup> aiming to determine the optimal level of a hypertension treatment  $X$  for Alzheimer's disease  $Y^3$ . Alzheimer's aetiology is complex but it is well established that a patient's age  $Z$  and blood pressure  $W$  contribute to the development of the disease, and so do a number of (typically) unobserved factors, e.g., physical activity levels, socio-economic status, diet patterns, etc. [37] (encoded with a bi-directed dashed edge). Such data can be useful but has to be handled with care, especially if we suspect the clinical trial popula
Figure 1: Diagram encoding causal structure and differences across environments.
tion to differ in several aspects from that recorded in historical data. We would expect, for example, age distributions  $P(z)$  to differ. Fig. 1 graphically describes this scenario. A naive approach, ignoring the differences across populations, can be sub-optimal. Take an instance where variables  $X, Z, W, Y, U \in \{0,1\}$ ; their values are decided by functions:  $X \gets U$ ,  $W \gets X$ ,  $Y \gets Z \oplus (W \cdot U)$ ;  $\oplus$  represents the exclusive-or operator;  $U$  is independently distributed in  $\{0,1\}$  but older individuals are historically over-represented  $P(Z = 1) = 0.6$  in comparison to the clinical trial population  $P(Z = 1) = 0.4$ . Historical data suggests that the better policy involves a lower dosage  $X = 0$ , as  $\mathbb{E}[Y \mid do(X = 0)] = 0.6 > 0.5 = \mathbb{E}[Y \mid do(X = 1)]$ , which is the opposite of what is optimal for the clinical trial population.
This example shows that differences across environments may be complex, subtle, and non-trivially influence optimal decision-making. Even when one is able to perfectly estimate reward distributions from historical data, the induced policy can still be sub-optimal depending on the location and magnitude of the changes expected across environments. In general, reward distributions will not straightforwardly extrapolate across different environments. In this paper, we attempt to capture this (structural) uncertainty through a causal lens. In the causality literature, this problem appears under the rubric of transportability theory [32, 3, 11]; several criteria, algorithms, and estimation methods have been developed for identifying when and how a causal effect can be computed across environments. Our task, in the bandit setting, is to define a learning agent that optimally exploits prior data with knowledge of the potential discrepancies across domains, for any given graph. Prior work [43] has considered specific instances, i.e. environments defined by the Bow and Instrumental Variables graphs, but a general approach applicable to arbitrary graphs and arbitrary differences across environments is still missing. Our approach is Bayesian, and involves posterior sampling of reward distributions defined by a parameterization informed by the underlying causal structure. Our contribution is to develop a novel bandit algorithm that achieves sub-linear cumulative regret with an explicit dependency on the entropy of an inferred prior, a quantity that implicitly captures the relatedness between environments. The significance of this result is that it guarantees consistent improvements on performance over methods not leveraging prior data. To the best of our knowledge, this is one of the first general attempts to consistently use prior data from related environments in general decision-making scenarios in which causal dependencies can be established.
We use capital and small letters to denote random variables and their values respectively, e.g.  $X$  and  $x$ , and bold capital and small letters to denote sets of variables and their values, e.g.  $X$  and  $x$ . The domain of variable  $X$  is indicated with  $\Omega_{X}$ .
A environment's data generating mechanism is described by a structural causal model (SCM) [31, Definition 7.1.1]. A SCM  $M$  is a tuple  $\langle \mathbf{V}, \mathbf{U}, \mathcal{F}, P(\mathbf{U}) \rangle$ , where  $\mathbf{V}$  is a set of endogenous (observed) variables,  $\mathbf{U}$  is a set of exogenous latent variables, and  $\mathcal{F} = \{f_V\}_{V \in \mathbf{V}}$  is a set of functions
<sup>1</sup>A more extensive review of related work is given in Appendix A.
[17] refer to adaptive clinical trials as the "chief practical motivation [for the design of bandit algorithms]".
3For this example,  $Y$  a measured biomarker of Alzheimer's disease that acts as a designated reward variable to be maximized.
such that  $f_V$  determines values of  $V$  taking as argument variables  $P a_V \subseteq V$  and  $U_V \subseteq U$ , i.e.  $V \gets f_V(P a_V, U_V)$ . Values of  $U$  are drawn from an exogenous distribution  $P(\boldsymbol{u})$ . We assume the model to be recursive, i.e. that there are no cyclic dependencies among the variables, such as to define a distribution  $P(V)$  over endogenous variables  $V$ . An intervention or action by an agent on a subset  $X \subset V$ , denoted by  $do(x)$ , is an operation that fixed values of  $X$  to constants  $x$ , replacing the functions  $\{f_X : X \in X\}$  that would normally determine their values. Let  $M_x$  denote the model induced by action  $do(x)$ . Accordingly,  $M_x$  induces a corresponding interventional distribution over  $V$ , denoted  $P(V_x) \coloneqq P(V \mid do(x))$ . We will consistently use  $X, Y \in V$  as designated action and reward variables, respectively.
Causal graphs  $\mathcal{G} = (\mathbf{V},\mathcal{E})$  describe the functional associations in an underlying SCM  $M$ . In particular, we draw a directed edge between two variables  $V\rightarrow W\in \mathcal{E}$  if  $V$  appears as an argument of  $f_{W}$  in  $M$ , and a bi-directed dashed edge between two variables  $V\leftrightarrow \dots \rightarrow W\in \mathcal{E}$  if  $U_{V}\cap U_{W}\neq \emptyset$ , i.e.  $V$  and  $W$  share an unobserved confounder. We will use standard family conventions for graphical relationships, e.g. parents  $pa(\pmb {X})_{\mathcal{G}}\coloneqq \cup_{X\in \pmb{X}}pa(X)_{\mathcal{G}}$  of a set of nodes  $\pmb {X}\subseteq \pmb{V}$  are all nodes in  $\mathcal{G}$  with directed edges into elements of  $\pmb{X}$ . Its capitalized version  $Pa$  includes the argument as well, e.g.  $Pa(\pmb {X})_{\mathcal{G}}\coloneqq pa(\pmb {X})_{\mathcal{G}}\bigcup \pmb {X}$ . We will make use a special clustering of the nodes in  $\pmb{V}$  called  $c$ -components [39]: two nodes are in the same  $c$ -component  $\pmb {C}\subseteq \pmb{V}$  if and only if they are connected by a bi-directed path.  $c$ -components form a partition over exogenous variables: a  $c$ -component  $\pmb {C}\subseteq \pmb{V}$  is said to cover an exogenous variable  $U$  if  $U\in \bigcup_{V\in C}\pmb{U}_{V}$ . We denote with  $C_U$  the  $c$ -component covering  $U$ . As an example, the diagram in Fig. 1 has  $c$ -components  $\{X,Y\}$ ,  $\{W\}$ , and  $\{Z\}$ ; and  $C_{U_{XY}} = \{X,Y\}$ ,  $C_{U_W} = \{W\}$ , and  $C_{U_Z} = \{Z\}$ . We refer the reader to [31, Chapter 7] for a more detailed review of SCMs.
2 Bandits with Transportability
From the agent's perspective, the point of departure with respect to conventional bandit instances is that in addition to the ability to take actions in a deployment environment  $\pi^{*}$ , the agent has access to data from one or more related environments  $\pi^a, \pi^b, \ldots$ , each characterized by SCMs  $M^a, M^b, \ldots$ . We assume that all environments have the same scope, i.e. the same sets  $\mathbf{V}$  and  $\mathbf{U}$ , but may differ in any other aspect. In the transportability literature [3, 32], such structural differences between environments are called domain discrepancies and can be encoded in selection diagrams.
Definition 1 (Domain Discrepancy). Let  $\pi^a$  and  $\pi^b$  be two domains with SCMs  $M^a$  and  $M^b$ . There exists a domain discrepancy between  $\pi^a$  and  $\pi^b$  if  $f_V^a \neq f_V^b$  or  $P^a(\mathbf{U}_V) \neq P^b(\mathbf{U}_V)$  for some  $V \in V$ .
Definition 2 (Selection diagram). Given domain discrepancy set  $\Delta^{a,b} \coloneqq \{V \in V : f_V^a \neq f_V^b$  or  $P^a(U_V) \neq P^b(U_V)\}$  between two domains  $\pi^a$  and  $\pi^b$  and a causal graph  $\mathcal{G}^a = (V, \mathcal{E})$ , let  $\mathbf{S} = \{S_V : V \in \Delta^{a,b}\}$  be called selection nodes. The graph  $\mathcal{G}^{a,b} = (\mathbf{V} \cup \mathbf{S}, \mathcal{E} \cup \{S_V \to V\}_{S_V \in \mathbf{S}})$  is called selection diagram.
Selection nodes indicate where structural discrepancies between two environments might take place. The absence of a selection node pointing to a variable represents the assumption that the causal mechanism responsible for assigning values to that variable is identical in both environments. In the clinical trial example, Fig. 1 shows a selection diagram comparing historical and clinical trial environments, denoted  $\pi^{*},\pi^{a}$  respectively; the presence of selection node  $S_{Z}$  indicates a potential difference in the assignment of  $Z$ , i.e., either  $f_{Z}^{*}\neq f_{Z}^{a}$  and / or  $P^{*}(\pmb{u}_{W})\neq P^{a}(\pmb{u}_{W})$  On the other hand, the absence of e.g. selection node  $S_{Y}$  indicates the assumption  $f_{Y}^{*} = f_{Y}^{a}$  and  $P^{*}(\pmb{u}_{Y}) = P^{a}(\pmb{u}_{Y})$  . With this formalism, the task is to leverage data from related environments in a consistent and efficient manner.
Definition 3 (Bandits with Transportability). Let  $\pi^{*}$  denote the deployment environment in which the agent acts. Given samples from  $P^a (\mathbf{V}), P^b (\mathbf{V}), \ldots$  and selection diagrams  $\mathcal{G}^{*,a}, \mathcal{G}^{*,b}, \ldots$ , in each round  $t = 1, \ldots, T$  the agent takes an action  $x^{(t)}$  and observes a sample from  $P^{*}(\mathbf{V}_{x^{(t)}})$ , adjusting its actions to minimize (expected) cumulative regret in  $\pi^{*}$ ,
$$
\mathbb {E} _ {P *} R _ {T} := \sum_ {t = 1} ^ {T} \mathbb {E} _ {P *} Y _ {\tilde {x}} - \mathbb {E} _ {P *} Y _ {x ^ {(t)}}, \tag {1}
$$
that compares the optimal intervention  $\tilde{x} = \arg \max_{x\in \Omega_X}\mathbb{E}_{P^*}Y_x$  with the agent's chosen intervention in each round.
Quantities such as  $\mathbb{E}_{P^*}Y_x$  or  $P^{*}(y_{x})$  are called transportability queries and their estimation with prior data, underlying structural assumptions and their use within active experimentation schemes will be the focus of this paper.
2.1 Informative priors for bandits
Before experimentation takes place there is a degree of unidentifiability of reward distributions  $P^{*}(y_{x})$  depending on causal assumptions and discrepancies between environments. For instance, revisiting the clinical trial example, if age distributions are allowed to vary arbitrarily across environments, values of  $P^{*}(y_{x})$  will similarly vary and thus involve a degree of uncertainty<sup>4</sup>. This unidentifiability feature is relevant even without major discrepancies across domains as  $P^{*}(y_{x})$  may still be unidentifiable in the presence of unobserved confounders. The Bow graph in Fig. 2 (ignoring the selection node) is a common example.
One may be tempted to conclude that prior data is rarely useful. However, even under multiple discrepancies across environments, causal effects  $P^{*}(y_{x})$  are rarely completely unconstrained. In general, causal effects lie in a non-trivial interval  $[a,b], 0 < a \leqslant b < 1$ . For instance, for the graph  $\mathcal{G}^{a,*}$  in Fig. 2  $P^{*}(y_{x})$  can be shown to be contained in  $[P^a (x,y),P^a (x,y) + 1 - P^a (x)]$ . In particular, with a probabilistic or Bayesian interpretation of unknown quantities in a SCM, that is with an explicit probability measure over SCMs  $M \in \mathcal{M}(\mathcal{G}^{*})^{5}$ , one
Figure 2: Bow graph and posterior reward density.
can define a distribution of reward probabilities  $P^{*}(y_{x})$  (or expected rewards) that honestly captures prior uncertainty in their values. For example, given prior samples  $\bar{\pmb{v}}^a = \{\pmb{v}_{(j)}^a : j = 1, \dots, 500\}$  independently drawn from a source distribution  $P^a(\mathbf{V})$  (and a particular prior model over SCMs, to be discussed in Sec. 2.2), a posterior density over  $\mathbb{E}_{P^*}Y_x$  can be evaluated and sampled from, see Fig. 2 for an illustration, where theoretical bounds are shown with vertical lines.
From this perspective, the problem of doing inference on reward distributions given prior data and knowledge of structural discrepancies can be formulated as an optimization problem, that is, evaluate,
$$
P _ {M \sim \mathcal {M} (\mathcal {G} ^ {*})} \left(\mathbb {E} _ {P _ {M}} \left[ Y _ {x} \right] \mid \bar {\boldsymbol {v}}\right), \quad \text {s u c h t h a t} \quad \forall V \notin \Delta^ {*, i}: f _ {V} ^ {*} = f _ {V} ^ {i}, P _ {\pi^ {*}} \left(\boldsymbol {u} _ {V}\right) = P _ {\pi^ {i}} \left(\boldsymbol {u} _ {V}\right), \tag {2}
$$
where  $\bar{\pmb{v}} := \{\bar{\pmb{v}}^a, \bar{\pmb{v}}^b, \dots\}$ ,  $\bar{\pmb{v}}^i = \{\pmb{v}_{(j)}^i : j = 1, \dots, n_i\}$  is a set of  $n_i$  independent samples from  $P^i(\mathbf{V})$ . In words, the task is to evaluate a distribution over expected rewards under intervention over all deployment domains  $\pi^*$  compatible with our knowledge prior to experimenting in  $\pi^*$ .
It remains a question, however, how to define a model and parameterization of SCMs. Remember that only prior data and selection diagrams are assumed to be available to the researcher; any choices on the distribution of exogenous variables  $P(\mathbf{U})$  or functional form of deterministic structural assignments  $\mathcal{F}$  represent untestable assumptions that are difficult to justify in practice. Going forward we will restrict ourselves to SCMs with discrete endogenous variables (while exogenous variables may be arbitrarily defined, e.g. continuously-valued with arbitrary probability density functions).
2.2 General parameterization of reward distributions
Systems of discrete observables have the distinctiveness of involving a finite number of probabilities of the form  $P^{*}(y_{x})$ , i.e. one for each combination  $(x,y)$ . Reward distributions  $P^{*}(y_{x})$  in any underlying SCM  $M$ , however complex  $P(\mathbf{U})$  and  $\mathcal{F}$  may be, can logically be equivalently expressed by a corresponding discrete SCM  $N$  in which  $P(\mathbf{U})$  is discrete and  $\mathcal{F}$  is a discrete mapping between finite spaces [44, 6]. This observation is interesting because it allows us to consistently and uniquely parameterize  $P(\mathbf{U})$  and  $\mathcal{F}$ , without untestable choices on their form<sup>6</sup>.
Formally, we say there exists multiple SCMs  $M_1, M_2 \in \mathcal{M}$  with functional dependencies defined by  $\mathcal{G}^*$ , consistent with prior data and selection diagrams, such that  $P_{M_1}(y_x) \neq P_{M_2}(y_x)$ .
$^{5}\mathcal{M}(\mathcal{G}^{*})$  stands for the set of SCMs whose functional dependencies are given by  $\mathcal{G}^*$ .
A similar reasoning does not apply for continuous endogenous variables that would require continuous exogenous variables and therefore a (untestable) choice of parametric family for all variables.
Algorithm 1 Thompson Sampling with Transportability (tTS)
Input: Selection diagrams  $\{\mathcal{G}*\ast ,a,\mathcal{G}^{*,b},\ldots \}$  prior data  $\bar{\pmb{v}}\coloneqq (\bar{\pmb{v}}^a,\bar{\pmb{v}}^b,\dots)$  decision variable  $X$  ,reward variable  $Y$  ,horizon  $T$    
for rounds  $t = 1,2,\ldots ,T$  do Approximate  $P(\pmb {\xi},\pmb {\theta}\mid \bar{\pmb{v}},\pmb{v}_{x^{(1)}},\dots ,\pmb{v}_{x^{(t - 1)}})$  Sample  $\pmb{\xi}^{(t)},\pmb{\theta}^{(t)}\sim P(\pmb {\xi},\pmb {\theta}\mid \bar{\pmb{v}},\pmb{v}_{x^{(1)}},\dots ,\pmb{v}_{x^{(t - 1)}})$ $x^{(t)}\gets \arg \max_{x}\mathbb{E}_{P*}\left[Y_{x}\mid \pmb{\xi}^{(t)},\pmb{\theta}^{(t)}\right]$  Take action  $x^{(t)}$  and observe  $\pmb{v}_{x^{(t)}}$  in  $\pi^*$    
end for
Corollary 1 (Proposition 2.7. [44]). For any causal graph  $\mathcal{G}$ , let  $M$  be an arbitrary SCM compatible with  $\mathcal{G}$ . For any sets  $Y, X \subset V$ , the interventional distribution  $P(\boldsymbol{y}_x)$  could be parameterized as
$$
\sum_ {\boldsymbol {v} \backslash \{\boldsymbol {x} \cup \boldsymbol {y} \}} \sum_ {u = 1, \dots , d _ {U},} \prod_ {V \in \boldsymbol {V}} \mathbb {1} \left\{\xi_ {V} ^ {\left(\boldsymbol {p a} _ {V}, \boldsymbol {u} _ {V}\right)} = v \right\} \prod_ {U \in \boldsymbol {U}} \theta_ {u}, \tag {3}
$$
where  $\theta_{u} := P(U = u)$  defines exogenous probabilities of discrete variables  $U \in U$  with cardinality  $d_{U} = \prod_{V \in \mathbf{C}_{U}} |\Omega_{\mathbf{P}\mathbf{a}(V)}|$ ; and each  $\xi_{V}^{(\mathbf{p}\mathbf{a}_{V}, \mathbf{u}_{V})}$  is a deterministic mapping between finite domains  $\Omega_{\mathbf{P}\mathbf{a}_{V}} \times \Omega_{\mathbf{U}_{V}} \mapsto \Omega_{V}$ .
For example,  $P^{*}(y_{x})$  in Fig. 1 can be parameterized by
$$
\sum_ {w, z, u _ {z}, u _ {w}, u _ {x y}} \mathbb {1} \left\{\xi_ {Y} ^ {(w, z, u _ {x y})} = y \right\} \mathbb {1} \left\{\xi_ {W} ^ {(x, u _ {w})} = w \right\} \mathbb {1} \left\{\xi_ {Z} ^ {(u _ {z})} = z \right\} \theta_ {u _ {x y}} \theta_ {u _ {z}} \theta_ {u _ {w}}, \tag {4}
$$
where, assuming  $X,Y,Z,W$  are binary,  $\theta_{u_z}$  is a discrete distribution over a finite domain  $\{1,2\}$  since  $|\Omega_{U_z}| = |\Omega_{Pa(Z)}| = |\Omega_Z| = 2$ ,  $\theta_{u_w}$  is a distribution over  $\{1,\dots,4\}$  since  $|\Omega_{U_W}| = |\Omega_{Pa(W)}| = |\Omega_X| \cdot |\Omega_Z| = 4$ , and  $\theta_{u_{xy}}$  is a distribution over  $\{1,\dots,32\}$  since  $|\Omega_{U_{XY}}| = |\Omega_{Pa(X)}| \cdot |\Omega_{Pa(Y)}| = |\Omega_Z| \cdot |\Omega_X| \cdot |\Omega_W| \cdot |\Omega_Z| \cdot |\Omega_Y| = 32$ . Coroll. 1 guarantees that for any value of  $P^*(y_x)$  induced by an arbitrary SCM  $M$  there exists a combination of parameters in Eq. (4) that reaches that exact same value. In other words, this parameterization is sufficiently expressive to encode any underlying reward distribution.
Parameters that define reward distributions are specific to the deployment environment  $\pi^{*}$ . The key observation, however, is that some of them can be inferred with prior data whenever there exists an invariance in  $P(U)$  or  $\mathcal{F}$  across environments as there exists a one-to-one relationship between model parameters and structural features of the underlying SCM. For example, given Fig. 1, the absence of a selection node into  $Y$  implies that both functional assignments and exogenous probabilities of  $Y$  agree across environments, that is  $\xi_{Y}^{*} = \xi_{Y}^{a},\theta_{u_{Y}}^{*} = \theta_{u_{Y}}^{a}$ . Both may thus be approximated with prior data which in turn constraints or informs  $P^{*}(y_{x})$  even if other parameters in its expression in Eq. (4) remain unknown. The result is a non-trivial distribution over reward probabilities that may be used to warm-start bandit algorithms, even before any experimentation takes place.
2.3 Bandit algorithms
To exploit non-trivial parameter distribution given prior data, a bandit algorithm can be designed to choose actions in proportion to the probability that an intervention leads to highest reward, also known as posterior or Thompson sampling [38, 2]. Specifically, at a particular round  $t$  of experimentation in the deployment domain  $\pi^{*}$ , posterior parameter distributions  $P(\pmb{\xi},\pmb{\theta}\mid \bar{\pmb{v}},\pmb{v}_{x^{(1)}},\dots,\pmb{v}_{x^{(t - 1)}})$  can be evaluated to exactly capture uncertainty given both prior and experimental data up to round  $t$ . Action  $x^{(t)}$  is then chosen according to the one that gives highest reward, i.e.  $\arg \max_{x}\mathbb{E}_{P*}\left[Y_{x}\mid \pmb{\xi}^{(t)},\pmb{\theta}^{(t)}\right]$ , where  $(\pmb{\xi}^{(t)},\pmb{\theta}^{(t)})$  is an independent draw from its posterior distribution. In other words, the agent
<sup>7</sup>It will be useful to write  $\pmb{\xi} = \{\xi_V^{(pa_V,u_V)}:V\in V,Pa_V\subset V,U_V\subset U\}$  and  $\pmb{\theta} = \{\pmb{\theta}_U:U\in U\}$  to group all possible functional assignments and exogenous probability parameters, respectively. We will in general omit environment superscripts on parameters to lighten the notation.