SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the following, we restrict our attention to a basic instance of the AUF problem: The structural equations  $f_{i}$ s are linear and the desired set  $S$  is a convex polytope, i.e.,
$$
\mathbf {V} _ {i} ^ {c} := \boldsymbol {\beta} _ {i} ^ {T} \mathbf {P} \mathbf {A} _ {i} ^ {G} + \varepsilon_ {i}, \mathcal {S} = \left\{\mathbf {y} \in \mathbb {R} ^ {| \mathbf {Y} |} \mid \mathbf {M} \mathbf {y} \leq \mathbf {d} \right\}, \tag {3}
$$
where  $|\cdot|$  denotes the cardinality of a set,  $\mathbf{d} \in \mathbb{R}^s$ ,  $\mathbf{M} \in \mathbb{R}^{s \times |\mathbf{Y}|}$ , and  $\beta_i \in \mathbb{R}^{|\mathbf{P}\mathbf{A}_i^G| \times |\mathbf{V}_i^c|}$ .
4Rehearsal Learning
In this section, we present a rehearsal learning framework to address the AUF problem in Eq. (2) and present specific methods for the linear case in Eq. (3), in which we use sampling and rehearsal of actions as a basic building block to find and evaluate decisions.
To account for the uncertainty introduced by limited data, we resort to Bayesian inference. The state of knowledge of the SRM is described by the posterior distribution:
$$
\mathbb {P} (G, \boldsymbol {\theta} \mid D _ {t}) = \mathbb {P} (G \mid D _ {t}) \mathbb {P} (\boldsymbol {\theta} \mid G, D _ {t}), \tag {4}
$$
where  $D_{t}$  is the evidence collected until the beginning of round  $t$ , consisting of the initial observational data  $D$  and  $\{(\mathbf{x}_i,\xi_i,\mathbf{z}_i,\mathbf{y}_i)\}_{i = 1}^{t - 1}$ . To handle the exploration-exploitation trade-off between choosing alterations that help recovers the true SRM and alterations that can avoid undesired future in the current round, we leave the choice to the agent by introducing a hyper-parameter  $\tau$  that balances these two kinds of alterations. In the  $t$ -th round, we propose to solve
$$
\max  _ {\xi_ {t}} \mathrm {I} (\mathbf {G}, \boldsymbol {\Theta}; \mathbf {Z} _ {t}, \mathbf {Y} _ {t} \mid D _ {t}, \mathbf {x} _ {t}, R h (\xi_ {t})) \tag {5}
$$
$$
\text {s . t .} \quad \mathbb {E} _ {G, \boldsymbol {\theta} | D _ {t}} \left[ \mathbb {I} \left(\mathbf {Y} _ {t} \in \mathcal {S} \mid G, \boldsymbol {\theta}, \mathbf {x} _ {t}, R h (\xi_ {t})\right) \right] \geq \tau . \tag {6}
$$
The constraint in Eq. (6) reflects the required performance: Performing the alteration should make the belief that the desired future can be successfully achieved greater than  $\tau$  given knowledge of the SRM so far. The objective function in Eq. (5) denotes the mutual information between the SRM and the observations revealed after the alteration. Overall, the solution to the optimization problem should avoid undesired future and meanwhile be most informative about the true SRM.
The proposed optimization problem introduces computational challenges, e.g., one has to sum over a super-exponential number of rehearsal graphs to compute the mutual information or the success probability. We divide rehearsal learning into three approximately tractable components: Bayesian update of SRMs, candidate alteration selection, and mutual information maximization. To better guide decision-making, we also present a sampling-based probably approximately correct (PAC) bound to quantify the risk associated with the selected alteration.
4.1 Bayesian Update of SRMs
Recall that the posterior of SRMs is divided into the posterior of graphs  $\mathbb{P}(G\mid D_t)$  and that of structural equation parameters  $\mathbb{P}(\pmb {\theta}\mid G,D_t)$  in Eq. (4). For the former term, we adopt a bootstrapping-based approximation [18], which learns multiple rehearsal graphs on bootstrapped data to build a set of graphs  $\mathcal{G}$  and weigh each graph  $G\in \mathcal{G}$  equally. The approximated posterior of graphs is
$$
\mathbb {P} (G \mid D _ {t}) \approx \frac {1}{| \mathcal {G} |} \sum_ {G ^ {\prime} \in \mathcal {G}} \mathbb {I} \left(G ^ {\prime} = G\right). \tag {7}
$$
Consequently, for the second term  $\mathbb{P}(\Theta \mid \mathbf{G}, D_t)$ , we only need to estimate parameters corresponding to graphs that appear in  $\mathcal{G}$ , which greatly reduces computation. The parameter posterior factorizes as
$$
\mathbb{P}(\boldsymbol {\theta}\mid G,D_{t}) = \prod_{G^{\prime}\in \mathcal{A}(G),\mathbf{V}^{c}\in G^{\prime}}\mathbb{P}(\boldsymbol{\beta}_{\mathbf{V}^{c}},\boldsymbol{\Sigma}_{\mathbf{V}^{c}}\mid G^{\prime},D_{t}),
$$
where  $\mathcal{A}(G)$  is a set of graphs that includes  $G$  and related alteration graphs and  $\mathbf{V}^c$  denotes a maximal bi-directional clique.  $\mathbb{P}(\beta_{\mathbf{V}^c}, \Sigma_{\mathbf{V}^c} \mid G', D_t)$  is the posterior of a set of regression parameters and noise variance given data of  $\mathbf{V}^c$  and its parents in  $G'$ . So various Bayesian learning methods [19] are applicable for its estimation. We adopt Bayesian ridge regression [20] to learn the parameters. In addition, as graph learning is time-consuming, we use an incremental implementation: We build an initial graph posterior from purely observational data for  $t = 0$  using Eq. (7), then update the posterior with
$$
\mathbb {P} (G \mid D _ {t}) \propto \mathbb {P} (G \mid D _ {t - 1}) \cdot \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {P} (\mathbf {z} _ {t}, \mathbf {y} _ {t} \mid \mathbf {x} _ {t}, \xi_ {t}, G, \boldsymbol {\theta} ^ {i}),
$$
where  $\pmb{\theta}^i$  is sampled from  $\mathbb{P}(\pmb{\theta} \mid G, D_{t-1})$  and the averaged summation term is an empirical estimate of likelihood. With the above approximations and learning, we effectively obtain an approximate posterior and an efficient sampler over SRMs, which forms the basis of the following two steps.
4.2 Candidate Alteration Selection
In the second part, we find candidate alterations that satisfy the performance constraint in Eq. (6). The constraint cannot be accurately computed as it takes an expectation over discrete  $G$  and continuous  $\theta$ . To reduce the computational burden, we estimate the constraint with posterior samples. Note that the value of variables  $\mathbf{V}$  is uniquely determined by  $G$ ,  $\theta$ , and the noise  $\varepsilon$  in Eq. (1). We can evaluate all alterations by generating  $\mathbf{Y}_t$  from a common set of SRMs and noise samples. Suppose we have an i.i.d. sample  $S = \{\langle G_i, \theta_i, \varepsilon_i \rangle\}_{i=1}^n$ , the empirical estimate of Eq. (6) is
$$
\hat {p} _ {\xi_ {t}} (S) \triangleq | \left\{i \mid \mathbf {y} _ {t, \xi_ {t}} ^ {i} \in \mathcal {S} \right\} | / n,
$$
where  $\mathbf{y}_{t,\xi_t}^i$  is the concerned outcome generated following the SRM specified by  $\langle G_i, \pmb{\theta}_i, \pmb{\varepsilon}_i \rangle$  and  $Rh(\xi_t)$ . A general method for candidate alteration selection has three stages:
1. Sample two sets of i.i.d. parameters, the training set  $S_{tr} = \{\langle G_i,\pmb {\theta}_i,\pmb {\varepsilon}_i\rangle \}_{i = 1}^n$  and the validation set  $S_{val} = \{\langle G_i,\pmb {\theta}_i,\pmb {\varepsilon}_i\rangle \}_{i = n + 1}^{2n}$ , from  $\mathbb{P}(G,\pmb {\theta}\mid D_t)$  and  $\mathbb{P}(\pmb {\varepsilon}\mid \pmb {\theta})$ .
2. Build a candidate alteration set  $\mathcal{C}$  by conducting rehearsals on  $S_{tr}$  and finding alterations  $\xi_t$  that achieve required performance  $\tau$ , i.e.,  $\forall \xi_t \in \mathcal{C}, \hat{p}_{\xi_t}(S_{tr}) \geq \tau$ .
3. Validate alterations in  $\mathcal{C}$  by rehearsals on  $S_{val}$  and remove those with  $\hat{p}_{\xi_t}(S_{val}) < \tau$  from  $\mathcal{C}$ .
The validation procedure is to alleviate overfitting as in standard machine learning pipelines. If stage 2 or 3 outputs an empty set, the system should refuse to make any recommendations. The agent can lower  $\tau$  and restart if a smaller  $\tau$  is still adequate for the task at hand.
For a given alteration, validation is relatively easy to perform. The only issue to address is to find a set of alterations that pass the test on  $S_{tr}$ . An immediate approach would be enumerating possible alterations and making rehearsals on training samples to check the constraint. But the enumeration can be computationally prohibitive as the alteration space can be combinatorial and continuous. A compromise between the optimality of recommended alterations and the computational feasibility is to work on some finite subset of alterations using some discretization tricks or heuristics. Fortunately, due to the linear structure of the SRMs considered here, Prop. 5 shows that the outcome variables are linear in the alteration values, implying the existence of more efficient and effective solutions.
Proposition 5. Let  $\xi = \{(Z_{a_i},z_{a_i})\}_{i = 1}^k$  be a valid alteration where  $Z_{a_i}\in \mathbf{Z}_t$  and  $z_{a_i}\in \Delta (Z_{a_i})$ . Given  $\langle G,\pmb {\theta},\pmb {\varepsilon}\rangle$  and  $\mathbf{x}_t$ , the outcome variables are linear in the alteration values. We have
$$
\mathbf {Y} _ {t} = \mathbf {A} \mathbf {x} _ {t} + \mathbf {B} \mathbf {z} ^ {\xi} + \mathbf {C} \varepsilon , \tag {8}
$$
where  $\mathbf{z}^{\xi} = (z_{a_1},\dots ,z_{a_k})^T$  . A, B, and C are constant matrices of appropriate shapes and are uniquely determined by  $G,\theta$  and altered variables  $\mathbf{Z}^{\xi} = \{Z_{a_i}\}_{i = 1}^{k}$
Size-1 Alteration. We consider cases where an agent can alter only one variable in each round, i.e.,  $k = 1$ . Suppose the altered variable in round  $t$  is  $Z_{i}$ . Provided with a fixed SRM, along with fixed noise, Eq. (8) reduces to a line in  $\mathbb{R}^{|\mathbf{Y}|}$  with direction  $\mathbf{B} \in \mathbb{R}^{|\mathbf{Y}|}$ . Due to the convexity of the desired region  $S$ , if altering  $Z_{i}$  can make  $\mathbf{Y} \in S$ , then the set of feasible alteration values can be represented by an interval  $[u,v]$  (the endpoints can be infinity). Finding candidate alterations on the training data is then equivalent to finding elements that intersect at least  $n \cdot \tau$  intervals obtained from samples in  $S_{tr}$ , which is easy to compute. Alg. 1 includes the ba
sic training and validation stages of finding candidate alterations. The overall time complexity is  $O(|\mathbf{Z}_t| \cdot n \log n)$ . A more detailed algorithm and running time analysis are given in Appendix B.
Algorithm 1 Finding candidate alterations of size 1
Input: Str = {<Gi, θi, εi>}n i=1, Sval = {<Gj, θj, εj>}j
1: C ← ∅ ▷ Store candidate alterations
2: for Zi ∈ Z do
3: int1 ← FIND(Str, Zi, τ) ▷ Training
4: int2 ← FIND(Sval, Zi, τ)
5: C ← C ∪ {<Zi, int1 ⊩ int2>} ▷ Validation
6: function FIND(S, Z, τ) ▷ Find alteration values for Z
7: int ← ∅ ▷ Store valid intervals
8: for <G, θ, ε> ∈ S do
9: [u, v] ← solution of M(Ax + Bzξ + Cε) ≤ d
10: ▷ A, B, C are determined by G, θ, Z
11: int ← int ∪ {[u, v] ⊩ Δ(Z)}
12: return {z | nτ ≤ #intervals in int containing z}
Output: The found alterations C
Size- $k$  Alteration. By relating to the NP-hard maximum feasible subsystem problem [21], we prove Thm. 6, showing that finding a single  $\xi$  that satisfies the constraint, which is much simpler
than finding all valid alterations, is difficult if  $\mathrm{P} = \mathrm{NP}$ . This result suggests that we should focus on approximate solutions rather than exact ones. We also give a mixed integer linear programming approach in Appendix B.
Theorem 6. Unless  $\mathrm{P} = \mathrm{NP}$ , finding a  $k$ -dimensional  $\mathbf{z}^{\xi} \in \Delta(\mathbf{Z}^{\xi})$  that satisfies  $\sum_{i=1}^{n} \mathbb{I}(\mathbf{M}(\mathbf{A}_i \mathbf{x} + \mathbf{B}_i \mathbf{z}^{\xi} + \mathbf{C}_i \boldsymbol{\varepsilon}_i) \leq \mathbf{d}) \geq n \cdot \tau$  (if there is a valid solution) is not solvable with any algorithm of running time polynomial in  $k$  and  $n$ .
4.3 Mutual Information Maximization
The remaining task is to optimize the mutual information (5) over the found candidate alteration set  $\mathcal{C}$ . We factorize the objective function with two information entropy terms over  $\mathbf{Z}_t$  and  $\mathbf{Y}_t$ :
$$
I \left(\mathbf {G}, \boldsymbol {\Theta}; \mathbf {Z} _ {t}, \mathbf {Y} _ {t} \mid D _ {t}, \mathbf {x} _ {t}, \xi_ {t}\right) = H \left(\mathbf {Z} _ {t}, \mathbf {Y} _ {t} \mid D _ {t}, \mathbf {x} _ {t}, \xi_ {t}\right) - H \left(\mathbf {Z} _ {t}, \mathbf {Y} _ {t} \mid \mathbf {G}, \boldsymbol {\Theta}, D _ {t}, \mathbf {x} _ {t}, \xi_ {t}\right). \tag {9}
$$
Eq. (9) can be estimated with samples from  $\mathbb{P}(G,\pmb {\theta}\mid D_t)$  and  $\mathbb{P}(\mathbf{z}_t,\mathbf{y}_t\mid G,\pmb {\theta},\mathbf{x}_t,\xi_t)$ . Due to the sampling involved in the estimation procedure, Eq. (9) is expensive to evaluate. The continuous input space makes it difficult to find an optimal alteration value for a fixed alteration target  $\mathbf{Z}^{\xi}$ . Thus, we find  $\mathbf{z}^{\xi *}^*$  with Bayesian optimization [22], which seeks to optimize a black-box function  $g(\mathbf{z})$  with a small number of evaluations by iteratively querying function values at some data points. We run the optimization procedure for every alteration target in  $\mathcal{C}$  and find the corresponding best alteration values. Finally, the alteration that gives maximum mutual information is selected as the recommended output. Details about the estimation procedure and Bayesian optimization are given in Appendix C.
Given a decision, it is important to know how likely the decision can successfully avoid the undesired future. Though the candidate selection procedure already rules out potentially terrible alterations using  $S_{tr}$  and  $S_{val}$ , a more refined measure is preferable. Based on the scenario approach in robust control literature [23, 24], we derive a PAC-style bound on the posterior success probability in Thm. 7. The bound is easy to compute as it only requires posterior samples. The practical implication of the bound is that it quantifies the uncertainty, or risk, associated with the alteration, thus can help make better decisions. Notably, the bound holds for any SRM with arbitrary additive noise, which could be helpful for future extensions with non-Gaussian noise and nonlinear SRMs.
Theorem 7. Given an observed evidence  $\mathbf{x}_t$ , an alteration  $\xi$ , and a desired set  $S = \{\mathbf{y} \in \mathbb{R}^{|\mathbf{Y}|} \mid \mathbf{M}\mathbf{y} \leq \mathbf{d}\}$ , let  $S_{eval} = \{\langle G_i, \theta_i, \varepsilon_i \rangle\}_{i=1}^n$  be  $n$  i.i.d. samples from  $\mathbb{P}(G, \theta, \varepsilon \mid D_t)$ . Let  $\{\mathbf{y}^i\}_{i=1}^n$  be generated from  $Rh(\xi)$  and  $S_{eval}$  following the structural equations in Eq. (1). Define  $n_o \triangleq |\{i \mid \mathbf{y}^i \notin S\}|$ . Let  $\hat{p} \triangleq 1 - n_o / n$ ,  $p \triangleq 1 - F^{-1}(1 - \delta / (2n); n_o + 1, n - n_o)$  if  $n_o < n$  otherwise  $p \triangleq 0$ , and  $\bar{p} \triangleq 1 - F^{-1}(\delta / (2n); n_o, n - n_o + 1)$  if  $n_o > 0$  otherwise  $\bar{p} \triangleq 1$ , where  $F^{-1}(\cdot; \alpha, \beta)$  is the inverse cumulative distribution function of the beta distribution with parameters  $\alpha$  and  $\beta$ . For any  $\delta \in (0, 1)$ , with probability at least  $1 - \delta$ , we have
$$
\max  \left\{\underline {{p}}, \hat {p} - \sqrt {\ln (2 / \delta) / 2 n} \right\} \leq \mathbb {P} \left(\mathbf {Y} _ {t} \in \mathcal {S} | D _ {t}, \mathbf {x} _ {t}, R h (\xi)\right) \leq \min  \left\{\bar {p}, \hat {p} + \sqrt {\ln (2 / \delta) / 2 n} \right\}.
$$
The proposed rehearsal learning framework is general and flexible, with any of the three steps adjustable without interfering with the others. For scenarios more complicated than linear ones, one can replace methods in each step accordingly: The Bayesian update of SRMs can use any advanced Bayesian (structure) learning methods [25, 26, 27, 28] as long as they allow sampling from the posterior. The mutual information maximization step can leverage any black-box optimization methods [22, 29]. For the most important and difficult candidate alteration finding step, which is difficult even for the linear case when allowing joint alterations, efficient heuristics or linearization tricks can be used. Given the great uncertainty of real-world problems, exactly solving the AUF problem is extremely difficult if not impossible, while with the rehearsal learning framework, we provide a practical and systematic approximate solution that tackles the problem with certain guarantees, which exhibits that developing more efficient approximate solutions is a promising future direction.
For the SRM, there are some practical considerations. After a rehearsal operation, the change of the parental relations results in new sets of parameters. In the absence of further constraints, as is the case in this work, these new parameters may bear no direct resemblance to the previous ones. However, it is conceivable to impose specific assumptions on these parameters to achieve more
efficient modeling. For instance, one may assume the new parameters are connected with the original ones through operations like marginalization. In practical implementations, it is also reasonable to limit the size of a clique and maintain the number of parameter sets to an acceptable level. Moreover, it is unlikely that we would need to or be able to alter every possible subset of a clique, so one can reduce unnecessary computational burden by ignoring the parameters that would not be used.
There have been some efforts to combine structural models with decision-making, with most of them focusing on causal structures [30]. Some work explores the causal bandit problem, where the causal structure underlying rewards and actions is considered [31, 32, 33, 34, 35, 36]. In a different vein, Aglietti et al. [37] investigated a Bayesian optimization problem with a known causal DAG, and Aglietti et al. [38] extended this work to a dynamic setting. However, these studies typically assume that the true causal graph is known, a condition that is challenging to meet in real-world applications, although causal discovery methods could help to some extent [39, 40, 41, 42, 43, 44]. To remove the assumption, de Kroon et al. [45] leveraged separating sets, while Lu et al. [46] focused on the Markov equivalence class of the true causal graphs. However, even when causal relations are known, applying causal bandits or causal Bayesian optimization to AUF is not suitable because they often seek a single universally optimal action. In contrast, optimal decisions in AUF can vary depending on the context X. Some research approaches decision problems from the perspective of causal structures and causal effect estimation. For example, Wang et al. [47] developed a method for estimating a set of possible causal effects to aid decision-making. Additionally, there has been active research in estimating causal structures or effects in interactive environments [48, 49, 50, 51, 52].
The aforementioned efforts yielded effective methods for certain decision-related problems, but all of them rely on causal modeling, which could be too luxurious and restrictive for decision problems [7]. On the other hand, correlation, which is the basis of most ML models, is insufficient for decision-making. Therefore, we turn to the influence relation, which lies between correlation and causation and forms a basis of decisions [7]. Building on the influence relation, we propose SRM, which is capable of modeling interrelated but not necessarily causally linked variables and dynamically evolving decision systems. Moreover, the proposed rehearsal learning framework demonstrates the feasibility of building decision-making upon influence and rehearsal.
We evaluate the proposed approach on two datasets. We are mainly interested in if the proposed approach can successfully avoid the undesired future with high probability, the exploration-exploitation trade-off, and the informativeness of the PAC-style guarantees. For each dataset, we alter one variable in each round and repeat experiments with 100 rounds 20 times. The graph prior is initialized with samples from the learnable graph equivalence class instead of learning from data as well-established rehearsal graph learning methods are still in the process of development. The success probability is estimated with 1,000 samples from the true SRM. We compute the PAC bound with 1,000 samples from the posterior with  $\delta = 0.05$ . The observational dataset size is set to 10. We also compare with several reinforcement learning methods DDPG [53], PPO [54], SAC [55], and CATS [56].
Ride-Hailing Data. We abstract an SRM from a ride-hailing scenario, where a ride-hailing app needs to make decisions to promote user rating (RAT). We consider relevant variables including the weather condition, the number of users, traffic congestion (TRA), etc. The app can alter two variables, the discount level (DIS) and the recommendation level (REC) of a specific route. There exist interrelated variables in the scenario: If REC of a specific route is high, then TRA on that route will be high since more users will choose that route; and if there is a high TRA on that route, the REC should be low. The size of  $\mathbf{X}_t$ ,  $\mathbf{Z}_t$ , and  $\mathbf{Y}_t$  are 2, 4, and 1 respectively. The true structural equations are set according to domain knowledge. The feasible alteration values are  $[-2, 2]$  for DIS and REC. The range of RAT is  $[0, 1]$ , so we set the desired region to  $S = [0.8, 1]$  to avoid RAT below 0.8.
Bermuda Data. We take an example from ecology, where environment variables in Bermuda are recorded [57] and the variable generation order is available [58]. The size of  $\mathbf{X}_t$ ,  $\mathbf{Z}_t$ ,  $\mathbf{Y}_t$  are 3, 7, and 1 respectively. The true structural equations are obtained by performing linear regression on
Figure 4: Results on the Ride-hailing and the Bermuda data. The bands depict standard deviations.
Number of rounds
normalized data. We assume that 5 variables are actionable [37], with a feasible alteration region  $[-1,1]$ .  $\mathbf{Y}_t$  represents the net coral ecosystem calcification (NEC). We want to maintain a high NEC. So we set the desired region to  $\mathcal{S} = [0.5,2]$  which is above the 75th percentile of the dataset.
Table 1 shows the average probabilities of successfully avoiding the undesired future of several RL methods and the proposed rehearsal learning method with  $\tau = 0.7$ . Our method achieves the goal of AUF with a probability around 0.7, while the others fail to do so with merely 100 interac
Dataset	DDPG	PPO	SAC	CATS	τ = 0.7
Ride-hailing	0.173	0.154	0.177	0.104	0.714
Bermuda	0.230	0.190	0.205	0.215	0.679
Table 1: Avg. success probability.
tions with the environment, which further underscores the importance of considering structural information in the interaction-limited AUF problem. On the other hand, if we increase  $T$ , RL methods can achieve satisfying performance, e.g., DDPG achieves 0.688 average success probability when  $T = 10,000$  on the Bermuda data. For the ride-hailing data,  $95.8\%$  alterations recommended by the proposed method are to alter REC, which affects RAT through the interactions between REC and TRA and the direct link between TRA and REC:  $\mathrm{REC} \leftrightarrow \mathrm{TRA} \rightarrow \mathrm{RAT}$ . Note that REC cannot be identified as a valid cause so causal modeling is not suitable for this problem.