SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tian Qin, Tian-Zuo Wang, Zhi-Hua Zhou  
National Key Laboratory for Novel Software Technology  
Nanjing University, Nanjing, 210023, China  
{qint, wangtz, zhouzh}@lamda.nju.edu.cn
Machine learning (ML) models have been widely used to make predictions. Instead of a predictive statement about future outcomes, in many situations we want to pursue a decision: what can we do to avoid the undesired future if an ML model predicts so? In this paper, we present a rehearsal learning framework, in which decisions that can persuasively avoid the happening of undesired outcomes can be found and recommended. Based on the influence relation, we characterize the generative process of variables with structural rehearsal models, consisting of a probabilistic graphical model called rehearsal graphs and structural equations, and find actionable decisions that can alter the outcome by reasoning under a Bayesian framework. Moreover, we present a probably approximately correct bound to quantify the associated risk of a decision. Experiments validate the effectiveness of the proposed rehearsal learning framework and the informativeness of the bound.
Machine learning (ML) has achieved great success in various applications, including computer vision [1], natural language processing [2], recommender systems [3, 4], etc. In addition to perception tasks such as classifying an image, ML models have been widely used to make predictions, or forecasting, about future quantities of interest [5, 6]. In many scenarios, however, what we ultimately pursue is typically not attaining merely a predictive statement on what is likely to happen. Instead, as a potential next direction [7], we may seek a way, possibly through decision-making, to avoid the happening of the upcoming undesired future, if the ML model predicts so.
Suppose we use variables  $\mathbf{X}$  to predict a future outcome  $\mathbf{Y}$  with an ML model  $h$ . Given an instance  $\mathbf{x}$ ,  $h$  outputs a warning signal  $\hat{y} = h(\mathbf{x})$  which is outside our desired range. The problem is: what can we do to make the future  $\mathbf{Y}$  fall into the desired range? We assume that there is an intermediate stage where one can make actionable decisions to influence  $\mathbf{Y}$ ; e.g., after a sales prediction made at the beginning of a month, there is an intermediate time window before the month-end for the sales manager to take action. We consider decisions in the form of altering variables with fixed values, e.g., setting the discount to  $10\%$ . We henceforth call such decisions alterations. Let  $\mathbf{Z}$  denote the variables in the intermediate stage. The avoiding undesired future (AUF) problem is then how to alter variables in  $\mathbf{Z}$  so that the future  $\mathbf{Y}$  could be shifted to fall into the desired range.
Common decision-making methods under the umbrella of reinforcement learning (RL) [8] can be applied to the AUF problem but may not be the most appropriate. The reasons include that (a) RL mainly focuses on sequential decision-making tasks while AUF desires a direct decision to change the upcoming future outcome, for which considering decision sequences could be unnecessary or even unrealistic; (b) the success of RL in playing games [9] and autonomous control [10] rely on sufficient interactions between the agent and the environment, but interactions in real AUF problems can be extremely sparse, e.g., the sales manager in the above example can only interact with the environment and make decisions once a month; (c) the Markov decision process (MDP) formalism in RL abstracts decision-making into states, actions, and rewards, which may overlook useful fine-grained structural
information in AUF, e.g., the connections between variables that can be altered may help identify useful actions without any exploration. Explicitly incorporating such connections in modeling the AUF problem would be preferable since otherwise approximating them with MDPs may require unnecessarily large state or action spaces.
Therefore, we need a specific method for solving AUF that takes account of the structural information and avoids the use of the large number of interactions. An immediate thought would be seeking cause-effect relations between variables [12], which have been leveraged for some decision-related problems (see Section 5 for a discussion of related work). But as indicated by Zhou [7], causal relations can help but should not be taken as a prerequisite for decision-making for reasons that (a) humans can make good decisions without a thorough or faithful causal understanding of the surrounding
Figure 1: Relationship between correlation, influence, and causation [7, 11].
environment, sometimes correct decisions can even be made based on an incorrect causal understanding; (b) guiding decisions with causal relations is not always sensible as causation reveals truths that always hold, whereas decision-making is coping with real environments which are often open and dynamic; (c) causal modeling could be helpless for decision-making if the identified causal factors are unactionable; needless to say, it is very difficult to discover true cause-effect relations from data [13]. Recognizing that correlation is helpful for prediction but insufficient for decision-making and that causation is needed for scientific discovery but too luxury to be relied on for decision-making, Zhou [7] claimed that what is required for decision-making is a kind of intermediate relation that is stronger than correlation but less demanding than causation; this relation was later called influence relation [11]. Moreover, Zhou [7] attributed a decision to a series of hypothesized "rehearsal" of possible actions, which is an intuitive way of leveraging or even discovering the influence among variables involved in a decision task. Fig. 1 depicts an intuitive demonstration of the relationship among correlation, influence, and causation.
Based on the concept of rehearsal, we present the structural rehearsal model (SRM) to model the AUF problem. The SRM consists of a new probabilistic graphical model called rehearsal graphs and a collection of generative structural equations. In contrast to traditional structural causal models (SCM) [12], the SRM allows dynamic modeling and accommodates the influence relation among variables, effectively capturing the interactions between interrelated but not necessarily causally linked variables. Given the true SRM, the effect of alterations can be calculated or estimated by conducting rehearsals. Consequently, the AUF problem can be addressed by searching for optimal alterations that yield the desired outcomes, though the search process could be complex and difficult.
Note that structural models such as SRMs are generally not available in advance, and learning them from limited observational data is difficult as well. In solving the AUF problem, one needs to tackle challenges posed by limited data, the uncertainty of structural models, and the possibly huge search space for finding an optimal alteration. Moreover, a system outputting decisions should provide some kind of guarantee on the probability of successfully avoiding undesired future, especially in high-stake applications such as economic and safety tasks. The AUF problem can also come in an online fashion and induce an exploration-exploitation trade-off: a decision-maker can choose alterations that will reveal more structural information to benefit future decision-making or alterations that will maximize the success probability in the current decision round.
To address the aforementioned challenges and tackle the AUF problem, we introduce a rehearsal learning framework that integrates structural rehearsal models and Bayesian inference. This framework effectively captures the inherent uncertainty in decision-making processes aimed at averting undesired outcomes by unifying structural modeling, alteration finding, and success probability bounding. Our contributions are as follows:
1. Structural Rehearsal Model: We propose SRM, a novel modeling approach distinct from SCM, designed to model the influence relation, which is more essential than causal relations for decision-making problems.
2. Constrained Optimization for AUF: We formulate a constrained optimization problem to solve the AUF problem, striking a balance between averting undesired future scenarios and accurately learning rehearsal models.
3. Rehearsal Learning Framework: We develop a rehearsal learning framework tailored for solving AUF, showing that building decisions based on the influence relation is practical and feasible. We propose specific learning methods for the basic linear case and show that optimal solutions are not attainable in polynomial time in the framework if joint alterations are allowed, assuming  $\mathrm{P} \neq \mathrm{NP}$ . This insight underscores the significance of developing approximate solutions over exact ones.
4. Risk Quantification: We introduce a probably approximately correct bound to quantify the associated risk inherent in a given decision. Our experiments provide empirical evidence of the informativeness of this bound.
2 Structural Rehearsal Models
Conventionally, SCMs [12] are used to describe causal relations among variables, based on which actions that affect the outcome could be found. However, in some real-world problems, especially those involving decisions, causal relations are not adequate. For example, let P and Q denote the prices of a product in two stores, respectively. If the first store decreases P to attract customers, the second store will decrease Q accordingly because the second store will lose consumers otherwise. From a causal view, it seems that P causes Q. But obviously, Q also causes P by symmetry. So we have a bi-directional "causal" relation, which is invalid in causal modeling. Mistakes will occur if one applies SCMs to this example as only a one-way causal relation is allowed in SCMs. The issue of such interrelated but not necessarily causally linked variables, as well as several other issues such as the dynamic and time-dependent nature of decision-making, which will be discussed later, are taken into account in the following new structural model called the structural rehearsal model (SRM).
An SRM consists of a set of rehearsal graphs and corresponding structural equations. A rehearsal graph qualitatively describes the relations between variables, and the structural equations characterize the generating process of variables in detail. In contrast to static causal modeling, SRM allows dynamic modeling by defining both the graphs and equations over a time index  $t$ , which accounts for possible evolutions of the environment. We represent an SRM with  $\{\langle G_t, \theta_t \rangle\}_t$ , where  $G_t$  denotes the rehearsal graph at time  $t$  and  $\theta_t$  parameterizes corresponding structural equations.
Rehearsal graphs allow both directional and bi-directional edges. The directional edges depict the generation ordering of variables and the bi-directional ones indicate interrelated variables that are mutually influenced. Fig. 2a depicts an example. The definition of rehearsal graphs is given below:
Definition 1 (Mixed graph). Let  $G = (\mathbf{V}, \mathbf{E})$  be a graph, where  $\mathbf{V}$  denotes the vertices and  $\mathbf{E}$  the edges.  $G$  is a mixed graph if for any distinct vertices  $u, v \in \mathbf{V}$ , there is at most one edge connecting them, and the edge is either directional ( $u \to v$  or  $u \leftarrow v$ ) or bi-directional ( $u \leftrightarrow v$ ).
Definition 2 (Bi-directional clique). A bi-directional clique  $C = (\mathbf{V}^c, \mathbf{E}^c)$  of a mixed graph  $G = (\mathbf{V}, \mathbf{E})$  is a complete subgraph induced by  $\mathbf{V}^c \subseteq \mathbf{V}$  such that any edge  $e \in \mathbf{E}^c$  is bi-directional.  $C$  is maximal if adding any other vertex does not induce a bi-directional clique.
Definition 3 (Rehearsal graph). Let  $G = (\mathbf{V}, \mathbf{E})$  be a mixed graph. Let  $\{C_i\}_{i=1}^l$  denote all maximal bi-directional cliques of  $G$ , where  $C_i = (\mathbf{V}_i^c, \mathbf{E}_i^c)$ .  $G$  is a rehearsal graph if and only if:
1.  $\mathbf{V}_i^c\cap \mathbf{V}_j^c = \emptyset$  for any  $i\neq j$
2.  $\forall i\in [l],u\in \mathbf{V}\setminus \mathbf{V}_i^c$  , if there is any edge pointing from  $u$  to  $\mathbf{V}_i^c$  , then  $\forall v\in \mathbf{V}_i^c,u\to v$
3. The directional edges permit a topological ordering for  $\{C_i\}_{i=1}^l$
Each vertex in a rehearsal graph corresponds to a variable. And the variables are generated following the ordering depicted by the directional edges. The variables in a common maximal bi-directional clique, where only bi-directional edges exist, are mutually influenced instead of having a fixed generating order. Given a rehearsal graph  $G$ , structural equations accompanying the rehearsal graph are defined over cliques  $\{(\mathbf{V}_i^c, \mathbf{E}_i^c)\}_{i=1}^l$ . Let  $\mathbf{PA}_i^G \triangleq \{u \mid \exists v \in \mathbf{V}_i^c, u \to v \text{ in } G\}$  denote parents of the  $i$ -th maximal bi-directional clique. Suppose that the unobserved noise variables are Gaussian, then the structural equation describing the generation process of  $\mathbf{V}_i^c$  is parameterized by  $\{\beta_i, \Sigma_i\}_{i=1}^l \subseteq \pmb{\theta}$ :
$$
\mathbf {V} _ {i} ^ {c} := f _ {i} \left(\mathbf {P A} _ {i} ^ {G}; \beta_ {i}\right) + \varepsilon_ {i}, \tag {1}
$$
where  $f_{i}$  is a multi-valued function parameterized by  $\beta_{i}$  and  $\varepsilon_{i} \sim \mathcal{N}(0, \Sigma_{i})$  denotes the noise.
We denote the operation that alters variable  $\mathbf{X}$  with fixed value  $\mathbf{x}$  by  $Rh(\mathbf{X} = x)$ , meaning that we perform a rehearsal of setting  $\mathbf{X}$  to  $\mathbf{x}$  through realistic or hypothetical means. When  $Rh(\mathbf{X} = \mathbf{x})$  is
(a)  $G$
(b)  $G^{\mathrm{F}}$
(c)  $G^{\mathrm{B,E,F}}$
(d)  $\{\langle G_t,\pmb {\theta}_t\rangle \rangle_t$
Figure 2: (a) displays an example rehearsal graph. (b) and (c) displays its alteration graphs with rehearsal operations on  $\{\mathrm{F}\}$  and  $\{\mathrm{B},\mathrm{E},\mathrm{F}\}$  respectively. We use plate notation to simplify the edges: If an endpoint of an edge falls exactly on the boundary of a rectangle, then there are edges for every vertex in that rectangle. (d) shows a full SRM  $\{\langle G_t,\pmb {\theta}_t\rangle \}_{t}$ , where  $t$  is the time index and the rehearsal graph  $G_{t}$  and parameters  $\pmb{\theta}_{t}$  can evolve over time. Recall the example of two stores and let the variables follow the depicted SRM. Suppose that at  $t = 1$ , the second store takes a new marketing strategy that no longer follows the price of the first store, then P will not directly affect Q: The generation process evolves to  $\mathrm{P}\gets \mathrm{Q}$  in  $G_{1}$  instead of  $\mathrm{P}\leftrightarrow \mathrm{Q}$  in  $G_{0}$ . Then for the first store, a previously correct decision at  $t = 0$ , which aims to affect C by altering P, is ineffective at  $t = 1$ .
applied, we remove the incoming arrows of  $\mathbf{X}$  in the original rehearsal graph  $G$  to get an alteration rehearsal graph  $G^{\mathbf{X} = \mathbf{x}}$ , and the accompanying structural equations are modified according to the newly introduced parental relations in  $G^{\mathbf{X} = \mathbf{x}}$ , where a new set of parameters in  $\theta$  could be introduced for the unseen parental relations to offer more flexible modeling abilities. The distribution  $\mathbb{P}(\mathbf{V} \mid Rh(\mathbf{X} = \mathbf{x}))$  of all variables after applying  $Rh(\mathbf{X} = \mathbf{x})$  can be derived from Eq. (1). Example illustrations are in Figs. 2b-2c. The "invalid" bi-directional "causal" relations are valid in rehearsal graphs:  $\mathrm{P} \leftrightarrow \mathrm{Q}$  becomes  $\mathrm{P} \rightarrow \mathrm{Q}$  when applying  $Rh(\mathrm{P})$  and becomes  $\mathrm{P} \leftarrow \mathrm{Q}$  when applying  $Rh(\mathrm{Q})$ , capturing the interrelated influence between the two variables.
An important feature of SRM that distinguishes it from SCM is the capability of modeling dynamic and time-dependent real-world decision-making environments, where variable relations may evolve and correct decisions can vary dramatically at different times even for the same problem [14]. It is worth noting that these dynamic environments should not be confused with the conventional notion of dynamic systems, where the relationships among variables simply repeat over time. As causal modeling seeks to identify enduring cause-and-effect relationships, SCMs can be helpful in tasks such as "AI for science" [15, 16, 17], where the objective is to uncover persistent scientific truths. However, SCMs can be restrictive in describing the dynamic nature of decision problems. In contrast, the dynamic and time-dependent feature of environments is captured by SRM using evolving rehearsal graphs and corresponding parameters: At time  $t$ , the generating process characterized by  $\langle G_t, \theta_t \rangle$  can differ from previous ones. Fig. 2d illustrates an example where correct decisions vary across time and an SRM  $\{\langle G_t, \theta_t \rangle\}_t$  is used to model the dynamic decision environment.
Although  $\theta_{t}$  can be learned from data with general supervised learning methods given a fixed graphical structure, generally rehearsal graphs could not be uniquely identified from data. We present basic properties of rehearsal graphs and a preliminary graph class learning procedure in Appendix A.
3 The AUF Problem
We treat AUF as a multi-round online decision-making problem, where in the  $t$ -th decision round, an agent makes decisions to affect the outcome if it receives an undesired prediction, serving as a warning signal, from an ML model. In real problems, variables involved in each decision round generally do not appear simultaneously; e.g., if a single decision round spans a full month, then a decision-maker may encounter new variables every day. Ignoring the variable generation order can be problematic, but modeling the decision process with refined time granularity is too demanding.
Therefore, we make a midway proposal. We identify two important time points in AUF, the time the ML prediction is made and the time just before the generation of the concerned outcome, so the variables appearing in the  $t$ -th decision round fall into three consecutive time segments separated by the two time points:  $\mathbf{X}_t$ , variables appeared before the prediction is made;  $\mathbf{Z}_t$ , variables appeared after the prediction and before the generation of the outcome; and  $\mathbf{Y}_t$ , the outcome variable. After the prediction, if the agent decides to do something to change the future outcome, the only variables it can alter are those in  $\mathbf{Z}_t$ , since  $\mathbf{X}_t$  has already happened and cannot be altered. For example, suppose that in the  $t$ -th month, a sales manager predicts the month sales  $\mathbf{Y}_t$  on the first day of that month and designs promotions in that month accordingly. Then  $\mathbf{X}_t$  can be marketing variables that appeared
before the first day and  $\mathbf{Z}_t$  can be variables in the remaining days of that month, such as the price next week, which can be altered to affect month sales.
More specifically, the decision process in the  $t$ -th round is: An agent first observes  $\mathbf{X}_t$ , with which an ML model gives a prediction  $\hat{\mathbf{Y}}_t$  for the outcome; if  $\hat{\mathbf{Y}}_t \notin S$ , where  $S$  denotes the set of desired values, the agent should perform alterations on  $\mathbf{Z}_t$  to prevent  $\mathbf{Y}_t \notin S$ ; after the alteration, the agent can observe full  $\mathbf{Z}_t$  and the outcome  $\hat{\mathbf{Y}}_t$ . We assume that the generation mechanism of the encountered variables can be described by an SRM  $\{\langle G_t, \boldsymbol{\theta}_t \rangle\}_t$ , where  $\langle G_t, \boldsymbol{\theta}_t \rangle$  characterizes the generation process in the  $t$ -th round. Fig. 3 depicts an example. For simplicity, we assume that the environment is stable, i.e.,  $G_t = G$ ,
Figure 3:Rehearsal graph  $G_{t}$  in the  $t$  -th round. The timeline shows the order in which the three sets of variables are generated and events that happened to an agent at different times in one round.
$\theta_{t} = \theta$  for all  $t$ . Note that the agent does not have access to the true SRM, instead, it can access historical data  $D = \{(\mathbf{x}^{i},\mathbf{z}^{i},\mathbf{y}^{i})\}_{i = 1}^{m}$ , from which some information about the SRM can be learned.
In real scenarios, some variables cannot be manually altered or cannot be set to certain values. We denote the set of feasible alteration values for an actionable variable  $Z_{i} \in \mathbf{Z}$  by  $\Delta(Z_{i})$ , which we assume to be a closed interval. An alteration is denoted by  $\xi_{t} = \{(Z_{a_{i}}, z_{a_{i}})\}_{i=1}^{k}$ , where  $Z_{a_{i}} \in \mathbf{Z}$  is the variable to be altered,  $z_{a_{i}} \in \Delta(Z_{a_{i}})$  is the alteration value, and  $k$  is the alteration size. Applying  $\xi_{t}$  is treated as performing a rehearsal  $Rh(\xi_{t}) \triangleq Rh(\{Z_{a_{i}} = z_{a_{i}}\}_{i=1}^{k})$  on  $G$ .
The overall AUF problem is then given  $D$  and  $\mathbf{x}_t$ ,  $t = 1, \dots, T$ , that arrives sequentially, find alterations  $\xi_t$  in each round  $t$  to successfully avoid the undesired future as many times as possible. The problem can be formulated as
$$
\max  _ {\xi_ {1: T}} \quad \mathbb {E} _ {\boldsymbol {\varepsilon} _ {1: T}} \left[ \sum_ {t = 1} ^ {T} \mathbb {I} \left(\mathbf {Y} _ {t} \in \mathcal {S} \mid \mathbf {x} _ {t}, R h \left(\xi_ {t}\right)\right) \right], \tag {2}
$$
where  $\varepsilon$  is the noise in Eq. (1) and  $\mathbb{I}(\cdot)$  is the indicator function.
If the underlying SRM is known, one can obtain  $\mathbb{P}(\mathbf{Y}_t\mid \mathbf{x}_t,Rh(\xi_t))$ , and the only issue is searching for a  $\xi_{t}$  that maximizes  $\mathbb{P}(\mathbf{Y}_t\in S\mid \mathbf{x}_t,Rh(\xi_t))$ . The main obstacle to solving AUF, however, is the uncertainty of the SRM. The uncertainty leads to an exploration-exploitation trade-off: In each round, if the agent chooses alterations that help reveal true SRMs (exploration), a more accurate environment model may be obtained and benefit decisions in future rounds, but it may fail in the current round. On the other hand, if the agent chooses a short-sighted decision that successfully avoids the upcoming undesired future (exploitation), it may learn little about SRMs, making future optimizations difficult. In addition, the effect of alterations can propagate to downstream variables through the directional edges in a rehearsal graph, thus the value of unactionable variables may be affected by altering the actionable ones. A feasible approach should take account of the uncertainty, exploration-exploitation trade-off, and propagation of effects in the online decision process and yield reasonable decisions.
Some extra knowledge can be utilized. First, the partition of the three sets of variables reflects the order in which the variables are generated. Any edge crossing the three sets should point from the previous one to the latter one, which reduces the uncertainty on  $G$ . Further, since an agent observes  $\mathbf{X}_t$  before making a decision, the generating mechanism inside  $\mathbf{X}_t$  does not affect the distribution of outcomes, as shown in Prop. 4, so we can safely pretend that  $G$  does not have edges inside  $\mathbf{X}_t$ .
Proposition 4. For any  $\langle G_1, \theta_1 \rangle$  and  $\langle G_2, \theta_2 \rangle$  on  $\mathbf{X}_t \cup \mathbf{Z}_t \cup \mathbf{Y}_t$ , if they differ only in describing the generation of  $\mathbf{X}_t$ , then for any  $\mathbf{X}_t = \mathbf{x}_t$  and alterations  $\xi$  on  $\mathbf{Z}_t$ , we have
$$
\mathbb {P} \left(\mathbf {y} _ {t} \mid \mathbf {x} _ {t}, R h (\xi); G _ {1}, \boldsymbol {\theta} _ {1}\right) = \mathbb {P} \left(\mathbf {y} _ {t} \mid \mathbf {x} _ {t}, R h (\xi); G _ {2}, \boldsymbol {\theta} _ {2}\right).
$$