Title: Avoiding Undesired Future with Minimal Cost in Non-Stationary Environments
Venue: NeurIPS 2024
Accepted: yes
Keywords: decision-making

structural rehearsal model

non-stationary environment

alteration cost
Abstract: Machine learning (ML) has achieved remarkable success in prediction tasks. In many real-world scenarios, rather than solely predicting an outcome using an ML model, the crucial concern is how to make decisions to prevent the occurrence of undesired outcomes, known as the *avoiding undesired future (AUF)* problem. To this end, a new framework called *rehearsal learning* has been proposed recently, which works effectively in stationary environments by leveraging the influence relations among variables. In real tasks, however, the environments are usually non-stationary, where the influence relations may be *dynamic*, leading to the failure of AUF by the existing method. In this paper, we introduce a novel sequential methodology that effectively updates the estimates of dynamic influence relations, which are crucial for rehearsal learning to prevent undesired outcomes in non-stationary environments. Meanwhile, we take the cost of decision actions into account and provide the formulation of AUF problem with minimal action cost under non-stationarity. We prove that in linear Gaussian cases, the problem can be transformed into the well-studied convex quadratically constrained quadratic program (QCQP). In this way, we establish the first polynomial-time rehearsal-based approach for addressing the AUF problem. Theoretical and experimental results validate the effectiveness and efficiency of our method under certain circumstances.

Avoiding Undesired Future with Minimal Cost in Non-Stationary Environments

Abstract

Machine learning (ML) has achieved remarkable success in prediction tasks. In many real-world scenarios, rather than solely predicting an outcome using an ML model, the crucial concern is how to make decisions to prevent the occurrence of undesired outcomes, known as the avoiding undesired future (AUF) problem. To this end, a new framework called rehearsal learning has been proposed recently, which works effectively in stationary environments by leveraging the influence relations among variables. In real tasks, however, the environments are usually non-stationary, where the influence relations may be dynamic , leading to the failure of AUF by the existing method. In this paper, we introduce a novel sequential methodology that effectively updates the estimates of dynamic influence relations, which are crucial for rehearsal learning to prevent undesired outcomes in nonstationary environments. Meanwhile, we take the cost of decision actions into account and provide the formulation of AUF problem with minimal action cost under non-stationarity. We prove that in linear Gaussian cases, the problem can be transformed into the well-studied convex quadratically constrained quadratic program (QCQP). In this way, we establish the first polynomial-time rehearsal-based approach for addressing the AUF problem. Theoretical and experimental results validate the effectiveness and efficiency of our method under certain circumstances.

1 Introduction

Machine learning (ML) models have found extensive application in prediction tasks [25]. However, in contrast to a sole emphasis on prediction, it is preferred in many real-world scenarios to further explore effective decisions if the predicted outcomes are undesired. For instance, imagine that a factory manager has trained an ML model on features X ( e.g. , economic indicators) to predict the outcome Y ( e.g. , monthly sales). Suppose at the beginning of a month, Y is predicted to be undesired, i.e. , the predicted sales of the month are lower than expected. In this case, the manager usually wants to take action by altering some intermediate variables Z during the month to avoid this undesired outcome happening, e.g. , modifying the discount to attract more customers. The problem of how to find effective actions in such situations is known as avoiding undesired future (AUF) [63].

It is worth noting that AUF tasks often involve limited opportunities for interaction with the decision environment [63]. For instance, in the aforementioned example, the factory manager can only adjust the selling strategy once per month. Therefore, decision-making algorithms that depend on numerous interactions, such as conventional reinforcement learning (RL) methods [4], are not well-suited for the AUF problem [40]. Additionally, fundamentally vital decisions need to be made with human judgment, and therefore, it is desired to enable human decision-makers to understand why and how some actions can change the outcome. Due to these challenges, the structural relations among variables, which contain fine-grained information and are usually interpretable, are worth being considered to make decisions [36, 40].

Causation is such a type of structural relation [36], which has been leveraged for some decision-making problems [5, 24, 42, 43]. Although causal relations can assist to some extent, identifying them is challenging and generally relies on some untestable assumptions [49, 44, 28]. Besides, causation should not be viewed as a prerequisite for decision-making problems, as humans can usually make good decisions without a thorough causal understanding [63]. Recognizing that correlation used

Correlation

Influence

Actionable

Causation

Prediction

Decision

Discovery

in prediction is inadequate for decision-making whereas causation is too luxurious to be relied on, Zhou [63] emphasized the necessity of an intermediate relation that is stronger than correlation but less demanding than causation; this relation was subsequently called influence relation [64]. The relationship among correlation , influence , and causation is illustrated in Fig. 1.

Based on the influence relations, Qin et al. [40] developed the first rehearsal learning framework, which can effectively suggest good decision actions for the AUF problem in stationary environments. In practice, however, the method by Qin et al. [40] may lose its power when different decision actions are associated with different costs or when the decision environment is non-stationary where quantitative influence relations can vary. For example, modifying the discount into different levels results in different expenses, and the quantitative influence relation between pricing and sales can change seasonally for products such as the coat. Besides, it is worth mentioning that exact solutions are intractable with any polynomial-time algorithm when considering decisions on multiple variables in the previous work [40], thus exploring more efficient approaches is necessary for the decisionsuggestion problem in the AUF problem.

To tackle these issues, in this paper, we propose the AUF-MICNS approach for the AUF problem with mi nimal c ost in n ons tationary environments. AUF-MICNS considers a multi-round decision-making process where it suggests decisions and collects feedback data during and after each decision round. Note that although multiple decision rounds are allowed, the limited number of rounds may still render RL methods ineffective [40]. In contrast to Qin et al. [40], we consider the non-stationary fact that influence relations may vary over decision rounds, and therefore, treating the round-wise collected data as i.i.d. samples for determining the influence relations is inappropriate. To this end, we present a sequential approach to maintain dynamic influence relations in non-stationary environments, and further propose an online-ensemble-based [60] sequential algorithm to deal with the unknown degree of non-stationarity. In addition, we design a cost function to quantify the costs of different decision actions. The cost function takes into account not only (a) different unit costs associated with different variables, but also (b) the distinct costs involved in altering a single variable to varying extents. Further, we expect that the suggested decisions can efficiently avoid undesired outcomes with a relatively high probability. Rather than using the sampling-based method that cannot be solved with any polynomial-time algorithm [40], we reveal that finding decision suggestions for AUF with minimal action cost can be modeled as a convex quadratically constrained quadratic program (QCQP), which is solvable in polynomial time O ( | V | 3 ) with respect to the number of variables | V | . Combining the parts above, we prove that our proposed approach can (a) accurately capture the dynamic influence relations with errors bounded by an exponentially decreasing term, and (b) efficiently suggest effective decision actions with minimal cost for the AUF problem.

Our main contributions are summarized as follows.

Wetry to tackle the AUF problem with minimal cost in non-stationary environments. Our modeling approach considers, for the first time, the decision action cost and the non-stationary fact that influence relations can vary over time in the AUF problem.

We present a novel sequential methodology to maintain dynamic influence relations. Theoretical results guarantee that the estimate error can be bounded by an exponentially decreasing term, as well as a fixed small value related to the problem difficulty.

We develop the AUF-MICNS algorithm, which is the first polynomial time rehearsal-based approach that can suggest effective decision actions for the formulated AUF problem. Our experimental results validate the effectiveness and efficiency of the method.

Organization. In Sec. 2, we review basic concepts and introduce our notation. In Sec. 3, we provide the formulation of AUF problem and provide the AUF-MICNS algorithm for solving the problem, together with theoretical guarantees. In Sec. 4, we introduce some related studies. In Sec. 5, we show the experimental results. At last, we discuss the limitations and conclude our work in Sec. 6.

X

Z

3

Y

1

Y

2

Z

2

Z

1

X

Z

Y

(a) rehearsal graph

G

X

Z

3

Y

1

Y

2

z

2

Z

1

X

Z

Y

(b)

G

Rh

(

Z

2

=

z

2

)

X

z

3

Y

1

Y

2

Z

2

Z

1

X

Z

Y

(c)

G

Rh

(

Z

3

=

z

3

)

X

z

3

Y

1

Y

2

z

2

Z

1

X

Z

Y

(d)

G

Rh

(

Z

2

=

z

2

,

Z

3

=

z

3

)

2 Preliminaries

A novel probabilistic graphical model called structural rehearsal model (SRM) is proposed by Qin et al. [40] to characterize influence relations among variables in the AUF problem. The SRM consists of a set of rehearsal graphs and corresponding structural equations {〈 G t , θ t 〉} t ( t denotes the decision round). The detailed definition of SRM is listed in Appendix B.1.

Arehearsal graph, denoted by G = ( V , E ) , models the qualitative influence relations among variables. Specifically, the vertices V denote the variable set of the AUF problem and the edges E denote the influence relation among vertices. There are two types of edges in the rehearsal graph, the directional edge X → Y means that X influences Y , and the bi-directional edge X ↔ Y means that X and Y are mutually influenced. Additionally, the corresponding structural equations of variable V j s can be parameterized by { β j,t , σ 2 j,t } | V | j =1 ⊆ θ t that:

where V j ∈ V denotes the j -th vertex in G t , PA G t j glyph[defines] { u | u → V j in G t } represents the parents of V j in G t , and the noise ε j,t follows the distribution N (0 , σ j 2 ) for all t . Note that on one hand, causal relations are not always necessary in real world decision-making problems [63]; on the other hand, causal models are sometimes insufficient for capturing the full scope of relationships between variables. For instance, the pressure and the temperature within a fixed volume of ideal gas are mutually influenced, as changes in either one of them affect the other. Such bi-directional influence relations are not well-represented by causal models, but can be naturally expressed by SRM [40].

Besides, finding suitable alterations is involved in addressing the AUF problem. An alteration ξ means a decision action that is specified by human decision-makers, denoting by a set of vertex-value pairs, e.g. , ξ = { Z 2 = z 2 } in Fig. 2(b). Meanwhile, rehearsal operation, denoted as Rh ( · ) , represents executing a certain alteration, which changes the original graph structure as illustrated in Fig. 2(b)Fig. 2(d). Specifically, rehearsal operation breaks original influence links that point into any vertices contained in ξ , and fixes the values in ξ to their associated vertices; while this operation maintains influence relations among other vertices in the associated alteration graph G . Since alteration and rehearsal operations are always considered together in the AUF problem, for simplicity, we will use the term alteration exclusively throughout the paper, unless otherwise specified.

3 The proposed approach

This section is dedicated to addressing the aforementioned AUF problem. In Sec. 3.1, we provide the probelm formulation. In Sec. 3.2, we propose the AUF-MICNS method for AUF. Later in Sec. 3.3, theoretical results are provided to ensure the effectiveness of our approach.

3.1 Formulation

This paper focuses on suggesting decisions to avoid undesired futures when an undesired outcome is predicted by an ML model. Since effective prediction models are widely applied in various domains [25] and work well even in non-stationary environments [62], we consider the case that the predictive ML model is always available and are not concerned about how to train it.

We formulate the AUF problem with minimal cost in non-stationary environments as a multi-round online decision-making process, where the decision-maker should perform round-wise alterations to

avoid undesired outcomes. In each decision round, there are two essential time points, the time that the ML prediction is made, and the time just before the generation of the concerned outcome Y . As separated by the two time points, the variables fall into three consecutive time segments: X , Z , and Y as illustrated in Fig. 2(a). In t -th round, a decision-maker first observes variables X t = x t , and an ML model provides a prediction ˆ Y t as the outcome subsequently. Denote the desired region of Y t as S , if ˆ Y t glyph[negationslash]∈ S , the decision-maker would perform alterations on Z t (only once) based on the whole historical data, and the true Y t will occur after the alteration.

It is worth mentioning that in decision round t , two crucial concerns for making decisions are (a) how to perform effective alterations that make Y t ∈ S with high probability; and (b) how to minimize the alteration cost as much as possible. For the former, we seek to ensure that P ( Y t ∈ S | θ t , x t , Rh ( ξ t )) ≥ τ ; where ξ t is the selected alteration, and τ is the expected probability that the alteration can successfully avoid the undesired outcome. For the latter, we generalize a continuous cost function from the discrete cost measure [61] to quantify the cost for different alterations. The cost function is defined as the sum of the respective costs associated with each altered variable, and follows the increasing marginal cost property (see Appendix B.2 for details) in economics [32], properly measuring the alteration cost. Given the expectation that the future outcome will likely fall into S after performing the alteration, the decision maker consistently prefers the alteration with the least alteration cost. Thus, the AUF problem can be ideally formulated as follows in practice:

where w i > 0 represents the cost coefficient for each intermediate variable Z i ∈ Z , Z ξ t i represents the values of Z i after alteration ξ t while Z 0 i is the datum point that associates with the minimum alteration cost relatively. Besides, ϑ glyph[star] t = arg min ϑ E ε ‖ ϑ -θ t ‖ is the ideal estimation of θ t since θ t is not available in practice. Note that w i s and Z 0 i s are user-specified because the cost of the same decision alteration can vary for different decision-makers.

In the following, we focus on a basic but essential class of the AUF problem, where the structural equations f t in Eq. (1) are linear but dynamic, the desired set S in Eq. (2) is a convex polytope, and the rehearsal graph G is known and fixed ( i.e. , G t glyph[defines] G ) for a convenient illustration. Let d ∈ R s , M ∈ R s ×| Y | , β j ∈ R | PA G j | × 1 , dynamic structural equations and the desired region can be formulated as:

Note that ϑ glyph[star] t in Eq. (2) is usually not available as discussed later in Sec. 3.2.1, thus finding surrogate estimations ˆ θ t with bounded error E ε ‖ ˆ θ t -θ t ‖ is necessary. Noticing that the noise distributions for ε j,t s in Eq. (3) do not change over time, the variances of ε j,t s can be estimated by various Bayesian learning methods [6]. Thus, we mainly focus on estimating influence parameters β j,t s in what follows, and we use θ t to represent { β j,t } s only unless otherwise specified. Another challenge lies in how to efficiently solve the optimization Eq. (2) under the probabilistic constraint, since the sampling method used in the previous work has been proven to be time-consuming [40]. In Sec. 3.2, we propose the AUF-MICNS algorithm to tackle the aforementioned two challenges. AUF-MICNS can maintain the dynamic influence relations accurately and suggest alterations to avoid undesired outcomes effectively, and the performance can be guaranteed by theoretical results in Sec. 3.3.

3.2 Our proposed AUF-MICNS algorithm

In this subsection, we propose the AUF-MICNS algorithm to address the AUF problem as formulated in Eq. (2). AUF-MICNS consists of two components, influence maintenance and alteration suggestion. We first introduce the components respectively, then we discuss how to combine them together.

3.2.1 Dynamic influence maintenance

As illustrated in Eq. (2), the precondition for making effective decisions in the AUF problem lies in the accurately estimated θ t s. Specifically, to accurately estimate the parameter vector β j,t ∈ θ t in t -th decision round, we would like to minimize the expected squared loss E ε j [ V j -β glyph[latticetop] j,t PA j ] 2 ,

where V j is the j -th vertex in rehearsal graph, and PA j denotes the parents of V j . Ideally, according to the law of large numbers (LLN), if we could obtain sufficient i.i.d. samples in time t , we can estimate β j,t by minimizing the empirical error glyph[lscript] j,t ( β j,t ) as a substitute:

where n is the number of samples that we can obtain in round t , and the superscript k means that it is the k -th sample. It can be guaranteed in lemma C.3 that this estimation converges to the true value as n increases. However, in real-world cases, we can only obtain a single sample after each decision round, so it is inappropriate to calculate the parameters as above, since the solution to Eq. (4) is far from accurate in this data-limited situation, where the surrogate loss is:

Besides, mixing the round-wise selected { V j,t , PA j,t } T t =1 to estimate parameter β j,t as the classic empirical risk minimization (ERM) [34] methods do is inappropriate as well, because data selected in different rounds possibly obeys different distributions in non-stationary environments. Fortunately, in the field of online learning [52, 14, 17], there exist many types of algorithms to estimate parameters sequentially with limited samples. Online gradient descent (OGD) [65, 47] is a typical class of online learning algorithms, which takes advantage of the gradient descent idea to handle the round-wise selected data. Once the selection of the step size (learning rate) in OGD is tailored to the varying speed of the environment, i.e. , employing a larger step size for rapid changes and a smaller one for gradual changes, parameters can be effectively updated with limited data after each round. We customize OGD for estimating quantitative influence relations ˆ β j,t ∈ θ t sequentially in Algorithm 1.

By using the OGD-based approach in Algorithm 1, we can obtain a sequence of estimates { ˆ β j, 1 , . . . , ˆ β j,T } for j ∈ [ | V | ] . Roughly speaking, the quality of the estimates heavily depends on the choice of step size η . If we have full prior knowledge of the non-stationarity degree, such as the changing speed of the influence relations, we can pre-determine an optimal step size η glyph[star] to achieve favorable estimates. However, in practical sce-

Algorithm 1 OGD-based estimator for ˆ β j

Input: The step size η

1: Initialize ˆ β η j, 0 with any point in domain B

3: Receive (PA j,t , V j,t ) ; Continue if V j,t ∈ ξ t

2: for t = 1 to T do

4: Estimate gradient ˆ g j,t = ( PA glyph[latticetop] j,t ˆ β η j,t -V j,t ) PA j,t

Output: estimated { ˆ β η j,t } T t =1

narios, η glyph[star] is not available, and the random choice of the step size leads to unstable estimators. Since bad estimators of the influence relations will affect the accuracy of the estimated distribution P ( Y t ∈ S | ˆ θ t , x t , Rh ( ξ t )) , which will further affect the effectiveness of the suggested decisions for the AUF problem, thus the choice of the step size needs to be carefully considered.

To avoid this risk, we turn to use online-ensemble [60] based methods. Online ensemble is a type of algorithm that combines the idea of ensemble and sequential updating, maintaining multiple base learners and ensembling them together. Specifically, as illustrated in Algorithm 2, we maintain N j base estimators with a weight vector w . After each round, all experts update their estimates by the collected data, and the weight vector w will be updated as well by different losses related to different experts. By ensembling all experts with the weight w , we can obtain the final estimator.

Algorithm 2 Online-ensemble-based estimator for ˆ β j

Input: base estimators' number N j , weight parameter α

1: Set a set of learning rates H j = { η i | i = 1 , . . . , N j }

3: Activate estimators E η s for all η ∈ H j by OGD with η

2: Initialize weight vector w η t = 1 N j for each η ∈ H j

4: for t = 1 to T do

5: Receive β η j,t from each E η

6: Output ˆ β j,t as ∑ µ ∈H j w µ t β µ j,t

7: Update weights as

9: Receive (PA j,t , V j,t ) and send to each E η

Output: estimated { ˆ β j,t } T t =1

3.2.2 Efficient alteration suggestion

In t -th decision round, since we can obtain the estimation ˆ θ t for true θ t by Algorithm 2, then we substitute ϑ glyph[star] with ˆ θ t in Eq. (2). To suggest alterations efficiently and effectively, another crucial aspect is finding ways to delineate the feasible domain of the probabilistic constraint in Eq. (2). We first present a supporting result in Lemma 3.1 as follows, which is needed in our method to find the alteration that can make Y t ∈ S with an expected probability τ .

Lemma 3.1 (Qin et al., 2023) . Given x t , θ t , it holds that:

where A , B , C are constant matrices of appropriate shapes based on θ t , ε t = [ ε 1 ,t , . . . , ε | V | ,t ] ∼ N ( 0 , Σ ) , and z ξ t are intermediate variables with alteration ξ .

Recall from Eq. (2) that we want to find alterations that satisfy P ( Y t ∈ P | θ t , x t , Rh ( ξ t )) ≥ τ . Recognizing that solving optimization with probabilistic constraint is generally intractable and the previous sampling method is time-consuming, we attempt to construct a surrogate deterministic constraint that can be handled efficiently to replace the probabilistic constraint. Fortunately, this idea is feasible because Lemma 3.1 shows that once the alteration is selected, the randomness of Y t only arises from ε t , since x t has been observed and A , B , C are constant matrices given θ t . Thus, the probability density function (PDF) of Y t is available and we can directly analyze the PDF to find probability regions as in Prop. 3.2, which aids in constructing the surrogate deterministic constraint.

Proposition 3.2. The following probability region P satisfies P ( Y t ∈ P | θ t , x t , Rh ( ξ t )) = τ :

where µ y t = Ax t + Bz ξ t , u is an arbitrary point in the unit sphere in R | Y t | , and χ -1 ( · ) denotes the quantile function of the χ 2 distribution with degrees of freedom λ = | Y t | .

The proof of Prop. 3.2 is provided in Appendix C.4. We raise the power of 1 / 2 to the matrix because it is always positive semi-definite since χ -1 ( · ) ≥ 0 and CΣC glyph[latticetop] glyph[followsequal] 0 ( Σ is the covariance

(a) Samples from F y t

matrix). In practical scenarios where Σ is not available, the estimation ˆ Σ can be used as replacements. Let F y t denote the cumulative distribution function (CDF) of Y t , Fig. 3 illustrates a 2-dimensional example of samples from F y t , together with the estimation of the associated region ˆ P . It illustrates that ˆ P can properly draw the probabilistic region of F y t with expected probability τ .

Moreover, recognizing that the desired region S defined in Eq. (3) is a convex polytope, i.e. , S = { y ∈ R | Y | | My ≤ d } , we utilize the defined probability region P in Prop. 3.2, and use a deterministic constraint to replace the original probabilistic constraint in Eq. (2). In this case, the alteration-suggestion method can be formulated as follows:

where P = ( χ -1 ( τ ) CΣC glyph[latticetop] ) 1 2 , ‖ · ‖ 2 , row means an operator that takes 2-norm for each row of the matrix thus outputs a row-dimensional vector. The objective to be minimized is the vector representation of the cost function as explained in Eq. (2), where W = diag( w 1 , . . . , w | z ξ t t | ) is positive definite since the cost is non-negative, i.e. , w i > 0 for ∀ i .

Note that W is positive definite and the constraint is linearly associated with z ξ t t , thus Eq. (6) is a convex QCQP, which can be cast as a second-order cone program [35, 30]. Specifically, it can be solved in polynomial time O ( | Z ξ t t | 3 · L ) by interior-point method [2], where L is the iteration rounds

for solving the QCQP and is not associated with | Z ξ t t | . Meanwhile, constructing matrices A , B , C needs an element-wise traverse thus O ( | V | 2 ) time and constructing the probability region P as in Prop. 3.2 runs under O ( | V || Y | 2 ) because of the matrix multiplication, so the whole running time of alteration suggestion is O ( | V | 3 ) as | Z ξ t t | = Θ( | V | ) , | Y | = Θ( | V | ) and L is a constant.

3.2.3 AUF-MICNS

By combining the influence maintenance step and the alteration suggestion step, our proposed approach for addressing the AUF problem with minimal cost in non-stationary environments is formulated in Algorithm 3, which attempts to avoid undesired outcomes in each decision round. Specifically, in t -th round, the algorithm first receive X t = x t , then if the predicted outcome ˆ Y t glyph[negationslash]∈ S , the

Algorithm 3 AUF-MICNS

Input: sequential coming data { x t , z t , y t } T t =1

1: Initialize { ˆ β j, 0 } | V | j =1 for Algorithm 2

2: for t = 1 to T do

3: Select alteration ξ t by solving Eq. (6)

4: Receive y t and sent { x t , z ξ t t , y t } to Algorithm 2

5: Update { ˆ β j,t } | V | j =1 by Algorithm 2

Output: suggested alterations { ξ t } T t =1

algorithm performs the suggested alteration ξ t on Z t by solving Eq. (6). Subsequently, true Y t = y t occurs, and the algorithm collects { x t , z ξ t t , y t } to update influence relations by Algorithm 2. By using this algorithm, one can tackle the formulated AUF problem with the suggested alterations.

3.3 Theoretical results

In this subsection, we present the theoretical analysis of our proposed method. All proofs are given in Appendix C. First, we can determine the dynamic influence relations in non-stationary environments with theoretical guarantees. Specifically, by using Algorithm 1 to estimate β j,t ∈ θ t , the error gap between the estimate and the true parameter value ( E ε j ‖ ˆ β j,t -β j,t ‖ 2 ) is proved to be bounded by an exponentially decreasing term, as well as a fixed value related to the choice of step size and the inherent problem difficulty. It reveals that the performance of Algorithm 1 depends on the choice of step size η heavily, and is detailed in Thm. 3.3, where { glyph[lscript] j,t ( · ) } T t =1 s are defined in Eq. (4).

Theorem 3.3. Let β j,t ( j ∈ [ | V | ] ) denote the true parameter value of β j in time t , and choose η j ∈ (0 , 1 / 2 L j ] as the step size used in Algorithm 1, then it can be bounded that:

where µ j and L j are the minimal and maximal eigenvalues of { glyph[lscript] j,t ( · ) } T t =1 's Hessian matrices, σ 2 upper-bounds the variance of ˆ g j,t , ∆ j ≥ max t ‖ β j,t +1 -β j,t ‖ upper-bounds the varying speed of the environment, and m is the longest continuously altered rounds of V j , for most of the V j s, m = 1 .

glyph[negationslash]

In Thm. 3.3, it holds that µ j η j ∈ (0 , 1 / 2] , which derives that (1 -µ j η j ) ∈ [1 / 2 , 1) . This shows that as time progresses, the OGD estimator ˆ β j,t will gradually converge towards the true value β j,t in the expected sense. Specifically, (a) the convergence speed depends on the choice of the initial point ˆ β j, 0 , and is limited by the inherent difficulty of the problem implied in µ j /L j ; and (b) the convergence result will suffer a δ j gap from the true value, depending on the varying speed of the environment ( ∆ j ) and the choice of step size η j . If η j is appropriately chosen, the gap δ j will be small, e.g. , if all hyperparameters are available and ∆ j = 0 , choosing η glyph[star] j = min { 1 / 2 L j , 3 √ 2( m ∆ j √ µ j σ ) 2 3 } can achieve the smallest δ j . Meanwhile, the hyperparameter m appears in the bound according to the properties of alterations shown in Fig. 2. If V j is continuously altered in m rounds, all incoming arrows of V j will be removed and the parameters associated with the arrows will not be updated in those rounds. Because only a part of vertices in Z t might be altered in any rounds, m = 1 holds for most of the V j s.

In practice, we do not know exact parameters such as ∆ j , thus η glyph[star] is not pre-available. Recognizing that the random choice of η leads to unstable estimations, we turn to use Algorithm 2 to estimate ˆ β j,t s. Though we do not know η glyph[star] as well, by using the online-ensemble-based Algorithm 2, we can get more stable estimations as guaranteed by the following regret bound in Prop. 3.4.

Proposition 3.4. Assume { ˆ glyph[lscript] j,t ( · ) } T t =1 s are bounded for ∀ β i ∈ B and t ∈ [ T ] ; then for any η ∈ H j , estimations ˆ β j,t s from Algorithm 2 satisfies that

by choosing α = √ ln N j /T in Algorithm 2, where N j is the number of base-learners, β η j,t is the estimation from any expert η in expert set H j in Algorithm 2.

Prop. 3.4 shows that the cumulative loss of the estimation obtained by Algorithm 2 is comparable with the best expert in H j , thus by Thm. 3.3, though ˆ θ t may be far from θ t at the first few rounds, it will converge towards θ t . Meanwhile, in some certain cases, the best step size η glyph[star] j in Thm. 3.3 can be included in H j with N j = O (log T ) [60]. Due to these, Algorithm 2 provides more stable estimates for the influence relations, which can further aid in making decisions to address the formulated AUF problem. As to the decision-making processes, AUF-MICNS can suggest alterations with theoretical guarantees. Specifically, the suggested alterations are guaranteed by Thm. 3.5.

Theorem 3.5. By using the suggested alterations ξ t from Eq. (6), it can be guaranteed that:

Thm. 3.5 illustrates that the suggested alterations ξ t by the AUF-MICNS algorithm can effectively avoid the undesired outcomes as the formulated AUF problem expects, under the distribution conditioned on estimation ˆ θ t rather than true θ t . Note that it is guaranteed in Thm. 3.3 and Prop. 3.4 that ˆ θ t is not far from θ t . Meanwhile, we provide the experimental analysis in Sec. 5 to show that P ( Y t ∈ S | θ t , x t , Rh ( ξ t )) ≥ τ holds practically. Besides, this method achieves a super-exponential improvement over the time complexity of the previous method [40] as discussed in Sec. 3.2.2.

4 Related work

RL Methods. RL approaches have demonstrated success in numerous domains [51], particularly in game-playing [33] and autonomous control [21]. However, the Markov Decision Process (MDP) formalism in RL abstracts decision-making processes into states, actions, and rewards, potentially overlooking useful fine-grained structural information. While hybrid online and offline RL methods have been introduced [48, 37], they overlook fine-grained structural information as well and require large offline datasets in practice. Moreover, it is worth noting that a fundamental limitation of applying RL to the AUF problem is that RL methods require a substantial number of interactions, which may be too luxurious or simply not tolerated in many real-world applications.

Causality. Identifying causal systems from observational and interventional data has been extensively studied [44, 50, 7, 58, 9], but these methods typically do not actively select interventions. Furthermore, significant research has focused on identifying causal structures or effects in interactive environments [18, 22, 53, 55, 39, 54, 57, 41], which predominantly aim to identify causal structures or effects. To incorporate additional utilities for decision-making, causal bandits and causal RL methods have been proposed to determine where to intervene [5, 24, 46, 26, 59, 12, 31]. The aforementioned methods generally rely on causal modeling, which may be luxurious or restrictive in some real-world decision-making cases [63, 40]. As identifying causations rely on some strong restrictions or assumptions, it is possible that we cannot find a feasible alteration. Conversely, correlation, which underpins most ML models, falls short of providing a solid foundation for making decisions. As a middle ground between causation and correlation, the influence relations form a more practical basis for decision-making [63]. Building on this concept, we employ the SRM developed by Qin et al. [40], which can adapt to dynamically evolving decision systems. In particular, our approach incorporates the context x t into decision-making, and can sugggest decision alterations effectively even when the parameters of the system are non-stationary and the actions come with varying associated costs.

Other related topics. Our approach builds upon several classic ML techniques. The action cost measure employed in our method generalizes principles from cost-sensitive learning [13, 61]. Additionally, we adapt the online ensemble methodology [60] to update the SRM parameters. Online emsemble framework has been used in several areas, such as online convex optimization [56], online label shift [38], and reinforcement learning [27]. Further exploration and advancements in these techniques hold the potential to enhance our approach as well.

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Success Frequency

Success frequency and relative alteration cost

AUF-MICNS

QWZ23

DDPG

PPO

SAC

Desired

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

t

t

2

of estimations

MSE of different estimations

Online Ensemble

OGD with step size 0.002

OGD with step size 0.005

OGD with step size 0.025

1.4

1.5

1.6

1.7

1.8

1.9

Value of

(2)

NCT

and

(2)

NCT

True influence relation and the estimation

True Value

(2)

NCT

,

t

Estimation

(2)

NCT

,

t

0.0

0.2

0.4

0.6

0.8

1.0

Relative Alteration Cost

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Success Frequency

AUF-MICNS

QWZ23

DDPG

PPO

SAC

Desired

20

40

60

80

100

Round

0.0

0.2

0.4

0.6

0.8

t

t

2

of estimations

Online Ensemble

OGD with step size 1.000

OGD with step size 2.500

OGD with step size 5.000

20

40

60

80

100

Round

0.6

0.7

0.8

0.9

1.0

1.1

Value of

(1)

CO2

and

(1)

CO2

True Value

(1)

CO2,

t

Estimation

(1)

CO2,

t

5 Experiments

We evaluate the proposed AUF-MICNS algorithm on two datasets and focus on four aspects including (a) success frequency for Y t ∈ S ; (b) average alteration cost; (c) average executing time; and (d) mean square error of the estimated parameters ˆ θ t . For each dataset, we repeat experiments with 100 rounds 20 times. The observational dataset size is set to 10. We compare our proposed method with previous rehearsal learning approach (QWZ23 [40]), and we also compare with several classic RL methods including DDPG [29], PPO [45], and SAC [15]. Experimental details and additional experiments are listed in Appendix D.

Market-Managing Data. We abstract a dynamic SRM from a market-managing scenario, where a manager of the market needs to make decisions to promote the total profit (TPF) and the number of customers (NCT). We consider variables that may affect TPF and NCT, the pricing for the product (P), the pricing of the competitor market (E), the cost of raw materials (C), etc. We assume the manager can alter two variables, P and C. There exist mutually influenced variables in the scenario: If P is set to be small, then E will also be small to stay competitive on price; and vice versa. The sizes of X t , Z t , and Y t are 2, 4, and 2 respectively. The parameters of the dynamic structural equations are manually set according to the domain knowledge. For example, parameters associated with variable C vary over time with a periodic term, i.e. , β C,t = ¯ β C (1 + a sin( wt )) . The feasible alteration values are [ -3 , 3] for centralized P and C, associated with cost coefficients 1 . 0 and 2 . 0 respectively since altering P is easier than C. We want to maintain high TPF and high NCT at the same time, so the desired region S is set to be S = { TPF ≥ s 1 , NCT ≥ s 2 , TPF + NCT ≥ s 3 } , and more than 80% of the original data falls outside this range, as shown in Fig. 7(a).

Bermuda Data. This is an ecology dataset that records environment variables in Bermuda [10], and the generation order of variables is available [3]. The sizes of X t , Z t , and Y t are 3, 7, and 1. The structural equations are obtained by fitting linear models on normalized data [40], and we manually add the varying trend, e.g. , considering the annual increase in CO 2 emissions, we posit that there is an increasing trend in the influence relation between temperature and CO 2 concentration, i.e. , β CO 2 ,t = ¯ β CO 2 ,t · (1 -a e -wt ) . We assume that 5 variables in Z t are actionable [1] and the feasible alteration values are [ -1 , 1] for each of them with different cost coefficients. The concerned outcome Y t represents the net coral ecosystem calcification (NEC) in Bermuda. To make the coral reef ecosystem healthy, a relatively large NEC is preferred, so the desired region S is set to be S = { 0 . 5 ≤ NEC ≤ 2 } , and more than 75% of the original data falls outside it, as shown in Fig. 7(b).

Fig. 4 shows the full experimental results. The desired probability is set to τ = 0 . 7 . Two rows of figures denote the results of the Market data and the Bermuda data respectively. Specifically, the first column of Fig. 4 shows that (a) the success frequency of making the concerned outcome Y t falls into the desired region S ; and (b) relative alteration cost among methods, i.e. , the normalized cost among different methods. The proposed AUF-MICNS algorithm outperforms other competitors in both aspects. If we increase the interaction rounds T , RL methods can achieve satisfying performance, e.g. , DDPG achieves 0.6955 average success frequency when T = 4000 on the Market data.

The second column of Fig. 4 shows the mean-square error of estimates for β NCT in Market data and estimates for β CO 2 in Bermuda data respectively. The exponential convergence speed and the convergence error ( δ j ) in Thm. 3.3 are illustrated in these two figures. Specifically, for Marketmanaging data, the online-ensemble-based sequential method can achieve a similar convergence speed like OGD with η = 0 . 025 does when T ≤ 20 , and it results in a smaller δ j as illustrated by the enlarged part ( T ≥ 40 ). For Bermuda data, OGD with η = 2 . 5 outperforms other step sizes, since η = 1 . 0 converge slowly, and η = 5 . 0 suffers a notably bigger gap δ j . By using the online-ensemblebased sequential method, we can obtain a comparable performance with the best step size η = 2 . 5 . These results illustrate that our proposed method mitigates the risk substantially, as the inappropriate choice of η will affect the practicality of the estimation and η glyph[star] is not available practically. Meanwhile, the third column of Fig. 4 shows true parameter values and the estimates by the online-ensemble-based method. The superscript (1) / (2) means the first/second dimension of the associated parameter vector. It shows that the error gap between ˆ β j,t and β j,t converges to a small value rapidly, which guarantees accurate estimates of the quantitative influence relations in possibly non-stationary environments.

At last, the average whole-executing time of the 20-times experiments is recorded in Table 1. We mainly focus on the comparison between AUF-MICNS and QWZ23 since both of them maintain influence relations other than purely suggesting deci-

sions. It shows that our proposed method is more time-efficient than QWZ23.

6 Conclusion

Practically, different decision actions might correspond to different costs, and the influence relations might vary over time in non-stationary environments. In this paper, we try to tackle the AUF problem considering the aforementioned aspects. Specifically, we propose the AUF-MICNS algorithm that can capture the dynamic influence relations in non-stationary environments and suggest actions based on the influence relations. This method can suggest decision actions under polynomial time. Meanwhile, theoretical results show that the suggested actions can effectively avoid the undesired outcomes with probability larger than τ , and the suggested actions get more accurate as time progresses. Experimental results validate the effectiveness and efficiency of our proposed method.

Our approach primarily focuses on scenarios where the influence relations among variables are linear and is currently not applicable to non-linear cases. Additionally, selecting the hyperparameter τ in existing frameworks remains a significant challenge, as an inappropriate τ may lead to the failure of solving Eq. (6). To address these limitations, we plan to develop methods that handle non-linear cases and reduce sensitivity to the hyperparameter τ in future work.

References

Virginia Aglietti, Xiaoyu Lu, Andrei Paleyes, and Javier González. Causal Bayesian optimization. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics , pages 3155-3164, 2020.

Erling D. Andersen, Cornelis Roos, and Tam'as Terlaky. On implementing a primal-dual interior-point method for conic quadratic optimization. Math. Program. , 95(2):249-277, 2003.

Andreas Andersson and Nicholas Bates. In situ measurements used for coral and reef-scale calcification structural equation modeling including environmental and chemical measurements, and coral calcification rates in bermuda from 2010 to 2012 (BEACON project), 2018. http://lod.bco-dmo.org/id/dataset/720788.

Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine , 34(6):26-38, 2017.

Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: A causal approach. In Advances in Neural Information Processing Systems 28 , pages 1342-1350, 2015.

C Bishop. Pattern Recognition and Machine Learning . Springer, 2006.

Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. In Advances in Neural Information Processing Systems 36 , page to appear, 2023.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

Wei Chen, Zhiyi Huang, Ruichu Cai, Zhifeng Hao, and Kun Zhang. Identification of causal structure with latent variables based on higher order cumulants. In The 38th AAAI Conference on Artificial Intelligence , pages 20353-20361, 2024.

Travis A Courtney, Mario Lebrato, Nicholas R Bates, Andrew Collins, Samantha J De Putron, Rebecca Garley, Rod Johnson, Juan-Carlos Molinero, Timothy J Noyes, Christopher L Sabine, et al. Environmental controls on modern scleractinian coral and reef-scale calcification. Science Advances , 3(11):e1701356, 2017.

Joshua Cutler, Dmitriy Drusvyatskiy, and Zaïd Harchaoui. Stochastic optimization under distributional drift. Journal of Machine Learning Research , 24(147):1-56, 2023.

Arnoud A. W. M. de Kroon, Danielle Belgrave, and Joris M. Mooij. Causal discovery for causal bandits utilizing separating sets. CoRR , abs/2009.07916, 2020.

Charles Elkan. The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence , pages 973-978, 2001.

Yoav Freund and Robert E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences , 55(1):119-139, 1997.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning , pages 1861-1870, 2018.

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: Data mining, inference, and prediction . Springer, 2009.

Elad Hazan. Introduction to online convex optimization. Foundations and Trends in Optimization , 2(3-4): 157-325, 2016.

Yangbo He and Zhi Geng. Active learning of causal networks with intervention experiments and optimal designs. Journal of Machine Learning Research , 9(Nov):2523-2547, 2008.

Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association , 58(301):13-30, 1963.

Thomas Hofmann, Bernhard Schölkopf, and Alexander J Smola. Kernel methods in machine learning. Annals of Statistics , 36(3):1171-1220, 2008.

Bahare Kiumarsi, Kyriakos G. Vamvoudakis, Hamidreza Modares, and Frank L. Lewis. Optimal and autonomous control using reinforcement learning: A survey. IEEE Transactions on Neural Networks and Learning Systems , 29(6):2042-2062, 2018.

Murat Kocaoglu, Alex Dimakis, and Sriram Vishwanath. Cost-optimal learning of causal graphs. In Proceedings of the 34th International Conference on Machine Learning , pages 1875-1884, 2017.

Kira Lancker and Martin F. Quaas. Increasing marginal costs and the efficiency of differentiated feed-in tariffs. Energy Economics , 83:104-118, 2019.

Finnian Lattimore, Tor Lattimore, and Mark D. Reid. Causal bandits: Learning good interventions via causal inference. In Advances in Neural Information Processing Systems 29 , pages 1181-1189, 2016.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436-444, 2015.

Sanghack Lee and Elias Bareinboim. Characterizing optimal mixed policies: Where to intervene and what to observe. In Advances in Neural Information Processing Systems 33 , pages 8565-8576, 2020.

Long-Fei Li, Peng Zhao, and Zhi-Hua Zhou. Dynamic regret of adversarial linear mixture MDPs. In Advances in Neural Information Processing Systems 36 , pages 60685-60711, 2023.

Zihao Li, Hui Lan, Vasilis Syrgkanis, Mengdi Wang, and Masatoshi Uehara. Regularized deepiv with model selection, 2024. URL https://arxiv.org/abs/2403.04236 .

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceedings of the 4th International Conference on Learning Representations , 2016.

Miguel Sousa Lobo, Lieven Vandenberghe, Stephen Boyd, and Hervé Lebret. Applications of second-order cone programming. Linear Algebra and its Applications , 284(1-3):193-228, 1998.

Yangyi Lu, Amirhossein Meisami, and Ambuj Tewari. Causal bandits with unknown graph structure. In Advances in Neural Information Processing Systems 34 , pages 24817-24828, 2021.

N Gregory Mankiw. Principles of Economics . Cengage Learning, 9th edition, 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR , abs/1312.5602, 2013.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . MIT Press, 2nd edition, 2018.

Yurii Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming . SIAM, 1994.

Judea Pearl. Causality: Models, Reasoning and Inference . Cambridge University Press, 2nd edition, 2009.

Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline metareinforcement learning with online self-supervision. In Proceedings of the 39th International Conference on Machine Learning , pages 17811-17829, 2022.

Yu-Yang Qian, Yong Bai, Zhen-Yu Zhang, Peng Zhao, and Zhi-Hua Zhou. Handling new class in online label shift. In Proceedings of the 23rd IEEE International Conference on Data Mining , pages 1283-1288, 2023.

Tian Qin, Tian-Zuo Wang, and Zhi-Hua Zhou. Budgeted heterogeneous treatment effect estimation. In Proceedings of the 38th International Conference on Machine Learning , pages 8693-8702, 2021.

Tian Qin, Tian-Zuo Wang, and Zhi-Hua Zhou. Rehearsal learning for avoiding undesired future. In Advances in Neural Information Processing Systems 36 , pages 80517-80542, 2023.

Tian Qin, Long-Fei Li, Tian-Zuo Wang, and Zhi-Hua Zhou. Tracking treatment effect heterogeneity in evolving environments. Machine Learning , 113(6):3653-3673, 2024.

Kangrui Ruan and Xuan Di. Learning human driving behaviors with sequential causal imitation learning. In Proceedings of the 36th AAAI Conference on Artificial Intelligence , pages 4583-4592, 2022.

Kangrui Ruan, Junzhe Zhang, Xuan Di, and Elias Bareinboim. Causal imitation learning via inverse reinforcement learning. In Proceedings of the 11th International Conference on Learning Representations , 2022.

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5): 612-634, 2021.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017.

Rajat Sen, Karthikeyan Shanmugam, Alexandros G Dimakis, and Sanjay Shakkottai. Identifying best interventions through online importance sampling. In Proceedings of the 34th International Conference on Machine Learning , pages 3057-3066, 2017.

Shai Shalev-Shwartz, Yoram Singer, and Nathan Srebro. Pegasos: Primal estimated sub-gradient solver for SVM. In Proceedings of the 24th International Conference on Machine Learning , pages 807-814, 2007.

Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make RL efficient. In Proceedings of the 11th International Conference on Learning Representations , 2022.

Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, Prediction, and Search . MIT Press, 2000.

Chandler Squires, Anna Seigal, Salil S Bhate, and Caroline Uhler. Linear causal disentanglement via interventions. In Proceedings of the 40th International Conference on Machine Learning , pages 3254032560, 2023.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT Press, 2018.

V. G. Vovk. Aggregating strategies. In Proceedings of the 3rd Annual Workshop on Computational Learning Theory , pages 371-386, 1990.

Tian-Zuo Wang, Xi-Zhu Wu, Sheng-Jun Huang, and Zhi-Hua Zhou. Cost-effectively identifying causal effects when only response variable is observable. In Proceedings of the 37th International Conference on Machine Learning , pages 10060-10069, 2020.

Tian-Zuo Wang, Tian Qin, and Zhi-Hua Zhou. Estimating possible causal effects with latent variables via adjustment. In Proceedings of the 40th International Conference on Machine Learning , pages 36308-36335, 2023.

Tian-Zuo Wang, Tian Qin, and Zhi-Hua Zhou. Sound and complete causal identification with latent variables given local background knowledge. Artificial Intelligence , 322:103964, 2023.

Yu-Hu Yan, Peng Zhao, and Zhi-Hua Zhou. Universal online learning with gradual variations: A multi-layer online ensemble approach. In Advances in Neural Information Processing Systems 36 , pages 37682-37715, 2023.

Jiaqi Zhang, Louis Cammarata, Chandler Squires, Themistoklis P Sapsis, and Caroline Uhler. Active learning for optimal intervention design in causal models. Nature Machine Intelligence , pages 1-10, 2023.

Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. In Advances in Neural Information Processing Systems 36 , page to appear, 2023.

Junzhe Zhang and Elias Bareinboim. Near-optimal reinforcement learning in dynamic treatment regimes. In Advances in Neural Information Processing Systems 32 , pages 13401-13411, 2019.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Adaptivity and non-stationarity: Problemdependent dynamic regret for online convex optimization. Journal of Machine Learning Research , 25(98): 1-52, 2024.

Zhi-Hua Zhou. Cost-sensitive learning. In International Conference on Modeling Decisions for Artificial Intelligence , pages 17-18, 2011.

Zhi-Hua Zhou. Open-environment machine learning. National Science Review , 9(8):nwac123, 2022.

Zhi-Hua Zhou. Rehearsal: Learning from prediction to decision. Frontiers of Computer Science , 16(4): 1-3, 2022.

Zhi-Hua Zhou. Rehearsal: Learning from prediction to decision. Keynote at the CCF Conference on AI, Urumqi, China, 2023.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning , pages 928-936, 2003.

A Discussion

A.1 Discussion on the AUF-MICNS algorithm

Two crucial points should be discussed in this approach, i.e. , (a) dynamic influence modeling; and (b) the defined cost function for alterations. For the former, we consider a linear case with additive noise, and for more complicated scenarios than linear ones, one can modify the structural equations in Eq. (3) by using methods such as the kernel method [20]. Note that Theorem 3.3 always holds as long as the new loss functions glyph[lscript] j,t ( · ) s are still µ -strongly convex and L -smooth in those cases, similar to Eq. (4). For the latter, we use a quadratic cost function in this paper. Since the alteration cost is user-specified, in certain problems, if the quadratic cost is not appropriate, one can replace it with any other convex functions under the increasing marginal cost (IMC) property [32]. In those cases, the alteration selection method is a convex optimization as well, similar to Eq. (6). It is worth noting that our work mainly focuses on cases where samples are rare, thus only user-specified hyperparameters { w j , z 0 j } j ∈ [ | V | ] can be relied upon. If we can obtain a sufficient number of 〈 alteration, cost 〉 pairs, hyperparameters { w j , z 0 j } j ∈ [ | V | ] can be accurately estimated using quadratic fitting techniques [16].

Additionally, for the probabilistic constraint in Eq. (2), if the noise is not assumed to be normal anymore, one can analyze the CDF of the noise distribution to find a new probability region with a similar approach. This can further assist in constructing a substitute constraint. Moreover, there are instances where the feasible domain of the constraint in Eq. (6) may not exist due to its stringent nature, e.g. , one aims to completely avoid undesired futures with 100% probability, or the desired region S is not practically achievable. In such cases, the decision-maker should consider verifying specified region S or reducing hyperparameter τ to some extent.

Last but not least, though we mainly focus on the scenario where the graph structure is fixed and known, i.e. , G t = G , however, our approach can be straightforwardly adapted to the case where G t = G but unknown as assumed in Qin et al. [40] as detailed in Appendix A.3. We also provide a simple comparable experiment on Bermuda data in the setting of Qin et al. [40], as illustrated in Appendix D.1.

A.2 Comparation with causality

The SRM and Rh ( · ) operations are similar to their counterparts in causal inference but have different applicating scenarios. The differences between rehearsal learning and causal learning are listed as follows:

1 Most of the causal learning problems focus on structure or effect identification, while the decisionmaking process is not directly involved in the modeling. In contrast, rehearsal learning mainly focuses on a class of decision-making problems that specializes in the goal of avoiding undesired future using SRM-based modeling.

2 Causal learning utilizes the SCM. In contrast, rehearsal learning uses the SRM (a new probabilistic graphical model [40]) to model the influence relations between the variables toward addressing the AUF problem. The modeling granularity of SRM is more flexible, as the influence relationship can be evolved, and mutually influenced. Specifically,

Possible dynamic influence relationships. For instance, the influence relationship between pricing and sales of coats can vary cyclically and trend-wise over time. Thus, a given price may result in fewer sales during summer compared to winter. In addition, the coat's style may gradually become outdated, leading to reduced sales at the same price a year later compared to the present.

Possible mutually influenced relationships. For instance, the ideal gas law states that the pressure p , volume V , amount of substance n , and absolute temperature T obey the equation pV = nRT ( R is the ideal gas constant). When a fixed volume V of an ideal gas is considered, with n and R held constant, the pressure p and volume V can be represented as a pair of mutually influenced variables within an SRM modeling.

A.3 Possible types of graph structures

There are four possible cases of the graph structure, specifically:

1 G t fixed, known. It's the setting in our paper.

2 G t fixed, unknown. It's the setting in Qin et al. [40]. Our method can be straightforwardly adapted to this case. Technically, since the distribution P ( G ) is discrete, we can initialize and update the probability mass function (PMF) P ( G ) in the same manner as Qin et al. [40]. Thus, when we want to select decision alterations, the maintained PMF P ( G ) can be utilized to marginalize G from P ( Y ∈ S | G, · · · ) . thereby the expectation of matrices A , B , C in Eq. (6) can be obtained and used.

3 G t not fixed, known. Our method can be directly used because once the graph structure is known, G is not a stochastic component. Thus, matrices A , B , C in Eq. (6) can be obtained in the same manner. It

is worth noting that the changing graph structure will lead to the birth of new parameters at each round. Therefore, though the method can be directly used in such case, establishing theoretical guarantees is difficult.

4 G t not fixed, unknown. In this case, neither θ nor P G can be accurately estimated as there is only one sample per round. Thus, dealing with such a case may need to additional assumptions. For example, consider there are a sufficient number of samples per round. In this case, our method can model θ and P G accurately, thereby suggesting good decisions.

B Definitions

B.1 Details about Structural Rehearsal Models

In this section, we present full definitions and discussions for the Structural Rehearsal Model (SRM), which is a probabilistic graphical model proposed by [40] to characterize the influence relations among variables.

· Definition of the Rehearsal Graph

Definition B.1 (Mixed graph, [40]) . Let G = ( V , E ) be a graph, where V denotes the vertices and E the edges. G is a mixed graph if for any distinct vertices u, v ∈ V , there is at most one edge connecting them, and the edge is either directional ( u → v or u ← v ) or bi-directional ( u ↔ v ) .

Definition B.2 (Bi-directional clique, [40]) . A bi-directional clique C = ( V c , E c ) of a mixed graph G = ( V , E ) is a complete subgraph induced by V c ⊆ V such that any edge e ∈ E c is bi-directional. C is maximal if adding any other vertex does not induce a bi-directional clique.

Definition B.3 (Rehearsal graph, [40]) . Let G = ( V , E ) be a mixed graph. Let { C i } l i =1 denote all maximal bi-directional cliques of G , where C i = ( V c i , E c i ) . G is a rehearsal graph if and only if:

glyph[negationslash]

V c i ∩ V c j = ∅ for any i = j .

∀ i ∈ [ l ] , if there is any edge pointing from some u ∈ V \ V c i to some v ∈ V c i , then ∀ v ∈ V c i , u → v .

There exists a topological ordering for { C i } l i =1 following the directions of directional edges between C i s.

It can be found that the topological ordering for bi-directional cliques { C i } l i =1 in the rehearsal graph reflects the time order of the generation process of variables.

· Discussion of the Structural Equations

In this paper, we present the definition of dynamic structural equations in Eq. (1), which provides a quantitative computational formulation for the influence relations in non-stationary environments.

In fact, the structural equations are defined among the bi-directional cliques { C i } l i =1 as detailed in Qin et al. [40]. Specifically, the dynamic structural equations can be denoted as θ t , which consists of the set of parameter matrices { β i,t } T t =1 and the covariance matrix Σ i of each clique C i in the rehearsal graph G . It is noteworthy that the quantitative influence relationship of a directional edge A → B for A ∈ C a , B ∈ C b at time t is modeled in the parameter matrix β b,t since the topological ordering for bi-directional cliques reflects the time order of the generation process; while the quantitative influence relationship of a bi-directional edge D 1 ↔ D 2 for D 1 , D 2 ∈ C d at time t is modeled in the covariance matrix Σ d since they are in the same bi-directional clique which is viewed as happening in the same time.

We present a simplified version of the definition of dynamic structural equations in Eq. (1) for a convenient presentation, which defines the dynamic structural equations on the variable level rather than the clique level. Note that for each variable V j ∈ C i , the associated parameter vectors { β j,t } T t =1 can be directly found in the corresponding parameter matrices { β i,t } T t =1 ; while the variance σ 2 j of each variable V j ∈ C i can be derived from the corresponding covariance matrix Σ i by marginalizing the desired dimension out, since the marginalization operator for multi-normal distribution is available.

B.2 Increasing Marginal Cost with an Example

In this section, we introduce the increasing marginal cost (IMC) property. Increasing marginal cost (IMC), or rising marginal cost, is an important property in microeconomics [23, 32]. We first present a famous example in the field of microeconomics, called Thirsty Thelma's Lemonade Stand.

Lemonade Stand is a business simulation game devised in 1973 by Bob Jamison under the Minnesota Educational Computing Consortium. It offers players the experience of managing a lemonade stand across multiple rounds. At the commencement of each round, players make decisions regarding their stock, pricing, and advertising based on their current financial standing. The outcomes in each round are determined by the player's choices; and are further influenced by random events like thunderstorms and street closures. After each round, a summary of the current status of the player is provided, and the game concludes after 12 rounds. In this game, an essential quantity is the total cost of lemonade per glass, which will increase in a quadratic way, as it has a linearly increasing derivative (named the marginal cost) as shown in Fig. 5.

0

2

4

6

8

10

Quantity of Lemonade (glass per hour)

0

2

4

6

8

10

12

14

Costs (dollars $)

Total Cost Curve

Marginal Cost Curve

This phenomenon is commonly observed in real-world scenarios, and the upward slope of the total cost curve reflects the property of the law of diminishing marginal product [32]. When Thirsty Thelma produces a small quantity of lemonade, her workforce is limited, and much of her equipment remains idle. The ease of utilizing these unused resources results in a substantial marginal product for an additional worker and a corresponding small marginal cost for producing an extra glass of lemonade. In contrast, as she increases lemonade production, her shop becomes crowded with workers, and most of her equipment is fully utilized. While additional workers can contribute to increased production, they must operate in crowded conditions and may face delays in equipment usage. Consequently, when the quantity of lemonade produced is already high, the marginal product of an extra worker diminishes, leading to a significant increase in the marginal cost of producing an extra glass of lemonade. It is a famous and widely used example in the field of microeconomics, and it has many similarities with the AUF scenario. In our proposed cost-minimal AUF problem, when the decision-maker decides to alter some variables in Z t , the cost associated with different alterations vary. For the same variable, the cost of altering it from a datum point should have a similar shape to the total cost curve in Fig. 5, varying with an increasing derivative. That is how we define the cost function in this paper. The cost function could be replaced by any other convex function in different situations.

C Proofs

In this section, we provide proof for claims in the main text.

C.1 Proof for Proposition 3.2

Lemma C.1. If Σ is a symmetric positive definite matrix, the following two sets are equivalent:

where x and u are vectors with a center at the origin of the coordinate system O .

Proof. Assume P 1 = { x : x glyph[latticetop] Σ -1 x ≤ 1 } and P 2 = { x : x = Σ 1 2 u | ‖ u ‖ 2 ≤ 1 } . Then what we need to prove is equivalent to:

Points contained in P 1 have to be contained in P 2 .

Points contained in P 2 have to be contained in P 1 .

For (1), it holds that x glyph[latticetop] Σ -1 x ≤ 1 , which is equivalent to ( Σ -1 2 x ) glyph[latticetop] ( Σ -1 2 x ) ≤ 1 . Thus, let u = Σ -1 2 x , it can be derived that x = Σ 1 2 u where ‖ u ‖ 2 ≤ 1 , and (1) is proved.

For (2), we replace x glyph[latticetop] Σ -1 x with x = Σ 1 2 u , then it can be derived that:

which shows that points contained in P 2 are also contained in P 1 , so (2) is proved. In conclusion, the lemma is proved.

Proposition 3.2. The following set P is a probability region that satisfies P ( Y t ∈ P | θ t , x t , Rh ( ξ t )) = τ :

where µ y t = Ax t + Bz ξ t , u is an arbitrary point in the unit sphere in R | Y t | , and χ -1 ( · ) denotes the quantile function of the χ 2 distribution with degrees of freedom λ = | Y t | .

Proof. Recall from Lemma 3.1 that:

From the property of multi-normal distributions, it holds that:

Let µ y t = Ax t + Bz ξ t , then we normalize the distribution above, it can be derived that:

So it can be derived that:

where the χ 2 distribution has the degrees of freedom λ = | Y t | .

Let χ -1 ( · ) denote the quantile function of the distribution χ 2 λ above, then P can be given as:

which is equivalent to:

By lemma C.1, it is equivalent to:

where µ y t = Ax t + Bz ξ t , and u is an arbitrary point in the unit sphere in R | Y t | .

C.2 Proof for Theorem 3.3

Lemma C.2 (Theorem 3 of Cutler et al. [11]) . Consider a sequence of stochastic optimization problems min β f t ( β ) indexed by time t ∈ N . Let ˆ g t denote the estimation for the gradient of f t , and β glyph[star] t denote the minimizer of the L -smooth and µ -strongly convex function f t . Suppose it holds that max t ˆ g t ≤ σ 2 < ∞ and max t E ‖ β glyph[star] t +1 -β glyph[star] t ‖ ≤ ∆ < ∞ . Then the produced { β t } T t =0 with iterates β t +1 = Π B ( β t -η ˆ g t ) ( B = R | β | ) and constant learning rate η ≤ 1 / 2 L satisfies:

Lemma C.3 (Convergence of LSE) . Let β j,t denote the true parameter value of β j in time t , and ˆ β lse j,t denote the estimation according to Eq. (4). For ∀ ψ > 0 , ∃ n > 0 that holds:

Proof. Let P j,t = [ PA 1 j,t . . . PA n j,t ] glyph[latticetop] and v j,t = [ V 1 j,t , . . . , V n j,t ] glyph[latticetop] . Then the optimization in Eq. (4) is equivalent to:

Since PA j represents the true parents of the generation process for variable V j , P j,t is guaranteed to have full column rank. For ∀ n ≥ | PA j | , the solution to the optimization above is:

Because v j,t = P j,t β j,t + ε j where ε j ∼ N ( 0 , σ 2 j I n ) according to Eq. (3), we have:

Due to the linear combination property of the Gaussian distribution, it holds that:

Thus, it can be derived that:

Because ∥ ∥ ∥ ( P glyph[latticetop] j,t P j,t ) -1 ∥ ∥ ∥ 2 · ∥ ∥ P glyph[latticetop] j,t P j,t ∥ ∥ 2 ≤ ∥ ∥ ∥ ( P glyph[latticetop] j,t P j,t ) -1 P glyph[latticetop] j,t P j,t ∥ ∥ ∥ 2 = 1 , we have:

So for ∀ ψ > 0 , ∃ n = σ 2 j | PA j | 2 ( ψ min k ∑ | PA j | i =1 p 2 ik ) -1 that holds E ∥ ∥ ∥ ˆ β lse j,t -β j,t ∥ ∥ ∥ 2 2 ≤ ψ .

Lemma C.4 (Positive-definite Property) . Consider a set of m -dimensional vectors { v i } n i =1 , n ≥ m . If it contains at least one set of basis vectors in space R m , then the following matrix is positive-definite:

Proof. Let { u k } m k =1 ⊆ { v i } n i =1 denote a set of basis vectors, and let x ∈ R m denote an arbitrary m -dimensional points in the same space as v i s. Then it can be derived that:

glyph[negationslash]

because there at least exists one basis vector u h that holds x glyph[latticetop] u h = 0 . So M is positive-definite (thus M is full-rank).

Theorem 3.3. Denote the true parameter of β j in time t as β j,t ( j ∈ [ | V | ] ) and choose η j ∈ (0 , 1 / 2 L j ] as the step size used in Algorithm 1, it can be bounded in G that:

where µ j and L j are the minimal and maximal eigenvalues of { glyph[lscript] j,t ( · ) } T t =1 's Hessian matrices respectively, σ 2 upper-bounds the variance of ˆ g j,t used in Algorithm 1, ∆ j ≥ max t ‖ β j,t +1 -β j,t ‖ upper-bounds the varying speed of the environment, and m is the longest continuously altered rounds of V j , for most of the V j s, m = 1 .

Proof. For Eq. (4), the Hessain matrix of glyph[lscript] j,t ( · ) can be derived that:

Since n glyph[greatermuch] | PA j | and PA k j,t s are n discrete samples from a continuous distribution, so { PA k j,t } n k =1 contains a set of basis vectors in space R | PA j | with probability 1. By Lemma C.4, { H j,t } T t =1 are positive-definite.

Let µ j denote min t λ ( H j,t ) and L j denote max t λ ( H j,t ) respectively, then { glyph[lscript] j,t ( · ) } T t =1 are all L j -smooth and µ j -strongly convex. By Lemma C.2 and consider alterations on V j , estimations { ˆ β j,t } T t =1 obtained from Algorithm 1 hold that:

where σ 2 upper-bounds the variance of ˆ g j,t used in Algorithm 1, ˆ ∆ j ≥ max t ‖ β glyph[star] j,t +1 -β glyph[star] j,t ‖ upper-bounds the varying speed of the minimizer of glyph[lscript] j,t , and m is the longest continuously altered rounds of V j , for most of the V j s, m = 1 . This holds because m ˆ ∆ j ≥ m max t ‖ β glyph[star] j,t +1 -β glyph[star] j,t ‖ ≥ max t ∑ m k =1 ‖ β glyph[star] j,t + k -β glyph[star] j,t + k -1 ‖ ≥ max t ‖ β glyph[star] j,t + m -β glyph[star] j,t ‖ . Note that if V j is continuously altered ≤ m rounds, it can be viewed as its corresponding parameters are updated per m rounds.

Besides, since ˆ β lse j,t glyph[defines] β glyph[star] j,t +1 in such case, let ∆ j ≥ max t ‖ β glyph[star] j,t +1 -β glyph[star] j,t ‖ , by Lemma C.3, it holds that ˆ ∆ j glyph[lessorsimilar] ∆ j , ∥ ∥ ∥ ˆ β j, 0 -β glyph[star] j, 0 ∥ ∥ ∥ 2 glyph[lessorsimilar] ∥ ∥ ∥ ˆ β j, 0 -β j, 0 ∥ ∥ ∥ 2 , E ∥ ∥ β glyph[star] j,t -β j,t ∥ ∥ 2 ≤ ψ ; and we have:

C.3 Proof of Proposition 3.4

Lemma C.5 (Hoeffding, 1963) . Let X be a random variable with a ≤ X ≤ b . Then, for any s ∈ R ,

Proposition 3.4. Assume { glyph[lscript] j,t ( · ) } T t =1 s are bounded for ∀ β i ∈ B and t ∈ [ T ] ; then for any η ∈ H j , estimations ˆ β j,t s from Algorithm 2 satisfies that

by choosing α = √ ln N j /T in Algorithm 2, where N j is the number of base-learners, β η j,t is the estimation from any expert η in expert set H j in Algorithm 2.

Proof. Following previous studies [8] (Theorem 2.2 and Exercise 2.5), we define:

From the updating rule in Algorithm 2, it is easy to verify that:

First, it can be derived that:

Next, the related quantity ln ( W t /W t -1 ) can be bounded as follows when t ≥ 2 :

When t = 1 , it holds that ln W 1 = ln ( ∑ η ∈H w η 1 e -α ˆ glyph[lscript] j, 1 ( β η j, 1 ) ) , thus it can be derived that:

where the inequality in the second line is due to Lemma C.5, and the inequality in the second line is due to Jensen's inequality. By combining Eq. (8) and Eq. (9), it can be derived that:

Since we choose w η 1 = 1 N j in Algorithm 2, thus for η ∈ H j , by choosing α = √ ln N j /T , it holds that:

C.4 Proof for Theorem 3.5

Theorem 3.5. By using the suggested alterations from Eq. (6), it can be guaranteed that:

Proof. Recall that Eq. (6) suggests alterations as follows:

where P = ( χ -1 ( τ ) CΣC glyph[latticetop] ) 1 2 , and ‖ · ‖ 2 , row means an operator that takes 2-norm for each row of the matrix thus outputs a row-dimensional vector. We omit the subscript 2 of the norm in the following discussions.

Let γ i denote the i -th row-vector of matrix MP and r denote the row-dimension of MP , then it can be derived that:

where µ y t = Ax t + Bz ξ t t . Recall from Prop. 3.2 that P = { µ y t + ( χ -1 ( τ ) CΣC glyph[latticetop] ) 1 2 u } is the probability region that satisfies P ( Y t ∈ S | ˆ θ t , x t , Rh ( ξ t ) ) = τ (since A , B , C are generated from ˆ θ t rather than θ t in this case). Thus, Prop. 3.2 has been proved because (a) it has been derived that for ∀ y t ∈ P it holds that d ≥ My t ; and (b) the desired region S defined in Eq. (3) is S = { y ∈ R | Y | | My ≤ d } .

D Experimental details

The experiments are done by using macOS Monterey, Apple M1 Pro. All algorithms are running under the same environment.

D.1 Comparison experiment in the setting of Qin et al. [40]

We provide a comparison experiment in the setting of Qin et al. [40], where G ∼ P ( G ) and P ( G ) are unknown. Results on Bermuda data provided in the following table show that our approach exhibits a comparable performance with the result under the scenario where G t = G .

D.2 Market-Manage Data

In this section, we provide details about the Market-Manage data. The variables included in the generation process are:

Featureour : The feature used to predict the raw cost of our market;

Featurecpt : The feature used to predict the raw cost of the competitor market;

Cour : The raw cost of our market;

Ccpt : The raw cost of the competitor market;

Pour : The product price of our market;

Pcpt : The product price of the competitor market;

NCT: Customer numbers of our market;

TPF: Total profit of our market.

The rehearsal graph for the variables is illustrated in Fig. 6. The presumed actionable variables that can be altered by the manager are Cour and Pour . The hyperparameters associated with the cost function are set as Z 0 Cour = 0 . 75 , w Cour = 2 . 0 ; and Z 0 Pour = 0 . 0 , w Pour = 1 . 0 . The desired region S is shown in Fig. 7(a). We shade dynamic edges with red color.

feature

1

X

t

Z

t

Y

t

NCT

TPF

feature

1

Cour

Pour

Ccpt

Pcpt

6

4

2

0

2

4

6

4

3

2

1

0

1

2

3

4

Natural NCT Value

Original

Y

Distribution

Desired Region

Natural TPF Value

(a) Y in nature and region S for Market data

4

Original

Y

Distribution

Desired Region

3

2

1

0

1

2

3

Natural NEC (

Y

) Value

(b) Y in nature and region S for Bermuda data

D.3 Bermuda Data

In this section, we provide details about the Bermuda data. The Bermuda data is an environment dataset that involves some environmental variables in Bermuda [10]. The variables included in the generation process are:

Light: Light levels at the bottom;

Temp: Temperature at the bottom;

Sal: Sea surface salinity;

DIC: Dissolved inorganic carbon of seawater;

TA: Total alkalinity of seawater;

Ω A : Saturation with respect to aragonite in seawater;

Chla: Chlorophyll-a at sea surface;

Nut: PC1 of NH 4 , NiO 2 + NiO 3 , SiO 4 ;

pHsw: pH of seawater;

CO 2 : PCO 2 of seawater;

NEC: Net ecosystem calcification.

0.35

0.30

0.25

0.20

0.15

0.10

0.05

0.00

Y

Empirical CDF for

The rehearsal graph for the variables is illustrated in Fig. 8. The presumed actionable variables that can be altered by the decision-maker are DIC, TA, Ω A , Chla, and Nut according to Aglietti et al. [1], Qin et al. [40]. The hyperparameters associated with the cost function are set as Z 0 DIC = Z 0 TA = Z 0 Ω A = Z 0 Chla = Z 0 Nut = 0 . 0 ; and w DIC = 10 . 0 , w TA = 8 . 0 , w Ω A = 3 . 0 , w Chla = 5 . 0 , w Nut = 10 . 0 . The desired region S is shown in Fig. 7(b). We shade dynamic edges with red color.

Light

Temp

DIC

Ω

A

NEC

Sal

TA

Nut

pHsw

CO

2

Chla

X

t

Z

t

Y

t

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Contributions of the paper are summarized in the abstract and listed in the last paragraph of Sec. 1. Meanwhile, the scope is also introduced in Sec. 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the work are discussed in the separate "Discussion" section in Appendix A, as pointed out in the first paragraph of Sec. 3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are listed in Eq. 3 (with its contexts), and in each Theorem/Proposition. All proofs are given in Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The code and data for the main experimental results are provided in the supplemental material. Dataset and experimental details are introduced in Sec. 5 and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and data for the main experimental results are provided in the supplemental material. The dataset used in the experiments is also accessible from the link of the reference.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Dataset and experimental setting are introduced in the core of the paper (Sec. 5). The full details are provided in Appendix D and the code contained in the supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars and error bands are provided in Fig. 4. As introduced in the first paragraph of Sec. 5, we repeat the experiments 20 times (20 different random seeds).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The time of execution is denoted in Table 1. Information on the computer resources is introduced in the first paragraph of Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper has discussed in Sec. 1 and Sec. 6 that our proposed algorithm could be used to make decisions in certain cases, which may potentially lead to some positive societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work focuses on the methodology level and poses no such risks, because the datasets used in the paper are synthetic or have been used in previous academic articles. Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The dataset used in the experiment is properly cited including a URL in reference. Meanwhile, the original papers of models used in the competitive experiment are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code and model introduced in the paper are well documented, and the documentation is provided alongside the assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the non-stationary setting in avoiding undesired future (AUF) problems, where environmental shifts can cause the failure of existing AUF methods. It introduces an optimization problem for AUF with minimal action cost in non-stationary environments, formulated as a convex quadratically constrained quadratic program (QCQP) in each interaction. The paper also proposes a rehearsal-based algorithm to solve this problem, providing theoretical guarantees and numerical validations.\n\nSTRENGTHS:\nThe paper is well-written, introduces a practical and interesting setting for AUF problems, and presents an algorithm with theoretical guarantees and numerical validations to address the task.\n\nWEAKNESSES:\n(1) The provided algorithms lack a regret bound (or other theoretical guarantees) on the cost (i.e., the objective function), although it guarantees effective alterations (i.e., the constraint). Since the aim of this work is to avoid an undesired future with minimal cost, a regret bound analysis is, in my opinion, important.\n\n(2) In Theorem 3.3, the estimation error depends on the minimum eigenvalues of  the empirical error functions' Hessian matrices, which in turn depends on the previously taken alterations. This raises a concern about the exploration-exploitation tradeoff when making alterations. An extreme case is making uninformative alterations (e.g., setting 0 for all nodes), leading to no update by Algorithm 1 and rendering the error bound in Theorem 3.3 meaningless (since $\\mu_j$=0 in this case if I understand correctly). It is unclear how Algorithm 3 addresses this tradeoff and how $\\mu_j$’s can be bounded  below.\n\nQUESTIONS:\n(1) How do algorithms handle the exploration-exploitation tradeoff (if explorations are needed)?\n\n(2) Is it possible to establish a regret bound for the cost? If not, what are the challenges?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors address decision-making problem that sufficient interactions are not available. In this case, RL is not suitable. The authors model the structure among the observed variables, and use the structure to help the decisions. Compared to the previous studies [Qin et al. 37], the method can be used in a dynamic environment and can efficiently find the suggested decision (in polynomial time). To deal with the dynamic environment, they introduce the online learning method (Alg. 2). To efficiently find the suggested decision, they convert the optimization problem to a QCQP problem, which can be implemented in polynomial time. The experimental results verify the effectiveness.\n\nSTRENGTHS:\n1. The method of Qin et al. [37] suffers a high computational cost. In this paper, the authors convert the problem to a QCQP problem, which makes it computable in polynomial time. It is a valuable contribution.\n\n2. Theorem 3.3 presents an interesting and sensible theoretical guarantee. It is novel to see that some traditional online learning methods could be used in such decision tasks.\n\nWEAKNESSES:\nSome discussion about the offline RL are missing. See Questions for the details.\n\nGiven the results of theorem 3.5: I do not know where $\\tau$ is reflected in your algorithm. It seems that $\\tau$ is never mentioned in Section 3.3. It is a bit wired, and needs more illustrations.\n\nThe writing could be improved. There are some weird sentences. I suggest the authors carefully revise the paper. For example, \"We provide the theoretical guarantees of our method, and experimental results validate the effectiveness and efficiency of the method.\" -> \"We provide the theoretical guarantees for our method. Experimental results validate the effectiveness and efficiency of the method.\"\n\nQUESTIONS:\nI agree that RL is not suitable for the setting. However, I am wondering why offline RL cannot be used instead? Relevant discussions are missing.\n\nI can understand that the problem is hard in the non-linear case. Could authors have some discussions for the case that the data is non-linear?\n\nLIMITATIONS:\nNo limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors formulate the Avoiding Undesired Future (AUF) problem in real-world scenarios of decision-making, especially in non-stationary environments, and propose a method to avoid undesired outcomes with minimal costs. Here the non-stationarity majorly comes from the different costs corresponding to different actions, and the varying influence relations over time. They also provide theoretical guarantees of their method and empirical results demonstrate the effectiveness and efficiency of the proposal.\n\nSTRENGTHS:\n- This paper is written well and clearly, with intuitive motivation and clarified novelty.\n\n- This paper includes a complete theoretical analysis and algorithmic design. Their proposed problem formalization is more general and practical than existing methods [37]. In particular, they first proposed a sequential method to maintain the dynamical influence, with guarantees of estimation error bound. They entailed Proposition 3.2 and Theorem 3.5 to help find the efficient alteration for $Z_t$ with the minimal cost. They finally propose the whole algorithm called AUF-MICNS, to avoid undesired outcomes in each decision round.\n\n- Experimental results show the effectiveness and efficiency of their proposed algorithm, where the evaluation metrics are success frequency, estimation error, average running time, etc.\n\nWEAKNESSES:\nI think my major concerns have been settled by the Supplementary Materials. So I have no other comments about the weaknesses.\n\nQUESTIONS:\nFor the differences between SRM in the rehearsal graph and SCM in causality:\n- In the linear cases, it is easy to define the coefficients as the influences. When in the nonlinear cases, how to define the influences in the rehearsal graph? Is it the same as in causation, e.g., definitions of causal influence or causal effects? \n- Can the influence in rehearsal graphs (SRM) represent the bi-directional edge information? If not, I am confused what are the differences between such bi-directional relations in rehearsal learning and causality. Though in [35], the bidirectional edges are often due to common causes between two variables, there also exist some works that use causality to represent mutually influenced relationships[1*]. A causal graph can also include cycles.\n- The operators in Figure 2 seem identical to the Intervention operator in causality.\n\n[1*] Vimaleswaran K S, Berry D J, Lu C, et al. Causal relationship between obesity and vitamin D status: bi-directional Mendelian randomization analysis of multiple cohorts[J]. PLoS medicine, 2013, 10(2): e1001383.\n\nThere are other minor typo errors:\n- It seems that in Eq.(1) or Eq.(3), it is better to add $t$ as a subscript for $V_j$ and $\\varepsilon_j$?\n- In line 181, \"ound\" might be \"round\".\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-17"}