Title: Learning the Optimal Policy for Balancing Short-Term and Long-Term Rewards
Venue: NeurIPS 2024
Accepted: yes
Keywords: Policy Learning

Short-Term and Long-Term Rewards

Causal Inference

Decision Making
Abstract: Learning the optimal policy to balance multiple short-term and long-term rewards has extensive applications across various domains. Yet, there is a noticeable scarcity of research addressing policy learning strategies in this context. In this paper, we aim to learn the optimal policy capable of effectively balancing multiple short-term and long-term rewards, especially in scenarios where the long-term outcomes are often missing due to data collection challenges over extended periods. Towards this goal, the conventional linear weighting method, which aggregates multiple rewards into a single surrogate reward through weighted summation, can only achieve sub-optimal policies when multiple rewards are related. Motivated by this, we propose a novel decomposition-based policy learning (DPPL) method that converts the whole problem into subproblems. The DPPL method is capable of obtaining optimal policies even when multiple rewards are interrelated. Nevertheless, the DPPL method requires a set of preference vectors specified in advance, posing challenges in practical applications where selecting suitable preferences is non-trivial. To mitigate this, we further theoretically transform the optimization problem in DPPL into an $\varepsilon$-constraint problem, where $\varepsilon$ represents the minimum acceptable levels of other rewards while maximizing one reward. This transformation provides intuitive into the selection of preference vectors. Extensive experiments are conducted on the proposed method and the results validate the effectiveness of the method.

Learning the Optimal Policy for Balancing Short-Term and Long-Term Rewards

Abstract

Learning the optimal policy to balance multiple short-term and long-term rewards has extensive applications across various domains. Yet, there is a noticeable scarcity of research addressing policy learning strategies in this context. In this paper, we aim to learn the optimal policy capable of effectively balancing multiple short-term and long-term rewards, especially in scenarios where the long-term outcomes are often missing due to data collection challenges over extended periods. Towards this goal, the conventional linear weighting method, which aggregates multiple rewards into a single surrogate reward through weighted summation, can only achieve suboptimal policies when multiple rewards are related. Motivated by this, we propose a novel decomposition-based policy learning (DPPL) method that converts the whole problem into subproblems. The DPPL method is capable of obtaining optimal policies even when multiple rewards are interrelated. Nevertheless, the DPPL method requires a set of preference vectors specified in advance, posing challenges in practical applications where selecting suitable preferences is non-trivial. To mitigate this, we further theoretically transform the optimization problem in DPPL into an ε -constraint problem, where ε represents the minimum acceptable levels of other rewards while maximizing one reward. This transformation provides intuitive into the selection of preference vectors. Extensive experiments are conducted on the proposed method and the results validate the effectiveness of the method.

1 Introduction

Learning an optimal policy for balancing multiple short-term and long-term rewards holds extensive applications across various domains. For instance, content providers can optimize recommendations to avoid short-term clickbait strategies, ensuring sustained user engagement and revenue growth [1]. IT companies can design web pages catering to immediate user preferences while enhancing longterm engagement and satisfaction [2]. Economists explore the effects of early childhood interventions on lifetime earnings, seeking optimal policies (e.g., class size) maximizing short-term test scores and long-term earnings simultaneously [3]. Policymakers can improve job training program design, considering both immediate income impacts and subsequent employment status improvements [4, 5]. Medical practitioners can refine drug prescriptions, considering short-term alleviation and long-term outcomes in chronic diseases like Alzheimer's and AIDS [6]. Marketing professionals can optimize incentive strategies to positively influence customer behavior in both short and long terms [7].

Despite the importance of balancing multiple short-term and long-term rewards, policy learning methods in this area remain largely unexplored. Recent literature [8] employs a linear weighting method to achieve this goal. It combines multiple rewards into a single surrogate reward by weighted summation, which is optimized to learn the optimal policy. However, this strategy has several limitations. First, it can only find optimal solutions in convex regions of objective space and cannot obtain the optimal solutions in non-convex regions [9]. Second, it achieves the optimal solution only

when the rewards are independent of each other. When some of the rewards are interrelated, it can only achieve sub-optimal solutions [10]. Consequently, although the linear weighting method is easy to implement, the optimality of its solution cannot be guaranteed when balancing multiple objectives.

In this article, we propose a principled policy learning approach for balancing multiple long-term and short-term rewards (objectives). Specifically, we first formulate it as a multiple-objective problem (MOP) and aim to seek the Pareto optimal solutions (policies). A solution is Pareto optimal if improving one objective necessitates worsening other objectives. Then, we propose a novel decomposition-based policy learning (DPPL) method, which involves (1) introducing a set of preference vectors, (2) dividing the whole optimization problem into several subproblems based on the preference vectors, and (3) ultimately achieving different Pareto solutions for the objectives by solving these subproblems. Compared with the linear weighting method, it can obtain Pareto optimal solutions in non-convex regions and is applicable to cases where multiple objectives are interrelated.

While the proposed DPPL method can find Pareto optimal policies, it necessitates specifying a set of preference vectors in advance. In practical applications, decision-makers may encounter the challenge of determining which preference vector to choose. To mitigate this concern, we further theoretically transform the optimization problem in DPPL into an ε -constraint problem. This transformation can assist decision-makers in better understanding and selecting preference vectors.

The contributions of this paper are summarized as follows.

· We formulate the policy learning problem of balancing multiple long-term and short-term rewards as a multi-objective optimization problem and propose a decomposition-based Pareto policy learning (DPPL) method to obtain a set of Pareto optimal policies.

· We theoretically establish the connection between the DPPL method and the ϵ -constraint problem, offering an intuitive interpretation of preference vectors and guiding their selection.

· We conduct extensive experiments to demonstrate the effectiveness of the proposed method.

2 Problem Formulation

Throughout, we employ bold letters for vectors, uppercase letters for random variables, and lowercase letters for their realization values.

2.1 Notation

We introduce notations to delineate short-term and long-term causal effects. Let A denote the binary treatment indicator, where A = 1 represents the treated group and A = 0 represents the control group. X represents the features observed, S = ( S 1 , ..., S I ) ∈ R I and Y = ( Y 1 , ..., Y J ) ∈ R J represent the vector of short-term and long-term outcomes, respectively. Both short-term and long-term outcomes are observed after the treatment A , and associations among them may exist.

Utilizing the potential outcome framework [11], we denote S ( a ) = ( S 1 ( a ) , ..., S I ( a )) and Y ( a ) = ( Y 1 ( a ) , ..., Y J ( a )) for a = 0 , 1 as the potential short-term and long-term outcomes under treatment A = a , respectively. We assume that larger short-term and long-term outcomes are preferable. The observed short-term and long-term outcomes S and Y correspond to the potential outcomes of the actual treatment, that is, S = S ( A ) and Y = Y ( A ) .

In real-world applications, long-term outcomes often suffer from missing due to prolonged follow-up periods and budget constraints. In contrast, collecting short-term outcomes is more manageable. Therefore, we presume that all short-term outcomes S are observable, while long-term outcomes Y may be subject to missing. Let R = ( R 1 , ..., R J ) ∈ { 0 , 1 } J denote the indicator for observing the long-term outcome Y , where R j = 1 indicates that Y j is observed and R j = 0 indicates that Y j is missing. The missingness of Y would lead to identifiability and estimation problems [12-23].

2.2 Formulation

In this article, we aim to learn the Pareto optimal policy for balancing multiple correlated shortterm and long-term rewards, which has a wide range of application scenarios [1, 6, 8, 24]. Let π : X → { 0 , 1 } be a policy that maps from the individual context X = x to the treatment space

{ 0 , 1 } . For a given policy π ( θ ) = π ( X , θ ) parameterized by θ , the policy values for the i -th short-term outcome S i and the j -th long-term outcome Y j are defined as,

which are the i -th short-term reward and the j -the long-term reward induced by the policy π ( θ ) .

Conventionally, we convert maximization problems to minimization problems. Let ¯ V ( θ ; s i ) = -V ( θ ; s i ) , ¯ V ( θ ; y j ) = -V ( θ ; y j ) . The trade-off among multiple correlated long-term and short-term rewards can be formulated as a multi-objective optimization (MOP) problem given by

where M = I + J and the symbol ≜ means 'denoted as'. Generally, there is no single solution that can simultaneously optimize all objectives in problem (1) and thus we resort to the Pareto optimality. This concept is employed to define the optimal solutions for the MOP problem.

Definition 1. (Pareto optimality)

(a) Pareto dominance. For two points θ 1 , θ 2 . θ 1 dominates θ 2 if and only if ¯ V m ( θ 1 ) ≤ ¯ V m ( θ 2 ) , ∀ m ∈ { 1 , ..., M } and ¯ V m ′ ( θ 1 ) < ¯ V m ′ ( θ 2 ) , ∃ m ′ ∈ { 1 , ..., M }

(b) Pareto optimality. θ ∗ is a Pareto optimal point if there is no other solution ˆ θ that dominates θ ∗ .

Pareto optimality refers to a condition where improving one objective comes at the expense of worsening other objectives. The collection of Pareto optimal solutions is called the Pareto set. Our goal is to derive the set of Pareto optimal solutions (or Pareto optimal policies), each of them providing a distinct optimal trade-off among all objectives.

2.3 Identification and Estimation of Short-term and Long-term Rewards

The long-term and short-term rewards are causal parameters that cannot be identified without imposing causal assumptions [25-27]. Therefore, before seeking the Pareto optimal solutions for balancing multiple long-term and short-term rewards, it is necessary to consider the identification and estimation of long-term and short-term rewards. The proposed method is based on Assumptions 1 and 2 below.

Assumption 1 (Strong Ignorability) .

Assumption 1(a) suggests that, given the feature X , treatment assignment A is independent of the potential outcomes S ( a ) and Y ( a ) . This implies that confounding bias between the treatment A and the short/long-term outcomes ( S ( a ) , Y ( a )) can be eliminated by conditioning on X [28]. Assumption 1(b) ensures that for the subpopulation of X = x , units with both A = 1 and A = 0 exist. These assumptions are widely used in causal inference [11, 27, 29-35].

In addition to confounding bias, we also need to address the selection bias induced by the missingness of long-term outcomes [8]. Thus, we further invoke the Assumption 2.

Assumption 2 (Missing Mechanism of Long-term Outcome) . For a = 0 , 1 and j = 1 , ..., J ,

Assumption 2(a) can be reformulated as R j ⊥ ⊥ Y j | ( X , S , A ) , which means that R j relies only on the observed variables ( X , A, S ) . This assumption also ensures that P ( Y j = y | X , S , A, R j = 1) = P ( Y j = y | X , S , A, R j = 0) . This implies that we can utilize the available data to draw conclusions about the missing long-term outcome. Assumption 2(b) assumes that the long-term outcome for each unit has a non-zero probability of being observed. Assumptions 1 and 2 ensures the identifiability of V ( θ ; s i ) and V ( θ ; y j ) , as shown in Lemma 1.

Lemma 1 (Identifiability of Short-term and Long-term Rewards) . For i = 1 , ..., I and j = 1 , ..., J , (a) under Assumptions 1, the i -th short-term reward V ( θ ; s i ) is identifiable. (b) under Assumptions 1-2, the j -th long-term reward V ( θ ; y j ) is identifiable.

When we have access to only one short-term outcome and one long-term outcome, Lemma 1 reduces to the identifiability result presented in [8]. In this article, our focus is on achieving the Pareto optimal policy for multiple short-term and long-term rewards. Therefore, for the estimation of V ( θ ; s i ) and V ( θ ; y j ) , we defer it to Appendix A.

3 Pareto Policy Learning for Balancing Short-Term and Long-Term Rewards

In this section, we aim to learn Pareto optimal policies for the MOP problem (1). Section 3.1 gives the motivation for this work and Section 3.2 introduces the proposed policy learning approach. In Section 3.3, we theoretically establishe the connection between the linear weighting method, the MOP problem for a given preference vector, and the ε -constraint problem. This connection offers an intuitive interpretation and guides practitioners in selecting the preference vector.

3.1 Motivation

For seeking the optimal policy for balancing short-term and long-term rewards, previous work [8] adopted the linear weighting method. Specifically, the authors formulate the goal as

where ω m is the pre-specified weight for the m -th objective. The objective function in the optimization problem (2) is merely a linear combination of multiple objectives from the MOP problem (1). Due to its intuitiveness and simplicity, the traditional linear weighting method is commonly used for solving MOP or multi-task learning problems [36-38].

The linear weighting method simply combines multiple objectives into a single surrogate objective through weighted summation. While simple, it has several limitations. First, the optimal solution is found only in convex regions and not in non-convex regions [9]. Second, an optimal solution can only be achieved if the objectives are independent of each other. That is, if some objectives are interrelated, only a suboptimal solution can be obtained [10]. Thus, it does not guarantee the superiority of the solution or its solution may deviate from the Pareto optimal solution.

To overcome the limitations of the linear weighting method in [8], we first introduce a decompositionbased multi-objective optimization algorithm to achieve the Pareto optimal policy. However, this algorithm relies on pre-specified preference vectors, which are used to express a decision maker's degree of preference for multiple conflicting objectives. In practice, the explanation and selection of preference vectors is a challenging problem. To further tackle this issue, we establish a theoretical relationship between preference vectors and the ε -constraint method [39]. This relationship provides a clear interpretation on preference vectors, assisting in selecting more suitable ones.

3.2 Pareto Policy Learning for the MOP Problem

We introduce the decomposition-based Pareto policy learning (DPPL) method, which can generate the Pareto set containing policies that are optimum from a trade-off perspective. The main idea of the DPPL method is to first decompose the original MOP problem into several constrained subproblems based on a predefined set of preference vectors, and then obtain a set of Pareto optimal policies by solving these subproblems in parallel [40].

For obtaining the Pareto optimal policy for balancing M short-term and long-term objectives, first, we are given a set of K preference vectors { u 1 , u 2 , ..., u K } in R M + . Each element of a preference vector specifies the importance of the corresponding short-term or long-term reward. For each preference vector u k , the corresponding subproblem is given as

where G k ′ ( θ t ) ≤ 0 means that objective space 2 of the subproblem is restricted in the subregion Ω k , which is defined by Ω k = { v ∈ R M + | u T k ′ v ≤ u T k v , ∀ k ′ = 1 , ..., K } . Geometrically speaking, Ω k represents the set of v that forms the smallest acute angle with u k , which means that the optimal solution of the subproblem can be obtained by only searching the subregion. The preference vectors divide the objective space into different subregions.

Solving the subproblem (3) involves the following two steps:

Step (a) . Find a reasonable initial solution θ 0 . Specifically, we first randomly generate a solution θ r in the full decision space 3 , and then iteratively update it with the rule θ r t +1 = θ r t + η r d r t , where η r is the step size. For a given θ r t , the descent direction d r t is updated by solving (4).

where I ( θ r t ) = { k ′ |G k ′ ( θ r t ) ≥ 0 , k ′ = 1 , ..., K } is index set of all activated constraints, which means ¯ V ( θ r t ) not in Ω k . The problem (4) aims to find the descent direction d r t for each iteration t and then obtain the initial solution θ 0 such that ¯ V ( θ 0 ) in Ω k .

Step (b) . Solving the subproblem (3). The descent direction d t for the t -th iteration is obtained by

where I ϵ ( θ ) = { k ′ |G k ′ ( θ ) ≥ -ϵ } , and the threshold ϵ is a slack variable used to deal with the solutions near the constraint boundary. We further transform it into a dual problem which will greatly reduce the dimension of decision space. Based on the KKT conditions, we have d t = -( ∑ M m =1 λ m ∇ ¯ V m ( θ t ) + ∑ k ′ ∈I ϵ ( θ ) β k ′ ∇G k ′ ( θ t )) . Therefore, the dual problem is given as

where λ m ≥ 0 and β k ′ ≥ 0 are the Lagrange multipliers for the linear inequality constraints.

Step (a) is to find an initial solution θ 0 that is restricted in a subregion of the subproblem (3), and once a feasible solution is found or a predetermined number of iterations is reached, the step stops. For an given initial solution θ 0 , Step (b) is to find the optimal solution θ ∗ for the subproblem (3). We summarize the proposed policy learning approach in Appendix B.

Lemma 2 ([41]) . Let ( d t , α t ) be the solution to the t -th iteration of problem (5) .

(a) If θ t is Pareto optimal restricted on Ω k , then d t = 0 ∈ R m and α t = 0 .

(b) If θ t is not Pareto optimal restricted on Ω . then

̸

Lemma 2(a) implies that at the t -th iteration, no direction ( d t = 0 ) can simultaneously improve the performance for all objectives, confirming that the solution θ t satisfies Pareto optimality. Lemma 2(b) suggests that if θ t does not meet Pareto optimality, then the descent direction d t = 0 serves as the descent direction for all objectives, such that the solution of the next iteration is closer to the Pareto optimal solution. Thus, Lemma 2 demonstrates that we always attain Pareto optimal solutions for each subproblem using the update rule θ t +1 = θ t + η r d t . By solving all subproblems, we can acquire a diverse set of Pareto optimal solutions (or policies) confined to different subregions, even when the multiple objectives are correlated.

3.3 Deep Analysis of the Preference Vector

The DPPL method in Section 3.2 requires a set of pre-specified preference vectors, posing challenges in practical applications where selecting suitable preference vectors is non-trivial. To mitigate this problem, we provide a practical method for decision-makers to select appropriate preference vectors by theoretically establishing the connection between the DPPL method and the ε -constraint problem.

We first give a brief introduction to the ε -constraint problem [10], which is defined as follows,

̸

where ε m is pre-specified threshold. Compared to the MOP problem (1) and the linear weighting objective (2), a notable advantage of the ε -constraint problem is its interpretation on the threshold ε m , which represents the maximum acceptable value (i.e., the acceptable worst-case scenario) for the m -th objective. In contrast, the weights and the preference vectors in problems (1) and (2) are not straightforward for relating the resulting values of objectives. Thus, if we can establish the connection between ε = ( ε 1 , ..., ε M ) and the preference vector u k , then we can provide powerful guidance for choosing appropriate preference vectors.

Theorem 1. For the preference vector u k = ( u k 1 , ..., u kM ) in problem (1) , the weights ω = ( ω 1 , ...., ω M ) in problem (2) , and the thresholds ε in problem (8) , the following statements hold:

(a) the connection between ε and ω is given as

̸

where τ m ( X ) is the conditional average causal effects for m -th short/long-term outcome,

I ( · ) is the indicator function, and

(b) the connection between ω and u k is given as

where λ m and β k ′ are defined in Eq. (6) , and I ϵ ( θ ) = { k ′ |G k ′ ( θ ) ≥ -ϵ } defined in Eq. (4) .

Theorem 1 (see Appendix C for proofs) establishes a link between the preference vector u k and ε through ω in scenarios involving multiple long-term and short-term objectives. Specifically, Theorem 1(a) shows how to estimate the threshold ε for given weights ω , and Theorem 1(b) shows how to assign weights ω via preference vectors u k . This means that for the subproblem determined by preference vectors u k , we can ascertain the maximum acceptable threshold ε based on Theorem 1, thereby offering an intuitive interpretation of the preference vector u k .

There are several practical implications with Theorem 1. On one hand, it assists decision-makers in better understanding and selecting preference vectors in practical applications. In practice, we can initially pre-specify a set of preference vectors { u 1 , u 2 , ..., u K } in R M + , then derive the weights ω corresponding to each preference vector u k through Eq. (10), and finally substitute the obtained weight ω into Eq. (9) to calculate the threshold ε . Leveraging the intuitive interpretability of the threshold ε , decision-makers can select the appropriate preference vectors according to their specific requirements. On the other hand, it also provides guidance for specifying ε in the ε -constraint problem (8). Inappropriate selection of ε for this problem may result in an empty feasible region, yielding empty solutions. By utilizing a set of preference vectors, we can efficiently screen out some reasonable choices of ε and reduce the cumbersome trial-and-error process of testing different ε .

In conclusion, by establishing the connection between the DPPL method and the ε -constraint problem, we can harness the advantages of both methods while mitigating their respective weaknesses.

4 Experiments

Datasets. Following the previous studies [8], we use two widely used datasets: IHDP and JOBS, for evaluating the performance of the proposed method. The IHDP dataset explores the effectiveness of high-quality home visiting in promoting children's future cognitive development and covers a sample of 747 units, including 139 treated and 608 controlled. In addition, the dataset has 25 characteristics that provide a comprehensive picture of the children and their mothers. The second dataset, JOBS, explores the effects of job training on income and employment status. It consists of 2,570 units (237 treated, 2,333 controlled), with 17 covariates. Note that each unit in both datasets has only one observed outcome from a single treatment, and neither dataset collects long-term outcomes.

Simulating Outcome. Consider the case of one long-term reward and one short-term reward. Following the previous data-generation mechanisms [1, 42], for the n -th unit ( n = 1 , ..., N ), we simulate the potential short-term outcomes S (0) and S (1) as follows:

where σ ( · ) is the sigmoid function, w 0 ∼ N [ -1 , 1] (0 , 1) follows a truncated normal distribution, w 1 ∼ Unif( -1 , 1) follows a uniform distribution, ϵ 0 ,n ∼ N ( µ 0 , σ 0 ) and ϵ 1 ,n ∼ N ( µ 1 , σ 1 ) . We set µ 0 = 1 , µ 1 = 3 and σ 0 = σ 1 = 1 for the IHDP dataset, and we set µ 0 = 0 , µ 1 = 2 and σ 0 = σ 1 = 1 for the JOBS dataset. For generating long-term potential outcomes Y (0) and Y (1) , we introduce the time step t : we set the initial value at time step 0 as: Y 0 ,n (0) = S n (0) , Y 0 ,n (1) = S n (1) , then generate Y t,n (0) , Y t,n (1) according to the following equation and we eventually regard the outcome at the last time step T as the long-term outcome, Y n (0) = Y T,n (0) , Y n (1) = Y T,n (1) .

where β 0 is randomly sampled from { 0 , 1 , 2 , 3 , 4 } with probabilities { 0 . 5 , 0 . 2 , 0 . 15 , 0 . 1 , 0 . 05 } , β 1 ∼ 4 · N [0 , 4] (0 , 1) , and C = 1 /T is a scaling factor. For ϵ 0 ,n and ϵ 1 ,n , we set µ 0 = µ 1 = 0 , σ 0 = 1 and σ 1 = 3 for the IHDP dataset and set µ 0 = µ 1 = 0 , σ 0 = 1 and σ 1 = 1 for the JOBS dataset.

Assumption 2 shows that observing indicator R depends on the feature X , the treatment A , and shortterm outcome S . For a given missing rate r , we select the missing indexes for Y and derive the missing indicator R according to the following criterion: calculate the m n = 1 /D ∑ D d =1 ( X nd + s n ) , n = 1 , · · · , N, and choose the index of the row with the smallest rN values in { m n , n = 1 , · · · , N } as the missing indexes. D is the feature dimension and N is the sample size.

Experimental Details. In this paper, preference vectors are used to quantify an individual's preference for different objectives in the multi-objective optimization problem. For the case of two-objective, we randomly generate 10 unit preference vectors ( u 1 , u 2 , · · · , u 10 ) , where u k = ( u k 1 , u k 2 ) , u k 1 = cos ( t k ) , u k 2 = sin ( t k ) , t k ∈ (0 , 1) , which implies that the L 2 -norm of the preference vectors is 1, ensuring the consistency and comparability of the preference measures. u k 1 and u k 2 are the preferences for the short-term objective and the long-term objective, respectively. Each component of the preference vector u k represents the strength or importance of the decision maker's preference for different objectives. Preference vectors are used as weights in the linear weighting method, whereas our method uses them to divide the original problem (1) into several subproblems.

Evaluation Metrics. We measure the performance of our proposed method by three metrics: long and short-term rewards, the variance of long and short-term rewards, and the change in welfare. Formally, the short-term reward of the learned policy ˆ π ( X , θ ) is ˆ V ( θ ; s ) = ∑ N n =1 [ˆ π ( X n , θ ) S n (1) + (1 -ˆ π ( X n , θ )) S n (0)] , the long-term reward is ˆ V ( θ ; y ) = ∑ N n =1 [ˆ π ( X n , θ ) Y n (1)+(1 -ˆ π ( X n , θ )) Y n (0)] . Similar as [42, 43], the welfare changes are defined as ∆ W s = ∑ N n =1 [( S n (1) -S n (0)) · ˆ π ( X n , θ )] for the short-term reward, ∆ W y = ∑ N n =1 [( Y n (1) -Y n (0)) · ˆ π ( X n , θ )] for the long-term reward, ∆ W = 0 . 5∆ W s +0 . 5∆ W y for the overall balanced-base reward. Among these metrics, ∆ W is the most critical here, as it directly measures the balance reward achieved by the learned policy.

Policy learning with short-term and short-term reward. We choose MLP as the policy model π ( θ ) , and we average over 50 independent trials of policy learning with the short-term and long-term reward in IHDP and JOBS. We fix the missing ratio r = 0 . 2 and the time step T = 4 . We measure the uncertainty of the model by calculating the variance of the long and short-term reward over 50 experiments, and a smaller variance means a more stable model performance.

Performance Comparison. From our previous analyses, the linear weighting method generally achieves the sub-optimal policies. The proposed DPPL method can generate a set of Pareto optimal policies. First, for the long-term reward, the short-term reward, and ∆ W , it is not surprising to observe that for most of the preference vectors, DPPL's solutions have better performance. Second, for the variance, our method performs more stable in 50 experiments. Because we will divide the original problem into several subproblems according to preference vectors, and then solve the subproblems in a relatively small subregion to obtain the Pareto optimal solution, whereas the linear weighting method searches the entire objective space. The associated results are displayed in Table 1. More experimental results with missing ratio r = 0 . 3 are given in Appendix D.

Sensitivity Analysis. We perform the sensitivity analysis of missing ratio r and time step T on JOBS. Our method achieves better performance in all missing rates r = [0 . 2 , 0 . 3 , 0 . 4 , 0 . 5] with T = 4 , and r = 0 . 2 with time step T = [4 , 6 , 8 , 10] . Our method stably outperforms the linear weighting method under varying r and T , even in scenarios with a high missing ratio or a large time step. This further illustrates the effectiveness of our method. The associated results are displayed in Figure 1.

(a) r = 0.2 on JOBS

(b) r = 0.3 on JOBS

(c) r = 0.4 on JOBS

(d) r = 0.5 on JOBS

(e) T = 4 on JOBS

(f) T = 6 on JOBS

(g) T = 8 on JOBS

(h) T = 10 on JOBS

our

linear weighting

our

linear weighting

our

linear weighting

our

linear weighting

150

150

150

150

125

125

125

125

=

100

100

≥100

≥100

Delta_

75

Delta

75

Delta_

75

elta

75

50

50

50

50

25

25

25

25

0

12345678910

thei'th preference vector

0

12345678910

the i'th preference vector

0

12345678910

the i'th preference vector

0

12345678910

thei'thpreferencevector

our

linear weighting

our

linear weighting

our

linear weighting

our

linear weighting

150

150

150

150

125

125

125

125

100

≥100

100

Delta_

75

e

75

Delta_

75

Delta

75

50

50

50

50

25

25

25

25

0

2.3.45.678910

thei'thpreference vector

0

1

23.45.678910

the i'th preference vector

0

1

2.3.45678910

the i'th preference vector

0

2.3.45.678910

the i'th preference vector

Interpretation on preference vectors. By Theorem 1, for the set of pre-specified preference vectors ( u 1 , u 2 , · · · , u 10 ) , we transform the optimization subproblem corresponding to each preference vector into the ε -constraint problem as min θ ¯ V ( θ ; y ) , s.t. ¯ V ( θ ; s ) ≤ ε ( < 0) or

max θ V ( θ ; y ) , s.t. V ( θ ; s ) ≥ -ε and the threshold -ε are shown in Table 2. This value of -ε is the minimum value of the short-term reward that the decision maker can accept while maximizing the long-term reward. Our results show that as the second component of the preference vector increases, the value of -ε shows a decreasing trend. In essence, this signifies that a decision-maker who emphasizes the long-term reward must necessarily loosen constraints on the short-term reward. In practice, decision makers can determine the threshold based on their specific needs for the short-term reward, and then select the most appropriate preference vector from the set of pre-specify preference vectors with the help of the intuitive interpretability of the threshold according to Table 2. More experimental results with different missing ratios { 0 . 3 , 0 . 4 , 0 . 5 } are provided in Appendix D.

5 Related Work

Estimation of long-term causal effects. Assessing long-term causal effects is challenging due to the delayed long-term outcomes, posing significant difficulties in both identification and estimation. Recently, there has been increasing interest in using short-term surrogates to identify and estimate long-term causal effects, such as [4, 5, 7, 13, 44, 45]. In contrast to these previous works focusing on long-term causal effects, this paper aims to balance multiple short-term and long-term causal effects.

Trustworthy policy learning. Trustworthy policy learning ensures that the learned policies or models are reliable and dependable for practical applications. Traditional policy learning aims to identify individuals who would maximize the utility function based on their features if treated [46]. Recently, trustworthy policy learning has focused on ensuring that the learned policy adheres to principles such as beneficence, non-maleficence, autonomy, justice, no-harm, and explicability [42, 47-49]. Various counterfactual-based metrics have been suggested to assess a policy's trustworthiness [42, 50-53]. In this paper, we complement this series of work by developing a principled policy learning approach that can effectively balance multiple rewards.

Multi-objective optimization (MOP) . MOP aims to find compromises or trade-offs among multiple possibly contrasting objectives. It is widely used in the field of machine learning such as multi-task learning [40, 54], neural architecture search [55], and multi-objective reinforcement learning [5658]. We extend these works to a new setting by learning the optimal policy for balancing multiple long-term and short-term rewards. Additionally, we provide a practical method for interpreting and selecting preference vectors with theoretical guarantees.

6 Conclusion

In this paper, we focus on learning the optimal policy for balancing multiple long-term and shortterm rewards. We reveal the limitations of the previous linear weighting method, which usually results in sub-optimal policies in practice. To address these limitations, we formulate the policy learning problem as a multi-objective optimization problem and then propose the novel DPPL method to learn optimal policies. The DPPL method obtains a set of Pareto optimal policies by solving a series of subproblems based on pre-specified preference vectors, effectively balancing multiple objectives. Furthermore, we theoretically establish the connection between the optimization subproblems in the DPPL method and the ε -constraint problem. This connection aids decision-makers in better understanding and selecting preference vectors. We conducted extensive experiments on two benchmark datasets which validate the effectiveness of our proposed method. A limitation of this work is that it focuses on discrete treatments in identification and estimation (Section 2.3). In some application scenarios, continuous treatments (e.g., price) are of interest. Further investigation is required to extend the proposed method to accommodate such cases.

References

Lu Cheng, Ruocheng Guo, and Huan Liu. Long-term effect estimation with surrogate representation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining , page 274-282, 2021.

Henning Hohnhold, Deirdre O'Brien, and Diane Tang. Focusing on the long-term: It's good for users and business. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1849-1858, 2015.

Raj Chetty, John N. Friedman, Nathaniel Hilger, Emmanuel Saez, Diane Whitmore Schanzenbach, and Danny Yagan. How does your kindergarten classroom affect your earnings? evidence from project star. The Quarterly Journal of Economics , 126:1593-1660, 2007.

Susan Athey, Raj Chetty, Guido W Imbens, and Hyunseung Kang. The surrogate index: Combining short-term proxies to estimate long-term treatment effects more rapidly and precisely. Technical report, National Bureau of Economic Research, 2019.

Susan Athey, Raj Chetty, and Guido Imbens. Combining experimental and observational data to estimate treatment effects on long term outcomes. arXiv preprint arXiv:2006.09676 , 2020.

Wenjie Hu, Xiao-Hua Zhou, and Peng Wu. Identification and estimation of treatment effects on long-term outcomes in clinical trials with external observational data. Statistica Sinica , 2023.

Jeremy Yang, Dean Eckles, Paramveer Dhillon, and Sinan Aral. Targeting for long-term outcomes. Management Science , 2023.

Peng Wu, Ziyu Shen, Feng Xie, Zhongyao Wang, Chunchen Liu, and Yan Zeng. Policy learning for balancing short-term and long-term rewards. In ICML , 2024.

Yair Censor. Pareto optimality in multiobjective problems. Applied Mathematics and Optimization , 4(1):41-59, 1977.

Jürgen Branke. Multiobjective optimization: Interactive and evolutionary approaches , volume 5252. Springer Science & Business Media, 2008.

G. W. Imbens and D. B. Rubin. Causal Inference For Statistics Social and Biomedical Science . Cambridge University Press, 2015.

Susan Athey, Raj Chetty, Guido Imbens, and Hyunseung Kang. The surrogate index: Combining short-term proxies to estimate long-term treatment effects more rapidly and precisely. Working paper, National Bureau of Economic Research, 2019.

Nathan Kallus and Xiaojie Mao. On the role of surrogates in the efficient estimation of treatment effects with limited outcome data. arXiv preprint arXiv:2003.12408 , 2020.

Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, Peng Wu, and Peng Cui. Propensity matters: Measuring and enhancing balancing for recommendation. In ICML , 2023.

Peng Wu, Shanshan Luo, and Zhi Geng. On the comparative analysis of average treatment effects estimation via data combination. arXiv preprint arXiv:2311.00528 , 2024.

Sihao Ding, Peng Wu, Fuli Feng, Xiangnan He, Yitong Wang, Yong Liao, and Yongdong Zhang. Addressing unmeasured confounder for recommendation with sensitivity analysis. In KDD , 2022.

Haoxuan Li, Quanyu Dai, Yuru Li, Yan Lyu, Zhenhua Dong, Xiao-Hua Zhou, and Peng Wu. Multiple robust learning for recommendation. In AAAI , 2023.

Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, and Peng Wu. Balancing unobserved confounding with a few unbiased ratings in debiased recommendations. In WWW , 2023.

Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong, Xiao-Hua Zhou, Rui Zhang, Xiuqiang He, Rui Zhang, and Jie Sun. A generalized doubly robust learning framework for debiasing post-click conversion rate prediction. In KDD , 2022.

Peng Wu, Haoxuan Li, Yuhao Deng, Wenjie Hu, Quanyu Dai, Zhenhua Dong, Jie Sun, Rui Zhang, and Xiao-Hua Zhou. On the opportunity of causal learning in recommendation systems: Foundation, estimation, prediction and challenges. In IJCAI , 2022.

Haoxuan Li, Yan Lyu, Chunyuan Zheng, and Peng Wu. TDR-CL: Targeted doubly robust collaborative learning for debiased recommendations. In ICLR , 2023.

Haoxuan Li, Chunyuan Zheng, and Peng Wu. Stabledr: Stabilized doubly robust learning for recommendation on data missing not at random. In ICLR , 2023.

Haoxuan Li, Chunyuan Zheng, Sihao Ding, Peng Wu, Zhi Geng, Fuli Feng, and Xiangnan He. Be aware of the neighborhood effect: Modeling selection bias under interference for recommendation. In ICLR , 2024.

Guido Imbens, Nathan Kallus, Xiaojie Mao, and Yuhao Wang. Long-term causal inference under persistent confounding via data combination. arXiv preprint arXiv:2202.07234 , 2022.

Judea Pearl, Madelyn Glymour, and Nicholas P. Jewell. Causal Inference in Statistics: A Primer . John Wiley & Sons, 2016.

Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect . Hachette Book Group, 2018.

M.A. Hernán and J. M. Robins. Causal Inference: What If . Boca Raton: Chapman and Hall/CRC, 2020.

Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika , 70(1):41-55, 1983.

Kosuke Imai and Marc Ratkovic. Covariate balancing propensity score. Journal of the Royal Statistical Society (Series B) , 76(1):243-263, 2014.

Elizabeth A. Stuart. Matching methods for causal inference: A review and a look forward. Statistical Science , 25(1):1-21, 2010.

Paul R. Rosenbaum. Design of Observational Studies . Springer Nature Switzerland AG, second edition, 2020.

Peng Wu, Xinyi Xu, Xingwei Tong, Qing Jiang, and Bo Lu. Semiparametric estimation for average causal effects using propensity score-based spline. Journal of Statistical Planning and Inference , 212:153-168, 2021.

Peng Wu, Zhiqiang Tan, Wenjie Hu, and Xiao-Hua Zhou. Model-assisted inference for covariatespecific treatment effects with high-dimensional data. Statistica Sinica , 34:459-479, 2024.

Peng Wu, Shasha Han, Xingwei Tong, and Runze Li. Propensity score regression for causal inference with treatment heterogeneity. Statistica Sinica , 34:747-769, 2024.

Peng Wu, Peng Ding, Zhi Geng, and Yue Li. Quantifying individual risk for binary outcome: Bounds and inference. arXiv:2402.10537 , 2024.

Kalyanmoy Deb, Karthik Sindhya, and Jussi Hakanen. Multi-objective optimization. In Decision sciences , pages 161-200. CRC Press, 2016.

Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering , 34(12):5586-5609, 2022.

Haoxuan Li, Chunyuan Zheng, Yanghao Xiao, Hao Wang, Fuli Feng, Xiangnan He, Zhi Geng, and Peng Wu. Removing hidden confounding in recommendation: A unified multi-task learning approach. In NeurIPS , 2023.

Amir Ismail-Yahaya and Achille Messac. Effective generation of the pareto frontier using the normal constraint method. In 40th AIAA Aerospace Sciences Meeting & Exhibit , page 178, 2002.

Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. Advances in neural information processing systems , 32, 2019.

Jörg Fliege and Benar Fux Svaiter. Steepest descent methods for multicriteria optimization. Mathematical methods of operations research , 51:479-494, 2000.

Haoxuan Li, Chunyuan Zheng, Yixiao Cao, Zhi Geng, Yue Liu, and Peng Wu. Trustworthy policy learning under the counterfactual no-harm criterion. In International Conference on Machine Learning , pages 20575-20598. PMLR, 2023.

T. Kitagawa and A. Tetenov. Who should be treated? empirical welfare maximization methods for treatment choice. Econometrica , 86, 2018.

Lu Cheng, Ruocheng Guo, and Huan Liu. Long-term effect estimation with surrogate representation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining , pages 274-282, 2021.

Wenjie Hu, Xiaohua Zhou, and Peng Wu. Identification and estimation of treatment effects on long-term outcomes in clinical trials with external observational data. arXiv preprint arXiv:2208.10163 , 2022.

Michael R Kosorok and Eric B Laber. Precision medicine. Annual review of statistics and its application , 6:263-286, 2019.

Scott Thiebes, Sebastian Lins, and Ali Sunyaev. Trustworthy artificial intelligence. Electronic Markets , 31:447-464, 2021.

Floridi Luciano. Establishing the rules for building trustworthy ai. Nature Machine Intelligence , 1(6):261-262, 2019.

Davinder Kaur, Suleyman Uslu, Kaley J Rittichier, and Arjan Durresi. Trustworthy artificial intelligence: a review. ACM Computing Surveys (CSUR) , 55:1-38, 2022.

Silvia Chiappa. Path-specific counterfactual fairness. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 7801-7808, 2019.

Yongkai Wu, Lu Zhang, Xintao Wu, and Hanghang Tong. PC-Fairness: A unified framework for measuring causality-based fairness. Advances in Neural Information Processing Systems , 32, 2019.

Nathan Kallus. Treatment effect risk: Bounds and inference. Management Science , 69(8):45794590, 2023.

Nathan Kallus. What's the harm? sharp bounds on the fraction negatively affected by treatment. Advances in Neural Information Processing Systems , 35:15996-16009, 2022.

Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems , 31, 2018.

Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. Dpp-net: Deviceaware progressive search for pareto-optimal neural architectures. In Proceedings of the European conference on computer vision (ECCV) , pages 517-531, 2018.

Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL) , pages 191-199. IEEE, 2013.

Nan Xu, Nitin Kamra, and Yan Liu. Treatment recommendation with preference-based reinforcement learning. In 2021 IEEE international conference on big knowledge (ICBK) , pages 1-8. IEEE, 2021.

Jiangjiao Xu, Ke Li, and Mohammad Abusara. Preference based multi-objective reinforcement learning for multi-microgrid system optimization problem in smart grid. Memetic Computing , 14(2):225-235, 2022.

A Estimation of Short-Term and Long-Term Rewards

For a given policy π ( θ ) , the policy values for the short-term outcome S i and the long-term outcome Y j are defined as

Under Assumptions 1-2, the short-term reward V ( θ ; s i ) and long-term reward V ( θ ; y j ) are identified as

where µ ia ( X ) = E [ S i | X , A = a ] , ˜ m ja ( X , S ) = E [ Y j | X , S , A = a, R j = 1] for a = 0 , 1 . The identifiability results are derived using a similar approach to that outlined in Section 5 of [8]. In addition, for estimating the V ( θ ; s i ) and V ( θ ; y j ) , [8] proved the efficient bounds of V ( θ ; s i ) and V ( θ ; y j ) , which we list them below for the sake of self-containedness.

Lemma A.1 (Efficiency Bounds of V ( θ ; s i ) and V ( θ ; y j ) , [8]) . Let Z = ( X , A, S , Y ) , under Assumptions 1-2, we have that

(a) the efficient influence function of V ( θ ; s i ) is ϕ s i -V ( θ ; s i ) , where

and e ( X ) = P ( A = 1 | X ) is propensity score. The associated semiparametric efficiency bound of V ( θ ; s i ) is Var( ϕ s i ).

(b) the efficient influence function of V ( θ ; y j ) is ϕ y j -V ( θ ; y j ) , where

m ja ( X ) = E [ Y j | X , A = a, R j = 1] is the regression function for Y j , and r j ( A, X , S ) = P [ R j = 1 | X , S , A ] is selection score. The associated semiparametric efficiency bound of V ( θ ; y j ) is Var ( ϕ y j ) .

From Lemma A.1, for a given policy π ( θ ) , it is natural to define the estimators of V ( θ ; s i ) and V ( θ ; y j ) as

where N is the sample size. All of them can be identified from the observed data. And ˆ e ( x ) , ˆ µ ia ( x ) , ˆ m ja ( x ) , ˆ ˜ m ja ( x , s ) , and ˆ r j ( a, x , s ) for a = 0 , 1 are the estimators of e ( x ) , µ ia ( x ) , m ja ( x ) , ˜ m ja ( x , s ) and r j ( a, x , s ) respectively.

B Algorithm Flowchart for DPPL

Algorithm 1 DPPL Algorithm

1: Input: A set of preference vectors { u 1 , u 2 , ..., u K } (All subproblems can be solved in parallel)

2: for k = 1 to K do

3: randomly generate parameters θ ( k ) r

4: find the initial parameters θ ( k ) 0 from θ ( k ) r using gradient-based method (step a)

5: for t = 1 to T do

6: obtain λ ( k ) tm ≥ 0 , β ( k ) tk ′ ≥ 0 , ∀ m = 1 , ..., M, ∀ k ′ ∈ I ϵ ( θ ) by solving subproblem (6)

7:

calculate direction

d

(

t

=

-

(

∑

M

i

=

m

λ

(

k

)

tm

∇

¯

V

m

(

θ

(

t

)+

∑

k

′

∈

8: update the parameters θ ( k ) t +1 = θ ( k ) t + ηd ( k ) t

9: end for

10: end for

11: Output: The set of solutions for all subproblems with different trade-offs { θ ( k ) T | k = 1 , . . . , K }

C Proofs of Theorem 1

For the case of only have one long-term outcome Y and one short-term outcome S , considering the ε -constraint optimization problem

and the linear weighting optimization problem

which can be reformulated as where λ = ω 2 /ω 1 controls the balance between short-term and long-term rewards. Let τ s ( X ) = E [ S (1) -S (0) | X ] and τ y ( X ) = E [ Y (1) -Y (0) | X ] . When λ = 0 , it is equivalent to finding an optimal policy for minimizing ¯ V ( θ ; y ) alone, π ∗ y ( θ ) = arg min π ¯ V ( θ ; y ) = arg min π -E [ π ( θ ) τ y ( X )] = I ( τ y ( X ) ≥ 0) . When λ = ∞ , it is equivalent to finding an optimal policy for minimizing the ¯ V ( θ ; s ) alone, π ∗ s ( θ ) = arg min π ¯ V ( θ ; s ) = arg min π -E [ π ( θ ) τ s ( X )] = I ( τ s ( X ) ≥ 0) . We have the following theorem:

Theorem C.1. For the weights ω in problem (A.5) , and the thresholds ε in problem (A.1) , the following statements hold:

When ε < -E [ π ∗ s ( θ ) S (1) + (1 -π ∗ s ( θ )) S (0)] , the solution of the constrained optimization problem is empty.

When ε ≥ -E [ π ∗ s ( θ ) S (1)+(1 -π ∗ s ( θ )) S (0)] , the relationship between λ and α is described as follows:

-λ is the solution of the equation

It is important to note that for a given λ , we could solve the value of ε by solving the equation

as the left side of the equation is a monotone function of λ and the solution is unique, and all the quantities such as τ s ( X ) , τ y ( X ) , and µ 0 ( X ) are identifiable.

k

)

k

)

β

(

k

tk

)

′

∇G

′

(

θ

(

t

k

)

))

I

ϵ

(

θ

)

k

/

d

(

t

k

)

=

Proof. Initially, we recognize that ε cannot be too small so that no policy can satisfy the constraint of ¯ V ( θ ; s ) ≤ ε . The optimal policy of minimizing only the ¯ V ( θ ; s ) is π ∗ s ( θ ) = I ( τ s ( X ) ≥ 0) . Thus, ε ≥ -E [ π ∗ s ( θ ) S (1) + (1 -π ∗ s ( θ )) S (0)] .

When ε ≥ -E [ π ∗ s ( θ ) S (1) + (1 -π ∗ s ( θ )) S (0)] . First, the optimal policy of minimizing only ¯ V ( π ; y ) is given as π ∗ y ( θ ) = I ( τ y ( X ) ≥ 0) . Then, ε ≤ -E [ π ∗ y ( θ ) S (1) + (1 -π ∗ y ( θ )) S (0)] . otherwise, the constraint will be invalid and the constrained optimization problem becomes an unconstrained optimization problem with λ = 0 .

Second, when -E [ π ∗ s ( θ ) S (1) + (1 -π ∗ s ( θ )) S (0)] ≤ ε ≤ -E [ π ∗ y ( θ ) S (1) + (1 -π ∗ y ( θ )) S (0)] , we show that the optimal policy π ∗ ( θ ) parameterized by θ ∗ , for the constrained optimization problem

is obtained only when ¯ V ( θ ∗ ; s ) = ε . Below, we prove it with the method of reduction to absurdity. If -E [ π ∗ s ( θ ) S (1) + (1 -π ∗ s ( θ )) S (0)] ≤ ε ≤ -E [ π ∗ y ( θ ) S (1) + (1 -π ∗ y ( θ )) S (0)] , then there are some units that satisfies { τ s ( X ) < 0 , τ y ( X ) > 0 } that not being assigned treatment by π ∗ ( θ ) ; otherwise, the constraint ¯ V ( θ ; s ) ≤ ε will be violated. Thus, we could find another treatment policy ˜ π ∗ that assigns more treatment to the units with { τ s ( X ) < 0 , τ y ( X ) > 0 } , which yields a lower ¯ V ( θ ; y ) but increases the ¯ V ( θ ; s ) . That is, ˜ π ∗ will lead to a ¯ V ( θ ; s ) closer to ε but has a lower ¯ V ( θ ; y ) than π ∗ , thus, π ∗ is not the optimal policy, which contradicts its definition of π ∗ . Thus, the constrained optimization problem becomes

By introducing the Lagrange multiplier β , π ∗ satisfies

where β is the solution of ¯ V ( θ ∗ ; s ) = ε , i.e.,

This completes the proof for Theorem C.1.

We can further extend Theorem C.1 to situations where there are multiple long-term rewards and multiple short-term rewards. More generally, for the ε -constraint optimization problem

̸

and the linear weighting optimization problem

where ω m is the pre-specified weight for the m -th reward. We have the following theorem:

Theorem 1. For the preference vector u k in problem (1) , the weights ω in problem (2) , and the thresholds ε in problem (8) , the following statements hold:

(a) the connection between ε and ω is given as

̸

where τ m ( X ) is the conditional average causal effects for m -th short/long-term outcome,

and

and I ( · ) is the indicator function.

(b) the connection between ω and u k is given as

where λ m and β k ′ are defined in Eq. (6) , I ϵ ( θ ) = { k ′ |G k ′ ( θ ) ≥ -ϵ }

Proof. First, for the Theorem1(a), combining the TheoremC.1, more generally, for the ε -constraint problem

̸

and the linear weighting optimization problem

where ω m is the pre-specified weight for the m -th reward. By mathematical induction, we have:

̸

h m 0 ( X ) = { µ i 0 ( X ) = E [ S i | X , A = 0] , if ω m is the weight of ¯ V ( θ , s i ) , ˜ m j 0 ( X , S ) = E [ Y j | X , S , A = a, R j = 1] , if ω m is the weight of ¯ V ( θ , y j ) , This completes the proof for Theorem 1(a)

Second, for the Theorem (b), motivated by [40], for constraint problem

we have

Base on KKT conditions, we have

where λ m ≤ 0 and β k ′ ≤ 0 are the Lagrange multipliers. Then, the dual problem is given as

Substituting Eq.(A.8) into Eq.A.9, we have

For the problem(A.6), d t is the negative gradient direction. Thus, we have

where λ m and β k ′ is obtained from Eq.(A.10). This shows that the DPPL method can be transformed into the linear weighting method. This completes the proof for Theorem 1(b)

D Additional Experimental Results

D.1 Sensitivity Analysis on Missing Ratio

In the following, we show more experimental result with missing ratio r = 0 . 3 under IHDP and JOBS datasets, in table D1.

In additional, We show the corresponding ε value for each preference vector with different missing ratio { 0 . 3 , 0 . 4 , 0 . 5 } under IHDP and JOBS datasets, in tables D2, D3 and D7.

D.2 Sensitivity Analysis on Preference Vector

In the following, we show more experimental result with different numbers of preference vectors K = { 4 , 8 , 12 } under JOBS datasets, in table D5-D7.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See the abstract and the third and fourth paragraphs in Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See the conclusion (especially the last sentence.)

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In Section 2.3, we provide a detailed discussion of the adopted assumptions. Additionally, we present the complete proofs in Appendices A and C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Section 4, we provide a detailed description for the experimental datasets. In addition, we provide the datasets and codes in supplemental material to ensure easy reproduction of all reported results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the supplemental material for datasets and codes in a zip file to ensure easy reproduction of all reported results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 4 for the detailed description of simulating outcome and experimental details. In addition, we provide the supplemental material for datasets and codes in a zip file to ensure easy reproduction of all reported results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the variance for all experimental results in section 4, by replicating each experiment 50 times.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: All experimental results can be easily reproduced on a personal computer.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All experiments are conducted on publicly available datasets.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In the first paragraph of Section 1 (Introduction), we outline various potential applications of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: All experiments are conducted on publicly available datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In Section 4, we provide references for the datasets and the simulation setups of the data-generating process.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: In Section 4, we provide references for the datasets and the simulation setups of the data-generating process. In addition, we provide the supplemental material for datasets and codes in a zip file to ensure easy reproduction of all reported results.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We don't use a crowdsourcing service.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new way to balance multiple rewards with some long-term rewards potentially missing. It does so by using Pareto Policy Learning of optimizing each reward subject up to the tradeoff frontier. This can be more practical than simple linear weighting since the linear weighting strategy applies the constant weight regardless of the amount of conflict between pairs of objectives. Empirically, the papers show that the approach is superior to linear on two synthetic tasks with some real data. Overall I think the paper is promising and adding more realistic empirical evaluation can add values to the current state of the paper.\n\nSTRENGTHS:\n- Learning to combine multiple rewards is an important and well-motivated question, and has wide ranging implications.\n- The method proposed is mathematically sound. The paper shows theoretically that the input parameters can be interpreted as a form of worst case value on each objective. \n- The paper explains how the most popular approach of linear weighting can fall short, derives the method through first principles, and empirically demonstrates that the proposed method is superior.\n\nWEAKNESSES:\n- The main weakness of the paper is that the experimentation is rather limited. The experiment uses partial real data with synthetic generation of short-term and long-term rewards. For example, in robotic planning, the authors could show how their approach helps balance the long-term reward (e.g. goal reaching) / short-term reward (e.g. minimizing jerk). This is just an example, but including other more real-world planning and RL problems would seem beneficial.\n- It seems that compared to linear weighting, the proposed method seeks more short-term reward but is not necessarily better in terms of long-term reward. It may not be a weakness but reading the table does strike me that the method is more “short-sighted.”\n\nQUESTIONS:\n- The separation between short-term and long-term reward is practically meaningful, but mathematically the only difference is that one reward can be missing and the other is fully observable, since the Pareto Policy Learning treats all objectives the same. Do we necessarily need these separate definitions? Can we put everything as a long-term reward that can be missing sometimes?\n- How are the weighting and preference vectors chosen? Have the author considered running a sweep over different configurations to compare against linear weighting? Apologies if I overlook this detail from the paper.\n\nLIMITATIONS:\n- The limitation of the overall framework are mentioned but there is no much detail perhaps due to space constraint.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempts to address the challenge of learning the optimal policy for balancing multiple long-term and short-term rewards. The authors point out that the existing linear weighting method leads to a sub-optimal policy. To address this limitation, the authors propose formulating formulate the problem as a multi-objective optimization problem. They utilize the Lagrange algorithm to use preference vectors to solve the formulated multi-objective optimization problem and aim to learn the policy to meet Pareto optimization. In order to decide the preference vectors, the authors propose establishing the connection between the optimization problems and the ε-constraint problem. Experiments on IHDP and JOBS demonstrate the efficacy of the proposed method.\n\nSTRENGTHS:\n1.\tThe multi-object problem is practical in both reinforcement learning and other optimization scenarios. The paper provides a good summary of the limitations of the existing linear weighting method and introduces a novel perspective on solving the problem by resorting to the Lagrange algorithm and Pareto optimization.\n2.\tThe author has a solid mathematical foundation and is able to provide detailed mathematical descriptions and solutions to the proposed optimization problems.\n\nWEAKNESSES:\n1. The authors point out that the linear weighting method is suboptimal. However, there is no explanation in the method section or corresponding experiments to demonstrate that the proposed method (i.e. DPPL) is optimal.\n\n2. In line 38, the authors claim that when some of the rewards are interrelated, the linear weighting method can only achieve a suboptimal solution. The claim may not be rigorous as the linear weighting method might be able to model the relationship among the rewards. More explanation and experiments are required.\n\n3. In line 95, the definition of Pareto optimality, the condition for Pareto optimality by the author is to find the $\\theta$ that makes all $\\bar{\\mathcal{V}}$ optimal. However, is it possible that the $\\theta$ is not optimal for some $\\bar{\\mathcal{V}}$ but is optimal for the overall $\\bar{\\mathcal{V}}$?\n\n4. Some mathematical symbols and proprietary terms in the paper are not explained clearly. For example, what does the $e$ in line 110 mean? What does MOP represent? Does MOP represent multi-objective problems? What do $v$ and $R_{+}$ mean in line 171? What does the KKT condition mean? Is it the KKT condition in the Lagrange algorithm? What is the difference between the two descent directions $d_{rt}$ and $d_t$? There are many similar situations in the paper. I suggest providing necessary explanations for each noun and symbol that appears for the first time.\n\n5. In section Simulating Output and section Experimental Details, many parameters are defined by the authors themselves, but most of them do not have reasons or ablation experiments. For example, why is the number of preference vectors 10? In Line 253 to Line 254, why are some parameters truncated normal distributions and some Gaussian distributions?\n\n6. In Table 1 on the L-REWARDS metric, the proposed method is comparable to the linear weighting method. However, the authors claim that for most of the preference vectors, DPPL's solutions have better performance.\n\n7. In Figure 1, it seems that the effect on the $\\delta{w}$ from the missing rate and T is not obvious for either the proposed method or LW. More explanation is needed.\n\nQUESTIONS:\nPlease see the comments above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper studies the tradeoff between short-term and long-term rewards. The authors formulate the policy learning problem as a multi-objective optimization problem and propose a decomposition-based Pareto policy learning method. I only had experience in reinforcement learning in robotics five years ago. I tried my best to understand the paper, but I am not sure about my rating and comments.

STRENGTHS:
- This paper studies a quite interesting and important problem, and the proposed methods seem effective on these two benchmarks.
- The paper is well-organized, the division is relatively easy to follow, and the proposed method is well-motivated.

WEAKNESSES:
- Only the linear weighting method is used as the baseline. I am wondering if there are any other methods that can be used for comparison. If not, why? Since both IHDP and JOBS are widely used.

QUESTIONS:
Please see the weakness.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework for solving multi-objective optimization problems: multi-objective optimization problems are divided into sub-problems in different regions by setting different preference vectors. The parameter optimization direction of the sub-problem can be easily solved by transforming it into a dual problem through the KKT condition, and a Pareto optimal solution of the original problem can be obtained by solving the sub-problem. This paper uses this framework to balance the optimal strategy learning under multiple short-term rewards and long-term rewards and achieves better and more stable performance than the traditional linear weighted method in the constructed experimental environment.\n\nSTRENGTHS:\n1. This paper reveals in detail the connection between the proposed method and the linear weighted method and the epsilon-constrained optimization method. Based on this connection, the epsilon-constrained optimization method can provide interpretability for the method in this paper.\n2. The method in this paper theoretically overcomes the suboptimality problem of the linear weighted method and avoids the situation where the epsilon-constrained optimization method does not have a feasible solution.\n3. This paper obtains better and more stable results than the epsilon-constrained optimization method in the optimal strategy learning problem under multiple short-term rewards and long-term rewards constructed by the author.\n\nWEAKNESSES:\n1. This paper mainly proposes an important multi-objective optimization algorithm and compares it with two existing algorithms in theory. However, the title of this paper seems to be just a specific application scenario of the algorithm. In what other scenarios can this algorithm be applied?\n2. The experimental part is mainly conducted in a constructed environment, and it is unclear how difficult it is in the field of causal inference.\n3. The v in line 171 is missing \\bar. In Appendix B, t in line 5 of Algorithm 1 should start from 0.\n\nQUESTIONS:\n1. How to deal with a situation where long-term rewards are missing? Should the data be ignored when solving the network?\nI think the optimization method proposed in this paper and the connection with related methods are very interesting, and I will improve my score as appropriate.\n\nLIMITATIONS:\nHave addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}