Learning the Optimal Policy for Balancing Short-Term and Long-Term Rewards
Authors: Qinwei Yang
Xueqing Liu
Yan Zeng
Ruocheng Guo
Yang Liu
Peng Wu
Venue: NeurIPS 2024 poster (2024)
Keywords: Policy Learning
Short-Term and Long-Term Rewards
Causal Inference
Decision Making
Primary area: causal_inference
TL;DR: Learn the optimal policy that balances short-term and long-term rewards, especially in scenarios where the long-term outcomes are often missing due to data collection challenges over extended periods.

Abstract: Learning the optimal policy to balance multiple short-term and long-term rewards has extensive applications across various domains. Yet, there is a noticeable scarcity of research addressing policy learning strategies in this context. In this paper, we aim to learn the optimal policy capable of effectively balancing multiple short-term and long-term rewards, especially in scenarios where the long-term outcomes are often missing due to data collection challenges over extended periods. Towards this goal, the conventional linear weighting method, which aggregates multiple rewards into a single surrogate reward through weighted summation, can only achieve sub-optimal policies when multiple rewards are related. Motivated by this, we propose a novel decomposition-based policy learning (DPPL) method that converts the whole problem into subproblems. The DPPL method is capable of obtaining optimal policies even when multiple rewards are interrelated. Nevertheless, the DPPL method requires a set of preference vectors specified in advance, posing challenges in practical applications where selecting suitable preferences is non-trivial. To mitigate this, we further theoretically transform the optimization problem in DPPL into an $\varepsilon$-constraint problem, where $\varepsilon$ represents the minimum acceptable levels of other rewards while maximizing one reward. This transformation provides intuitive into the selection of preference vectors. Extensive experiments are conducted on the proposed method and the results validate the effectiveness of the method.

Learning the Optimal Policy for Balancing Short-Term and Long-Term Rewards
Qinwei Yang', Xueqing Liu', Yan Zeng', Ruocheng Guo?, Yang Liu?, Peng Wu BBijjing Technology and Business University BByteDance Research
SUC Santa Cruz
Abstract
Learning the optimal policy to balance multiple short-term and long-term rewards has extensive applications across various domains. Yet, there is noticeable scarcity research addressing policy learning strategies in this context. In this paper, we aim learn the optimal policy capable of effectively balancing multiple short-term and long-term rewards, especially in scenarios where the long-term outcomes are often missing due data collection challenges over extended periods. Towards this goal, the conventional linear weighting method, which aggregates multiple rewards single surrogate reward through weighted summation, can only achieve sub- optimal policies when multiple rewards are related. Motivated by this, we propose a novel decomposition- based policy learning (DPPL. method that converts the whole problem into subproblems. The DPPI. method is capable of obtaining optimal policies even when multiple rewards are interrelated. Nevertheless, the DPPL. method requires a set of preference vectors specified in advance, posing challenges in practical applications where selecting suitable preferences is non-trivial. To mitigate this, we further theoretically transform the optimization problem in DPPL into an s-constraint problem, where e represents the minimum acceptable levels other rewards while maximizing reward. This transformation provides intuitive into the selection preference vectors. Extensive experiments are conducted on the proposed method and the results validate the effectiveness of the method.
Introduction
Learning an optimal policy for balancing multiple short-term and long-term rewards holds extensive applications across various domains. For instance, content providers can optimize recommendations to avoid short-term clickbait strategies, ensuring sustained user engagement and revenue growth [[]. IT companies can design web pages catering to immediate user preferences while enhancing long- term engagement and satisfaction (2). Economists explore the effects of early childhood interventions on lifetime earnings, seeking optimal policies (e.g., class size) maximizing short-term test scores and long-tern earnings simultancously (3). Policymakers improve job training program design, considering both immediate income impacts and subsequent employment status improvements [4] 5] Medical practitioners outcomes in chronic diseases like Alzheimer's ' and AIDS 6). Marketing professionals can optimize incentive strategies to positively influence customer behavior in both short and long terms [7].
refine drug prescriptions, considering short-term alleviation and long-term
Despite the importance of balancing multiple short-term and long-term rewards, policy learning methods in this area remain largely unexplored. Recent literature [8] employs linear weighting method to achieve this goal. It combines multiple rewards into a single surrogate reward by weighted summation, which is optimized to learn the optimal policy. However, this strategy has several limitations. First, can only find optimal solutions in convex regions of objective space and cannot obtain the optimal solutions in non-convex regions [9], Second, it achieves the optimal solution only
*Corresponding author: Langwu@btbu.cdee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

when the rewards are independent of each other. When some of the rewards are interrelated, it can only achieve sub-optimal solutions [T]. Consequently, although the linear weighting method is easy to implement, the optimality of its solution cannot be guaranteed when balancing multiple objectives.
In this article, propose a principled policy learning approach for balancing multiple long-term and short-term rewards (objectives). Specifically, we first formulate it as multiplc-ojjective problem (MOP) and aim seek the Pareto optimal solutions (policies). A solution is Pareto optimal if improving one objective necessitates worsening other objectives. Then, we propose novel decomposition- based policy learning (DPPL) method, which involves (1) introducing a set of preference vectors, (2) dividing the whole optimization problem into several subproblems based on the preference vectors, and (3) ultimately achieving different Pareto solutions for the objectives by solving these subproblems. Compared with the linear weighting method, it can obtain Pareto optimal solutions in non-convex regions and applicable to cases where multiple objectives are interrelated.
While the proposed DPPL. method can find Pareto optimal policies, it necessitates specifying a set of preference vectors advance. In practical applications, decision-makers may encounter the challenge of determining which preference vector to choose. To mitigate this concern, we further theoretically transform the optimization problem in DPPL into s-constraint problem. This transformation can assist decision-makers in better understanding and selecting preference vectors.
The contributions of this paper summarized follows.
We formulate the policy learning problem of balancing multiple long-term and short-term rewards as multi-objective optimization problem and propose deccomposition-baseed Pareto policy learning (DPPL. method to obtain set of Pareto optimal policies.
We theoretically establish the connection between the DPPL method and the e-constraint problem, offering an intuitive interpretation of preference vectors and guiding their selection.
We conduct extensive experiments to demonstrate the effectiveness of the proposed method.
2
Problem Formulation
Throughout, we employ bold letters for vectors, uppercase letters for random variables, and lowercase letters for their realization values.
2.1 Notation
We introduce notations to delineate short-term and long-term causal effects. Let denote the binary treatment indicator, where A 1 represents the treated group and A represents the control group. X represents the features observed, S = (S1, ..., Sy) E I and Y = (Yı, ... Y,) IR' represent the vector of short-term and long-term outcomes, respectively. Both short-term and long-tew ter-termm outcomes are observed after the treatment A, and associations among them may exist.
Utilizing the potential outcome framework [II] we denote S(a) = (c)...... (a)) and Y()) (a), .... (a)) for a = 0, as the potential short-term and long-term outcomes under treatment A = respectively. We assume that larger short-term and long-term outcomes are preferable. The observed short-term and long-term outcomes S and Y correspond to the potential outcomes of the actual treatment, that is, S ((A) and Y (A).
In real-world applications, long-term outcomes often suffer from missing due prolonged follow-up periods and budget constraints. In contrast, collecting short-term outcomes is more manageable. Therefore, we presume that all short-term outcomes S are observable, while long-term outcomes Y may be subject to missing. Let R = (R..... R,) E {0. 1).' denote the indicator for observing the long-term outcome Y. where R, indicates that Y, observed and R, indicates that Y, is missing. The missingness of Y would lead to identifiability and estimation problems [2 23
2.2 Formulation
In this article, we aim to learn the Pareto optimal policy for balancing multiple correlated short- term and long-term rewards, which has a wide range of application scenarios [ 24]. Let X + {0, 1} be policy that maps from the individual context X to the treatment space

---

{0.1}. For given policy T0) = オ (X, 0) parameterized by 0. the policy values for the i-th short-term outcome S; and the j-th long-term outcome Y, are defined as,
V(O;s:) s.) (1), + (1 (0))
1,.., j = 1....,,
y(0 yy) = (1) + - (1-0)))+++1030000000000 (0))
which are the i-th short-term reward and the j-the long-term reward induced by the policy π x()).
Conventionally, we convert maximization problems to minimization problems. Let V(0; s;) = -V st), V(O; my) (0; 8j). The trade-off among multiple correlated long-ten and short-term rewards can be formulated as multi-objectivy optimization (MOP) problem given by
min ((0) = (V(0;s)), V(O: (1); (00 ... V(0:
_ (O) (2) (o)(
where M - 1 and the symbol _ means 'denoted as'. Generally, there is single solution that can simultaneously optimize all objectives in problem (T) and thus we resort to the Pareto optimality, This concept is employed define the optimal solutions for the MOP problem.
Definition 1. (Pareto optimality)
(a) Pureto dominance. For two points 02,02. 02 dominates 02 if and only if Vm. (02) S V.n. (0²), m E (1.... and V.n (0²) < Vnn m' E (1,...,M, M}
(b) Pareto optimality. 0" is a Pareto optimal point if there is no other solution i that dominates 0"
Pareto optimality refers to a condition where improving one objective comes at the expense of worsening other objectives. The collection of Pareto optimal solutions is called the Pareto set. Our goal to derive the set of Pareto optimal solutions (or Pareto optimal policies), each of them providing distinet optimal trade-off among all objectives.
2.3 Identification and Estimation of Short-term and Long-term Rewards
The long-term and short-term rewards are causal parameters that cannot be identified without imposing causal assumptions 27) Therefore, before seeking the Pareto optimal solutions for balancing multiple long-term and short-term rewards, it is necessary to consider the identification and estimation of long-term and short-term rewards. The proposed method is based Assumptions I Assumption 1 (Strong Ignorability).
and 2 below.
(a) (S) {(a)) (b) e(x) A P = X =:) for all æc
X for 0,1;
Assumption aa) suggests that, given the feature X, treatment assignment A is independent of
the potential outcomes S(a) and Y (a). This implies that confounding bias between the treatment
A and the short/long-term outcomes (S(a), (a)) can be eliminated by conditioning on X (28). Assumption bb) ensures that for the subpopulation of X æ, units with both A and A
exist. These assumptions are widely used in causal inference [1] 27 35
In addition to confounding bias, also need address the selection bias induced by the missingness of long-term outcomes g Thus, we further invoke the Assumption [
Assumption (Missing Mechanism Long-term Outcome). For = 0, and j I,...,.. J,
(a) Rg T1 (a) X, (a), = a;
(b) 0 < r 8) P(R; = X = σ, A a, -8.
Assumption a) can be reformulated as R, Y, (X,S, A), which means that R, relies only on the observed variables (X, A, S). This assumption also ensures that P(Y, = y| S. S. R, = P(, = y/ X, S, A, R2 = 0). This implies that we can utilize the available data to draw conclusions about the missing long-term outcome. Assumption b)) assumes that the long-term outcome for each unit has a non-zero probability of being observed. Assumptions and ensures the identifiability of (0; sæ) and (0; y,), shown in Lemma
3

---

Lemma (Identifiability of Short-term and Long-term Rewards). For = 1,... ..... I and j = -......J
(a) under Assumptions
the i-th short-term reward V(O; 84) is identifiable /
(b) under Assumptions the j-th long-term reward V(O: yj) is identifiabl.
When we have access to only one short-term outcome and one long-term outcome, Lemma 1 rrducces to the identifiability result presented in [8) In this article, our focus is on achieving the Pareto optimal policy for multiple short-term and long-term rewards. Therefore, for the estimation of V(0; 8,) and V(0: y, we defer it to Appendix A
3 Pareto Policy Learning for Balancing Short-Term and Long-Term Rewards
In this section, aim to learn Pareto optimal policies for the MOP problem (I). Section 3.1 gives the motivation for this work and Section 3.2 introduces the proposed policy learning approach. In Section 3.3] we theoretically establishe the connection between the linear weighting method, the MOP problem for a given preference vector, and the e-constraint problem. This connection offers an intuitive interpretation and guides practitioners in selecting the preference vector.
3.1 Motivation
For seeking the optimal policy for balancing short-term and long-term rewards, previous work adopted the linear weighting method. Specifically, the authors formulate the goal as
min (00)
Σ Wm (0),
m==1
where wmm is the pre-specified weight for the m-th objective. The objective function in the optimization
problem (2) is merely linear combination of multiple objectives from the MOP problem (I. Due to its intuitiveness and simplicity, the traditional linear weighting method is commonly used for solving MOP or multi-task learning problems 36
The linear weighting method simply combines multiple objectives into single surrogate objective through weighted summation. While simple, it has several limitations. First, the optimal solution is found only in convex regions and not in non-convex regions [9]. Second, an optimal solution can only be achieved if the objectives are independent of each other. That is, if some objectives are interrelated, only suboptimal solution can be obtained [10]. Thus, it does not guarantee the superiority of the solution or its solution may deviate from the Pareto optimal solution.
To overcome the limitations of the linear weighting method in [ we first introduce a decomposition- based multi-objective optimization algorithm to achieve the Pareto optimal policy. However, this algorithm relies on pre-specified preference vectors, which are used to express a decision maker's s degree of preference for multiple conflicting objectives. In practice, the explanation and selection of preference vectors is a challenging problem. To further tackle this issue, we establish a theoretical relationship between preference vectors and the e-constraint method (39]. This relationship provides a clear interpretation on preference vectors, assisting in selecting more suitable ones.
3.2 Pareto Policy Learning for the MOP Problem
We introduce the decomposition- based Pareto policy learning (DPPL.) method, which can generate the Pareto set containing policies that are optimum from a trade-off perspective. The main idea the DPPL method is to first decompose the original MOP problem into several constrained subproblems based on predefined set of preference vectors, and then obtain a set of Pareto optimal policies by solving these subproblems in parallel [40]
For obtaining the Pareto optimal policy for balancing M short-term and long-term objectives, first, we are given set of K preference vectors {#t U2 .... ux in R. Each element of a preference vector specifies the importance of the corresponding short-term or long-term reward. For each preference vector us the corresponding subproblem is given as
min ((0) (Vi (0). (M)
0
(3)
s.t. GV (0) (ug-
u V 0. AN = 1,..,,

---

where (x, I O means that objective space of the subproblem is restricted in the subregion Se which is defined by S2g = {2 E R 1u v I u v. A k' I.... K}. Geometrically speaking, Oz represents the set of v that forms the smallest acute angle with ugs which means that the optimal solution of the subproblem can he obtained by only searching the subregion. The preference vectors divide the objective space into different subregions.
Solving the subproblem 3 involves the following two steps:
Step (a). Find reasonable initial solution fg. Specifically, we first randomly generate solution 0, in the full decision space" and then iteratively update it with the rule rr,. = 0r, + n, where ty the step size. For a given 0,,, the descent direction d,, is updated by solving 4
(dr, (r.)) = arg min a + =|ld|? s.l. VG (0.,) S a, K ((,,).
dER" aER
where I(e-.) = G (0,) 0. = =1.... K} is index set of all activated constraints, which means ((,,,) not S4. The problem aims to find the descent direction d,, for each iteration t and then obtain the initial solution 0g such that ((O)) in 92g
Step (b). Solving the subproblem
(3). The descent direction d, for the t-th iteration is obtained by
(de, a4)) = aag min a ||dl dER" .EER
s.t. VVm (O.) S a, m = 10,m=1,,.... L,.,,
(5)
V9 (0,) < a, K (o,),,,
where (0) (0) 2 -e}, and the threshold
slack variable used deal with the solutions near the constraint boundary. We further transform it into a dual problem which will greatly reduce the dimension of decision space. Based on the KKT conditions, we have
d. -(EM Am VVm + EMEZ, BR (O.)). Therefore, the dual problem is given as
M
max Am P92
Am (.m (0:) +
Σ
By (0))11
m=1
VEL(
(6)
s.t.
Am +
Σ
Ber = 1, Am 2 0, Be 2 0, Vm I,...,
I,..,,,,k. M.W (.))..
(EL.(0)
where Am 2 0 and 3R 2 O are the Lagrange multipliers for the linear inequality constraints.
Step (a) is to find an initial solution O, that is restricted in a subregion of the subproblem (3) and once a feasible solution is found or predetermined number of iterations is reached, the step stops. For an given initial solution Oo. Step (b) is to find the optimal solution O" for the subproblem 33) We summarize the proposed policy learning approach in Appendix B
Lemma ((41). Let (di, ax) be the solution to the t-th iteration of problem
(a) If 0, is Pareto optimal restricted on S2g, then d, = 0 E R. and a₁ 0. (b) If 0g is not Pareto optimal restricted 94. then
0t S |1/ <0, VVn (0.) d, < or, m = 1,.., M
(7)
VG. (0.) S ot,k' (0.).
Lemma 2 a) implies that at the t-th iteration, no direction (d: 0) can simultancously improve the performance for all objectives, confirming that the solution 6, satisfies Pareto optimality. Lemma 2/b) suggests that if 0, does not meet Pareto optimality, then the descent direction d, * serves as the descent direction for all objectives, such that the solution of the next iteration is closer to the Pareto optimal solution. Thus, Lemma demonstrates that we always attain Pareto optimal solutions for each subproblem using the update rule θ 0, + n. By solving all subproblems, we can acquire diverse set of Pareto optimal solutions (or policies) confined to different subregions, even when the multiple objectives are correlated.
2 (0) is the objective vector, and the space spanned by the objective vectors is called the objective space Ω. Thhe parameter vector 0 represents the decision variable and the space spanned it is called the decision space.
5

---

3.3 Deep Analysis of the Preference Vector
The DPPL method in Section 3.2 requires a set of pre-specified preference vectors, posing challenges in practical applications where selecting suitable preference vectors is non-trivial. To mitigate this problem, we provide practical method for decision-maken by theoretically establishing the connection between the DPPI. method and the the e-constraint problem.
select appropriate preference vectors
We first give a brief introduction the e-constraint problem 100, which is defined as follows,
min Vi(O), Vm (0) S Em for all m = l..... M, m + L.
(8)
0
where em is pre-specified threshold. Compared to the MOP problem T and the linear weighting objective 2 notable advantage of the e-constraint problem is its interpretation on the threshold Em, which represents the maximum acceptable value (i.e., the acceptable worst-case scenario) for the m-th objective. In contrast, the weights and the preference vectors problems (T) and 2 are not straightforward for relating the resulting values of objectives. Thus, if we can establish the connection between e = (e1,...,& &M) ) and the preference vector U4 then we can provide powerful guidance for choosing appropriate preference vectors.
Theorem 1. For the preference vector un = (ukl,-... UKM) in problem ① the weights w = (WI,,-. ..... WM) in problem 2 and the thresholds in problem (8. the following statements hold:
(a) the connection between E and w is given as
Em (X( +
Wm
Tm (X) 0) Tm (X) + hm (X)), for = 1... M, and m + 1, (9)
WI
where Tm () the conditional averuge causul effects for mith short/long-temm outcome,
B1S:(1) - 5:(0) X|, if wm is the weight of V(0, s:); (1) X]] if Wm is the weight of ((0, ((0,,)) 9z),
Tm
u(()) is the indicator function, and
[0], if wm is the weight of V(0,s)); [0) S, R2 1}, if is the weight of (00 U5)-
hmm (X)
(b) the connection between w and uh is given
Wm
>m +
Σ
(u kxm), for m 1,.... M,,
(10)
(ET..
where Am and Bw are defined in Eq. 66. and (0) (K' Gv 2 -c} defined in Eq.
Theorem I (see Appendix C for proofs) establishes a link between the preference vector uy and through w scenarios involving multiple long-term and short-term objectives. Specifically, Theorem T[a) shows how to estimate the threshold e for given weights w, and Theorem [b) shows how to
assign weights w via preference vectors u, This means that for the subproblem determined by preference vectors uk. we can ascertain the maximum acceptable threshold g based on Theorem [
thereby offering intuitive interpretation of the preference vector ug.
There are several practical implications with Theorem On one hand, it assists decision-makers in better understanding and selecting preference vectors in practical applications. In practice, we can initially pre-specify a set of preference vectors {u,, u2, .... Ug] } in IR then derive the weights corresponding each preference vector uy through Eq. (LO), and finally substitute the obtained weight w into Eq. (9) to calculate the threshold e. Leveraging the intuitive interpretability of the threshold mecision-makers can select the appropriate preference vectors according to their specific requirements. On the other hand, it also provides guidance for specifying e in the e-constraint problem (8). Inappropriate selection of for this problem may result in an empty feasible region, yielding empty solutions. By utilizing a set preference vectors, we can efficiently screen out some reasonable choices of and reduce the cumbersome trial-and-error process of testing different
In conclusion, by establishing the connection between the DPPI. method and the e-cosstraint problem, we can harness the advantages of both methods while mitigating their respective weaknesses.

---

4 Experiments
Datasets. Following the previous studies 8 we two widely used datasets: IHDP and JOBS, for evaluating the performance of the proposed method. The IHDP dataset explores the effectiveness of high-quality home visiting in promoting children's future cognitive development and covers a sample of 747 units, including 139 treated and 608 controlled. In addition, the dataset has 25 characteristics that provide comprehensive picture of the children and their mothers. The second dataset, JOBS, explores the effects of job training on income and employment status. It consists of 2.570 units (237 treated, 2.333 controlled), with 17 covariates. Note that each unit in both datasets has only one observed outcome from single treatment, and neither dataset collects long-term outcomes.
Simulating Outcome. Consider the case of one long-term reward and one short-term reward. Following the previous data-generation mechanisms 42, for the n-th unit (n = =,..., N), we simulate the potential short-term outcomes 5(0) and 5(1) as follows:
S. (0) Bern(o + c0.n)), (1)
Bern(o(w; + +1,n),
where (+)) is the sigmoid function, wo NV 1) follows truncated normal distribution, W1 Unif(-1. 1) follows uniform distribution, co.n ~ NNoa, σo) and 61,n ~ ~N 01). We set Hiy = 1, μ1 = 3 and ou = 01 = for the IHDP dataset, and we set P4o = 00, μ1 = 2 and dg = 01 = 1 for the JOBS dataset. For generating long-term potential outcomes Y() and Y(1), we introduce
the time step 1: we set the initial value at time step O as: Yo.a = S. (0). Yo,, (1) (1), then generate Y,, (0), Yt,0 (1) according to the following equation and we eventually regard the outcome
at the last time step T as the long-term outcome, Y, (0) = Yr,, (0), (1) = Tr,(1)))
Y (0) ~ ((
Ye (0))+ (1) ~ (ern(o X,, + Σ
(0))+
P==0
where fg is randomly sampled from 1,2,3,43 4) with probabilities {0.5, 0.2, 0.15 0.1,0.01,.. 0.05), 31 ~ 4 1, (0, 1), and C = 1/T is a scaling factor. For co.- and G1,,, .. set (4 = 141 0, du 1 and 0 = 3 for the IHDP dataset and set /Ao- = # = 0, oo = and 0 = 1 for the JOBS dataset.
Assumption 2 shows that observing indicator R depends on the feature X, the treatment A, and short- term outcome S. For a given missing rate r, we select the missing indexes for Y and derive the missing indicator R according to the following criterion: calculate the m, = 1/ d== E/ (Xnd + sn), n = 1.... NN, and choose the index of the row with the smallest rN values in {m,, n = 1,... N) as the missing indexes. D is the feature dimension and N is the sample size.
Experimental Details. In this paper, preference vectors are used to quantify individual's 's preference for different objectives in the multi-objective optimization problem. For the case of two-objective, we randomly generate 10 unit preference vectors (u1, u2. U10), where un (wy1, Wk2), uk1 cos(tx), ú2g2 = sin(ts), ), t& E (0, 1), which implics that the Lg-norm of the preference vectors is ensuring the consistency and comparability of the preference measures. U41 and u&2 the preferences for the short-term objective and the long-term objective, respectively. Each component of the preference vector Us represents the strength or importance of the decision maker's preference for different objectives. Preference vectors are used as weights in the linear weighting method, whereas our method uses them to divide the original problem [ into several subproblems.
Evaluation Metrics. We measure the performance of our proposed method by three metrics: long and short-term rewards, the variance of long and short-term rewards, and the change in welfare. Formally, the short-term reward of the learned policy #(X, 0) is (0: = M [+ (1), + (1 (X,,, S,, (0)), the long-term reward is (0; y) = Σ [=(X,, Similar as 42 43,, the welfare changes are defined AW, & I (1, -
()),,))) (,))) *(,,, 0)]
for the short-term reward, AW, y = Σ> (1) - Y, (0)) (X,,, 0)] for the long-term reward, AW = 0.5 & + 0.5AW, for the overall balanced-bai reward. Among these metrics, AW is the most critical here, as it directly measures the balance reward achieved by the learned policy.
Policy learning with short-term and short-term reward. We choose MLP as the policy model (0), and we average over 50 independent trials of policy learning with the short-term and long-term reward in IHDP and JOBS. We fix the missing ratio 0.2 and the time step T = 4. We measure the uncertainty of the model by calculating the variance of the long and short-term reward over 50 experiments, and a smaller variance means more stable model performance.
7

---

Performance Comparison. From our previous analyses, the linear weighting method generally achieves the sub-optimal policies. The proposed DPPL method can generate policies. First, for the long-term reward, the short-term reward, and AW, it is not surprising to observe that for most of the preference vectors, DPPL's solutions have better performance. Second, for the variance, our method performs more stable in 50 experiments. Because we will divide the original problem into several subproblems according to preference vectors, and then solve the subproblems in relatively small subregion to obtain the Pareto optimal solution, whereas the linear weighting method searches the entire objective space. The associated results are displayed in Table More experimental results with missing ratio 0.3 given in Appendix D
Pareto optimal
Table 1: Comparison of our method (OURS) and linear weighting method (LW) on IHDP and JOBS, with Short-Term -erm Reward (S SSSREWARRDS and Long- Term Reward (L-REWARDS), AW and Variance (SR and L- VAR) as evaluation metries. The best result bolded.
IHDP
S-REWARDS
L-REWARDS
AW
S-VAR
L-VAR
PREFERENCE VECTOR
OURS
1.W
OURS
L.W
OURS
LW
OURS
LW
OURS
LW
(1.00, 0.00)
522.840
520.860
386.221
383.950
39.432
37.307
14.573
12.841
52.326
56,093
(0.98, 0.1
$21.820
524.660
382.774
387.102
37.199
40.782
13.275
11.079
54.181
59.895
(0.94, 0.34)
$23.000
521.840
372.418
394.386
32.610
43.014
11.588
13.578
50.138
62.584
521.060
382.419
36.641
11.512
(0.76, 0.64)
523.620
519.840
391.296
390.415
42.360
40.028
12.729
15.594
55. 883
48.308
6 (0.64, 0.76)
521.460
517.420
387.015
390.206
39.139
38.714
14.217
15.479
46.342
56,219
(0.50, 0.87)
523.800
514.480
383.424
381 331
38.514
802
12.797
18.758
55.118
56.
00.34, 0.94)
521.360
516.800
373.307
400.510
32.235
43.556
11.701
18.618
50,002
60.847
0.98)
522.240
397.640
396.214
42.842
40.529
12.913
10 (0.00, 1.00)
$23.600
516.780
387.933
390.387
40.668
38.485
11.531
21.079
60.714
54.246
JOBS
S-REWARDS
L-REWARDS
AW
S-VAR
L.
PREFERENCE VECTOR
OURS
1.W
OURS
I.W
OURS
LW
OLRS
LW
OURS
LW
(1.00, 0.00)
1613.140
1612.340
1230.918
1223.416
159.381
155.233
$4.724
56.502
84.846
88.008
(0,98, 0.17)
1618.400
1607.680
1219.517
1223.072
156.310
152.728
54.521
65.927
85.566
86,622
(0,94, 0.34)
1614,800
1220.316
1223.813
154,910
61.466
4 (0.86. 0.50)
1612,880
1598. 880
1217.305
1225.574
152.444
149.579
59.510
75.431
90.855
79.728
(0.77, 0.64)
1613.160
1602.320
1233.604
1227.886
160.734
152.455
$9.042
77.481
85.533
85.854
(0.64, 0.76)
1612.380
1595, 9695.96000
1218.100
1219.996
152.592
145.333
58.028
82. 03.
96.
94.424
(0.50, 0.86)
1608.860
1596. 280
1224,763
1230.471
154.163
150.727
58.706
06.083
89.392
92. 135
0.94)
1613.600
1595.720
1232.958
1217.118
57.000
9 (0. 17, 0.98)
1614.840
1596. 320
1225.607 607
1224.383
157.575
147.703
58.278
84 221
99. 329
82.437
(0.00. 1.00)
1610.380
1588.400
1228.679
1223.119
156.882
143.112
59,285
88.393
95.054
85,443
Sensitivity Analysis. We perform the sensitivity analysis of missing ratio and time step T' on JOBS.
Our method achieves better performance all missing rates r = [0.2, 0.3, 0.4, 0.5] with T 4, and 0.2 with time step T [4, 6, 8, 10). Our method stably outperforms the linear weighting method under varying r and T, even in scenarios with a high missing ratio or a large time step. This further illustrates the effectiveness of our method. The associated results are displayed in Figure
Insrniptt
254
(a) r= 0.2 on JOBS
(b) r 0.3 on JOBS
(c) r 0.4 on JOBS
(d) 0.5 on JOBS
T=) JOBS
T= JOBS
(g) T= JOBS
(h) T 10 JOBS
Figure 1: Comparison of two methods with different missing ratios {0.2, 0.3, 0.4, 0.5) on JOBS
Interpretation on preference vectors. By Theorem for the set of pre-specified prefer- ence vectors (M,, u2. ... u10), we transform the optimization subproblem corresponding to each preference vector into the e-constraint problem as ming V(O))) s. I <(< 0) or
8

---

maxe V(O;y), s.t. 2 -E and the threshold -E are shown in Table This value of -E is the minimum value of the short-term reward that the decision maker can accept while maximizing the long-term reward. Our results show that as the second component of the preference vector increases, the value of - shows a decreasing trend. In essence, this signifies that a decision-make maker who emphasizes the long-term reward must necessarily loosen constraints on the short-term reward. In practice, decision makers can determine the threshold based on their specific needs for the short-term reward, and then select the most appropriate preference vector from the set of pre-specify preference vectors with the help of the intuitive interpretability of the threshold according to Table 2 More experimental results with different missing ratios (0.3, 0.4, 0.5) are provided in Appendix
Table 2: The values correspond to each preference vector in IHDP and JOBS datasets, where T = 4 and r = 0.2, obtained according to Theorem
Preference Vector
-c IHDP
- JOBS
Preference Vector
on IHDP
on JOBS
(1.02, 0,00)
0,820
0.878
(0,00. 1.00)
0.522
0,737
(0.98. 0.17)
0.827
0.875
(0.17. 0.98)
0.522
0.716
(0.94, 0.34)
0.826
0.868
(0.34, 0.94)
0.511
0,704
(0.86. 0.50)
0.833
0.868
(0.50, 0.86)
0.557
0.746
(0.77, 0.64)
0.741
0.865
(0.64, 0.76)
0.659
0.808
5 Related Work
Estimation of long-term causal effects. Assessing long-term causal effects is challenging due to the delayed long-term outcomes, posing significant difficulties in both identification and estimation. Recently, there has been increasing interest in using short-term surrogates to identify and estimate long-term causal effects, such as 4 5 13 45). In contrast to these previous works focusing on long-term causal effects, this paper aims to balance multiple short-term and long-term causal effects.
Trustworthy policy learning. Trustworthy policy learning ensures that the learned policies or models are reliable and dependable for practical applications. Traditional policy learning aims to identify individuals who would maximize the utility function based on their features if treated [46]. Recently, trustworthy policy learning has focused on ensuring that the learned policy adheres to principles such as beneficence. non- maleficence, autonomy, justice, no-harm and explicability 42 49 Various counterfactual based metrics have been suggested assess a policy's trustworthiness 42 50 In
this paper, we complement this series of work by developing principled policy learning approach that can effectively balance multiple rewards.
Multi-objective optimization (MOP). MOP aims to find compromises or trade-off's among multiple possibly contrasting objectives. It is widely used in the field of machine learning such as multi-task neural architecture search [55], and multi-objective reinforcement learning [56]-
learning 40 54
58 We extend these works to a new setting by learning the optimal policy for balancing multiple long-term and short-term rewards. Additionally, we provide practical method for interpreting and selecting preference vectors with theoretical guarantees.
Conclusion
In this paper, we focus on learning the optimal policy for balancing multiple long-term and short- term rewards. We reveal the limitations of the previous linear weighting method, which usually results in sub-optimal policies in practice. To address these limitations, we formulate the policy learning problem as multi-objective optimization problem and then propose the novel DPPL method to learn optimal policies. The DPPL method obtains set of Pareto optimal policies by solving a series of subproblems based on pre-specified preference vectors, effectively balancing multiple objectives. Furthermore, we theoretically establish the connection between the optimization subproblems the DPPL method and the e-constraint problem. This connection aids in better understanding and selecting preference vectors. We conducted extensive experiments on two benchmark datasets which validate the effectiveness of our proposed method. A limitation of this work is that it focuses on discrete treatments in identification and estimation (Section 2.3). In some application scenarios, continuous treatments (e.g., price) are of interest. Further investigation required to extend the proposed method to accommodate such cases.

---

Acknowledgements
Qinwei Yang, Xueqing Liu, Yan Zeng. and Peng Wu were supported by the National Natural Science Foundation of China (No. 12301370, 62473009), the funding from the Beijing Municipal Education Commission for the Emerging Interdisciplinary Platform for Digital Business at Beijing Technology and Business University, the Beijing Key Laboratory of Applied Statistics and Digital Regulation, and the gift funding from ByteDance Research.
References
[1] Lu Cheng. Ruocheng Guo, and Huan Liu. Long-tern effect estimation with surrogate repre- sentation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, page 274-282, 2021.
[2] Henning Hohnhold, Deirdre Oriri,, and Diane Tang. Focusing on the long-term: It's good for users and business. In Proceedings of the 21th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 1849-1858, 2015.
[3] Raj Chetty, John N. Friedman, Nathaniel Hilger, Emmanuel Saez, Diane Whitmore Schanzen- bach, and Danny Yagan. How does your kindergarten classroom affect your earnings? evidence from project star. The Quarterly Journal of Economics, 126:1593-1666 2007.
[4] Susan Athey, Raj Chetty, Guido W Imbens, and Hyunseung Kang. The surrogate index: Combining short-term proxies to estimate long-term treatment effects more rapidly and precisely. Technical report, National Bureau of Economic Research, 2019.
[5] Susan Athey, Raj Chetty, and Guido Imbens. Combining experimental and observational data to estimate treatment effects on long term outcomes. arXiv preprint arXiv:2006.090 096706 2020.
[6] Wenjie Hu, Xiao-Hua Zhou, and Peng Wu. Identification and estimation of treatment effects on long-term outcomes in clinical trials with external observational data. Statistica Sinica, 2023.
[7] Jeremy Yang, Dean Eckles, Paramveer Dhillon, and Sinan Aral. Targeting for long-term outcomes, Management Science, 2023.
[8] Peng Wu, Ziyu Shen, Feng Xie, Zhongyao Wang, Chunchen Liu, and Yan Zeng. Policy learning for balancing short-term and Long-term rewards. In ICML, 2024.
[9] Yair Censor. Pareto optimality in multiobjective problems. Applied Mathematics and Optimiza- tion, 4(1):41-59, 1977.
[10] Jürgen Branke. Multiohjective optimization: Interactive and evolutionary approaches, volume 5252. Springer Science & Business Media, 2008
[11] G. W. Imbens and D. B. Rubin. Causal Inference For Statistics Social and Biomedical Science. Cambridge University Press, 2015.
[12] Susan Athey, Raj Chetty, Guido Imbens, and Hyunseung Kang. The surrogate index: Combining short-term proxies estimate long-term treatment effects more rapidly and precisely. Working paper, National Bureau of Economic Research, 2019.
[13] Nathan Kallus and Xiaojie Mao. On the role of surrogates in the efficient estimation of treatment effects with limited outcome data. arXiv preprint arXiv:2003. 12408, 2020.
[14] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, Peng Wu, and Peng Cui. Propensity matters: Measuring and enhancing balancing for recommendation. In 2023.
[15] Peng Wu. Shanshan Luo, and Zhi Geng. On the comparative analysis of average treatment effects estimation via data combination. arXiv preprint arXiv:2311.000 005028888.. 2024.
[16] Sihao Ding, Peng Wu, Fuli Feng, Xiangnan He, Yitong Wang, Yong Liao, and Yongdong Zhang. Addressing unmeasured confounder for recommendation with sensitivity analysis. In KDD, 2022.
10

---

[17] Haoxuan Li, Quanyu Dai, Yuru Li. Yan Lyu, Zhenhua Dong. Xiao-Hua Zhou, and Peng Wu. Multiple robust learning for recommendation. In AAAI, 2023.
[18] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, and Peng Wu. Balancing unobserved confound- ing with few unbiased ratings in debiased recommendations, In WWW, 2023.
[19] Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong, Xiao-Hua Zhou, Rui Zhang, Xiuqiang He, Rui Zhang, and Jie Sun. A generalized doubly robust learning framework for debiasing post-click conversion rate prediction. In KDD, 2022.
[20] Peng Wu, Haoxuan Li. Yuhao Deng, Wenjie Hu, Quanyu Dai, Zhenhua Dong, Jie Sun, Rui Zhang, and Xiao-Hua Zhou, On the opportunity of causal learning in recommendation systems: Foundation, estimation, prediction and challenges. In IJCAI, 2022.
[21] Haoxuan Li, Yan Lyu, Chunyuan Zheng, and Peng Wu. TDR-CL: Targeted doubly robust collaborative learning for debiased recommendations, In ICLR, 2023.
[22] Haoxuan Li, Chunyuan Zheng, and Peng Wu. Stabledr: Stabilized doubly robust learning for recommendation data missing not at random. In ICLR, 2023.
[23] Haoxuan Li, Chunyuan Zheng, Sihao Ding, Peng Wu, Zhi Geng, Fuli Feng, and Xiangnan He. Be aware of the neighborhood effect: Modeling selection bias under interference for recommendation. ICLR, 2024.
[24] Guido Imbens, Nathan Kallus, Xiaojie Mao, and Yuhao Wang. Long-term causal inference under persistent confounding via data combination. arXiv preprint arXiv:2202.07234, 2022.
[25] Judea Pearl, Madelyn Glymour, and Nicholas P. Jewell. Causal Inference in Statistics: A Primer. John Wiley & Sons, 2016.
[26] Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect. Hachette Book Group, 2018.
[27] M.A. Hernán and J. M. Robins. Causal Inference: What If. Boca Raton: Chapman and Hall/CRC, 2020.
[28] Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55 1983.
[29] Kosuke Imai and Mare Ratkovic, Covariate balancing propensity score. Journal of the Royal Statistical Society (Series B), 76(1):243-263, 2014.
[30] Elizabeth A. Stuart. Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1):1-21, 2010.
[31] Paul R. Rosenbaum. Design of Observational Studies. Springer Nature Switzerland AG, second edition, 2020.
[32] Peng Wu, Xinyi Xu, Xingwei Tong, Qing Jiang, and Bo Lu. Semiparametric estimation for average causal effects using propensity score-based spline. Journal of Statistical Planning and Inference, 212:153-168; 2021.
[33] Peng Wu, Zhiqiang Tan, Wenjie Hu, and Xiao-Hua Zhou. Model-assisted inference for covariate- specific treatment effects with high-dimensional data. Statistica Sinica, 34:459-479, 2024.
[34] Peng Wu, Shasha Han, Xingwei Tong, and Runze Li. Propensity score regression for causal inference with treatment heterogeneity. Statistica Sinica, 34:747-769, 2024.
[35] Peng Wu, Peng Ding, Zhi Geng, and Yue Li. Quantifying individual risk for binary outcome: Bounds and inference. arXiv:2402.105: 2024.
[36] Kalyanmoy Deb, Karthik Sindhya, and Jussi Hakanen. Multi-objective optimization. In Decision sciences, pages 161-200. CRC Press, 2016.

---

[37] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering. 34(12):5586-550 5609, 2022.
[38] Haoxuan Li, Chunyuan Zheng, Yanghao Xiao, Hao Wang, Fuli Feng, Xiangnan He, Zhi Geng, and Peng Wu. Removing hidden confounding in recommendation: A unified multi-task learning approach. In NeurlPS, 2023.
[39] Amir Ismail- Yahaya and Achille Messac. Effective generation of the pareto frontier using the normal constraint method. In 40th AIAA Aerospace Sciences Meeting & Exhibil, page 178, 2002.
[40] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning, Advances in neural information processing systems, 32, 2019.
[41] Jörg Fliege and Benar Fux Svaiter. Steepest descent methods for multicriteria optimization. Mathematical methods of operations research, 51:479-494, 2000.
[42] Haoxuan Li, Chunyuan Zheng, Yixiao Cao, Zhi Geng, Yue Liu, and Peng Wu. Trustworthy policy learning under the counterfactual no-harm criterion. In International Conference on Machine Learning, pages
PMLR, 2023.
[43] T. Kitagawa and A. Tetenov. Who should be treated? empirical welfare maximization methods for treatment choice. Econometrica, 86, 2018
[44] Lu Cheng, Ruocheng Guo, and Huan Liu. Long-term effect estimation with surrogate repre- sentation. In Proceedings the 14th ACM International Conference on Web Search and Data Mining, pages 274-282, 2021.
[45] Wenjie Hu, Xiaohua Zhou, and Peng Wu. Identification and estimation of treatment effects on long-term outcomes in clinical trials with external observational data. arXiv preprint arXiv:2208..0 10163, 2022.
[46] Michael R Kosorok and Eric Laber. Precision medicine. Annual review of statistics and its application, 6:263-286, 2019.
[47] Scott Thiebes, Sebastian Lins, and Ali Sunyaev. Trustworthy artificial intelligence. Electronic Markets, 31:447-464, 2021.
[48] Floridi Luciano. Establishing the rules for building trustworthy ai. Nature Machine Intelligence, 166):261-26,, 2019.
[49] Davinder Kaur, Suleyman Uslu, Kaley Rittichier, and Arjan Durresi. Trustworthy artificial intelligence: a review. ACM Computing Surveys (CSUR), 55:1-38, 2022.
[50] Silvia Chiappa. Path-specific counterfactual faimess.. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 7801-7808, 2019.
[51] Yongkai Wu, Lu Zhang, Xintao Wu, and Hanghang Tong. PC-Fairness: A unified framework for measuring causality-basee based fairness. Advances in Neural Information Processing Systems, 32, 2019.
[52] Nathan Kallus. Treatment effect risk: Bounds and inference. Management Science, 69(8)4577 4590, 2023.
[53] Nathan Kallus. What's the harm? sharp bounds the fraction negatively affected by treatment. Advances in Neural Information Processing Systems, 35:15996-16009 2022.
[54] Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. Advances in neural information processing systems, 31, 2018.
[55] Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. Dpp-net: Device- aware progressive search for pareto-optimal neural architectures. In Proceedings of the European conference computer vision (ECCV), pages 517-531, 2018.
12

---

[56] Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objectivn rein- forcement learning: Novel design techniques. In 2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pages 191-199. IEEF, 2013.
[57] Nan Xu, Nitin Kamra, and Yan Liu. Treatment recommendation with preference-base rein- forcement learning. 2021 IEEE international conference on big knowledge (ICBK), pages 1-8. IEEE, 2021.
[58] Jiangjiao Xu, Ke Li, and Mohammad Abusara. Preference based multi-objective reinforcement learning for multi-microgrid system optimization problem in smart grid. Memetic Computing, 14(2):225-23,, 2022.
13

---

A Estimation of Short-Term and Long-Term Rewards
For a given policy T the policy values for the short-term outcome S; and the long-term outcome y are defined as
V(O; 8;) Elm + (1- =(0)s, i= i,.... v(O:y,) (1)) + (1 - x (0), j= J,
Under Assumptions as
the short-term reward V(O; st) and long-term reward V(0:y,) are identified
V(O: (.) = (l)(0) (X) + - (0))
V(O: (0m(0m,) (X, + (1- S)].
(X, where Pia (X) = -[ A a) j S) identifiability results are derived using a similar approach to that outlined in Section [8. In addition, for estimating the V((;; s:) and V(0: y5). [8] proved the efficient bounds of V 8;) and V(O; U5), which we list them below for the sake of celf-containedness.
E|y X, XX R. 1] for 0. The
Lemma A.1 (Efficiency Bounds of V(0; s:) and V(O; vs). 8). Let Z = A,S, Y), under Assumptions we have that
(a) the efficient inftuence function of V(0;s, 8;) is Oo, V(O: 8;), where
2s. (Z; M5D- Ps1)
= {T(()) ()) + (1 - (0)) (X)}
(())((S; pa (X))
+ n T0)) A)(S: Moo (X))
e(X)
1 1-e(X)
and e(X) PPAA V(0; 8;) is Varl(s.)
1|X) is propensity score. The associated semiparametric efficiency bound of
(b) the efficient inftuence function V(O; y3) is 0R, - (0; (;), where
Ouz 0b (Z:e, m, m11 m,o. m,1)
(m(0)ms) (X) + (1 (0) mm (X))
(O)AR, (Y; - (X,) x A()) (X.S) 5) - -ny1 (X)) el (1,),
e(X)
(1 (0)) - (Y, - -x (x,
(1- (1-e(X))rs, (O, (O,
(1 - (0))(1 - A) (X - m X))
+
1 e (X)
mjaa (X)
E|y, X. |X Rg 1] the regression function for Y,, and (A, S)
P(R, = 1 A) is selection score. The associated semiparametric efficiency bound V(O: yÿ) is
Var(Wug)
From Lemma A.1 for a given policy x(0), natural to define the estimators of V(0; s:) and (0 uj)
((0
Σ
2. (Zn é, Paa, Aa),
N
v(0 v1)
Σ
by (Zni é, fj mjo. M,I mn mA1).
N ==1
where N the sample size. All of them can be identified from the observed data.
And ((x), (isa))))))) ma ma (ar, s), and ,y s) for = 0, are the estimators of (z)(())))))))))))))))))(()) (x), mjaa (x, 8) and ry (a, x, 8) respectively.
e(x), pio (x), m)),mja(ee
14

---

which can be reformulated as B Algorithm Flowchart for DPPL
Algorithm 1 DPPL Algorithm
1: Input: A set of preference vectors {u), u2,..., ux)
(All subproblems can be solved parallel)
2: for k = to K do
3: randomly generate parameters 0,*))
4: find the initial parameters (()) from 0²*²) using gradient-based method (step a)
5: for to T do
6: obtain X 2 0,
(k)
2 0, Vm ..... M, Ak' E (,) by solving subproblem 6
EK
calculate direction d -EE ii= Am Vnn (0())) ) E 8 V9 (0,*)) ld,
7:
- Exena 3/ (ux' Ukm / (0+) (step b)
8: update the parameters ++1 = 0(h)) nyd
9: end for
10:
end for
Output: The set solutions for all subproblems with different trade-offs (0 14 = 1,.... K}
11:
C Proofs of Theorem
For the case of only have long-term outcome Y and short-term outcome S, considering the g-constraint optimization problem
min (0: s.t., V(0:) S
(A.1)
and the linear weighting optimization problem
min w V(O;g) + + ww0 0
(A.2)
min V) + ():
where A w2/w1 controls the balance between short-term and long-term rewards. Let (X) = [E(((1) - S(O) X] and (X) = (1 Y (O) When À = 0. is equiva- lent to finding an optimal policy for minimizing V(O; y) alone, (0) = arg min, (00;)) = ((T, 2 0). When À = DO, equivalent to finding an optimal policy for minimizing the V(O;s) s) alone, (0) arg min, V(O s) arg min, -E]T
arg min, -Ri (0),,
[(r, 2 0). We have the following theorem:
Theorem C.1. For the weights w problem following statements hold:
and the thresholds e in problem the
When (0) + (1) the solution of the constrained optimization problem is empty.
When 2 as follows:
the relationship between X and a is described
- 0. if ()); (1-m7
A is the solution of the equation
[Ty XX + XT. (X) 0) +.(X) [X)] = +(1
if f + .. (()) < S (0) + (1 - ()))))())))))
It is important to note that for a given X, we could solve the value of by solving the equation
-E Xr. X 0) J. (X) 1 X = =
as the left side of the equation a monotone function of À and the solution is unique, and all the quantities such 7 (X), Ty (X), and (xX are identifiable.
15

---

Proof. Initially, we recognize that E cannot be too small so that no policy can satisfy the constraint of V(O: S E. The optimal policy of minimizing only the D(0:s) o: [x 2 0). Thus, & 2 [[)=(( + (1 -T (0)))(())
When 2 (0)S(())
=- (1)=((0))000)) First, the optimal policy of minimizing only V(T;; y) is given as " (0) (TTx(X 2 0). Then, I (0) + (1 क (0)) otherwise, the constraint will be invalid and the constrained optimization problem becomes an unconstrained optimization problem with À
Second, when -B(m + (1 म, (()) S S (0)((() + (1 - ()))((()))))) show that the optimal policy x² (0) parameterized by 0", for the constrained optimization problem
min v) s.t., (00 8) S
obtained only when (00 =. Below, we prove with the method of reduction to absurdity, -E (1)5(1) + (1 Tī (()) = -B + m (()) then there are some units that satisfies {Ta (X)
0, (2)) 0} that not being assigned treatment by x² (0); otherwise, s) I & will be violated. Thus, we could find another treatment policy #t that the constraint v)
assigns more treatment to the units with (X) < 0, Ty (X) > 0}. which yields a lower V(0:)) y) but increases the V( s). That is, #* will lead V(O; s) closer to E but has lower V y) than π², thus, ' is not the optimal policy, which contradicts its definition of π². Thus, the constrained optimization problem becomes
min v): 8.1., ((0;)) =
By introducing the Lagrange multiplier 8. n² satisfies
π = arg min
V(0;y) + (V = (l(TT(X))=--))) + (x, > 0),
where B is the solution of v(0";s)) = i.e.,
-
E (X) + BT& (X) 0)Tx [o( =E.
This completes the proof for Theorem
We can further extend Theorem C to situations where there are multiple long- term-ermm rewards and multiple short-term rewards. More generally, for the e-constraint optimization problem
min (0))), Pm (0) S cm for all -1..... M, m # I
(A.3)
0
and the linear weighting optimization problem
min ((
Σ Wm Dm (0),
(A.4)
0
i=m
where wm is the pre-specified weight for the m-th reward. We have the following theorem:
Theorem 1. For the preference vector ug in problem (T). the weights w in problem 2 and the thresholds in problem 88. the following statements hold:
(a) the connection between and is given
-B[[TT(X)
Wmn
Ten
(X) > 0) Fm (X) + hm (X)] = Em, for m 1... M, and m + I
Wi
where Tm (X) is the conditional average causal effects for m-th short/oon-term outcome,
[[(((1) - 5.(0) X), if wm is the weight of (O, si): (1) - ( X], if wm is the weight of ((0 u;),
(X)
and
(0) wm is the weight of (0..s))
hm (X)
(0) S. = 1], wf is the weight of (0, v,).
16

---

and I(-) is the indicator function.
(b) the connection between w and u... given
Wm = Am +
Bi (usmmm ukm); ; for m = -,.... M,
K'EI, (0)
where Am and By are defined in 6q. I. (0) (KIG -c}
Proof. First, for the Theorem TT) a), combining the Theorem C.I more generally, for the e-constraint problem
min (0),), s. Pm (0) Em for all ...... M, m + 1
(A.5)
and the linear weighting optimization problem
M
min V(O)
Wm Vm (0),
(A.6)
1=m
where wm is the pre-specified weight for the m-th reward. By mathematical induction, have:
[T((T((X) + wmn /im/wiTi X) > 0) (X) + Amo ()) = E;, 1... and m +
Ts. E[S,(1) XX) if wm is the weight of ((00ss)).
where Ty (X)
=
Tvo (1) (,) X], if wm is the weight of (0 3;),
X = E[S; X, A = 0). if wm is the weight of V(0 s;),
hmo (X)
myn( [X S) E X, S,A = Rg 1]. if wm is the weight of ((0,y)),
This completes the proof for Theorem I[]
Second, for the Theorem (b), motivated by 40 for constraint problem
1
(de, 01) =arg min a 1+1 dER", s.t. VV a, m I,.., M.
d|l
2
(A.7)
Vg (0))² S a, E (.()).
we have
M Vgw (u. ux) VV(0.) "
Σ
(ur' Ukvm VV (0.).
(A.8)
Base on KKT conditions, we have
M
d. Am (0.)+
Σ
Be Vg.
Σ Am
Σ
BR =1
(A.9)
WET.
k'EI.
where Am 0 and BR < 0 are the Lagrange multipliers. Then, the dual problem is given
M
max
Am (0,+
Σ
By Vg (0.)l1
Amm Bar
2
m=1
KEEL(0)
(A.10)
M
s.f.
Am
+
Σ
3R = Am 2 0, 2 0, Vm = 1, 1,... M.VK' E (I))).
(E).
Substituting Eq. A.8 A.8
into Eq
A.9
have
17

---

M Wkm) VDm (8.)))
de Σ Am (0))
Σ BRL
uk/m
(EE,(
(A.11)
-Amm +
Ukm
V
For the problem(A. de is the negative gradient direction. Thus, we have
M
((0)
= Σ
wm Vm (0), where Win
Am +
Σ
Ukm)
(A.12)
m=
where Am and Br obtained from (q.0A.10). This shows that the DPPL method can be transformed into the linear weighting method. This completes the proof for Theorem Tb)
D Additional Experimental Results
D.1 Sensitivity Analysis on Missing Ratio
In the following, show experimental result with missing ratio 0.3 under IHDP and JOBS datasets, in table[D]
In additional, We show the corresponding & value for each preference vector with different missing ratio (0.3, 0.4, 0.5) under IHDP and JOBS datasets, tables D2 D3 and D7
Table DI: Comparison our method (OURS) and linear weighting method (LW) with 10 preference vectors IHDP and JOBS, with Short-Tern Reward (S (S-REWARDS) and Long- -erm Reward (L-REWARDS), A AW and Variance (S VAR and L- VAR) as evaluation metrics. The missing ratio 0.3 and T = 4. The best result bolded.
IHDP
S-REWARDS
L-REWARDS
AW
S-VAR
L-VAR
PREFERENCE VECTOR
OURS
LW
OURS
OURS
OURS
LW
OURS
1 (1.00, 0.00)
523,060
520.760
389.488
385.990
41.174
38,277
12.673
13.621
49.054
58.344
524,900
12.336
63.531
(0.94, 0.34)
522.300
$22.440
385 931
393.533
39.517
42.889
14.998
12.181
51.183
59,204
(0.86, 0.50)
522.
523.300
376.300
376.515
34.642
34,559
12.591
15.040
61.746
51.064
0.64)
(0.64, 0.76)
$25.560
$17.800
387.263
390.716
41.313
39.160
13.7003
14.991
56.639
49.135
(0.50, 0.87)
523.420
517.440
389.624
383
41.424
36.528
12.594
18.091
59.317
59.628
0.94)
(0.17. 0.98)
$20.300
515.500
386. 994
399.012
549
42.308
13.622
19.584
$6.273
58,640
10 (0.00, 1,00)
522.500
514.980
381.171
385.961
36.737
35,372
12.455
19.711
43.415
49.718
JOBS
S-REWARDS
L-REWARDS
AW
S-VAR
L-VAR
OURS
1 (1.00, 0.00)
1615.540
1612.100
1221.629
1217.543
155.936
173
65.386 386
56.393
98. 666
92,897
1616.240
1600.280
1216.370
1217.547
153.657
75.467
(0.94, 0.34)
1616.380
1595.840
1229.393
1219.175
160.238
145.009
57.370
86.875
95.219
91,009
(0.86, 0.50)
1615.700
1592.200
1234.526
1201.847 847
162.465
134.375
56.556
88.052
89,535
94,647
(0.76,
1608.600
1595.260
1214.387
1219.359
148.846
57.526
95.379
79.273
99,852
(0.64, 0.76)
1612.120
1591,480
1222.689
1221.671
154.756
143.927
55.446
97.238
94.283
97,522
7 (0.50, 0.87)
1614.240
1588.660
1225.527
1220.786
157.235
142.075
58.574
104.776
85.414
108 986
(0. 0.94)
1607.880
1585.280
1227.527
1223.203
155.055
141.593
55.923
105.
85.365
101.940
(0. 0.98)
1610.600
1584.460
1221.183
1223.446
153.243
141.305
59,996
109.731
92.968
99,
10 (0.00, 1.00)
1612.740
1590.880
1211,837
1224.826
149.640
145.205
60.330
106.403
92.767
106.106
18

---

Table D2: The values corresponding to each preference vector in the two datasets IHDP and JOBS, where T and r 0.3, which are derived according to Theorem
IHDP
JOBS
PREPERENCEVECTOR
(1.00. 0.00)
0.827
0.865
(0,98, 0.17)
0.818
0.864
(0.94, 0.34)
0.825
0.859
(0.86. 0.50)
0.858
(0.77, 0.64)
0.800
0.858
(0.64, 0.76)
0.722
0.841
(0.50, 0.86)
(0.34, 0.94)
0.549
0.706
(0.17, 0.98)
0.539
0.726
(0.00, 1.00)
0.557
0.779
Table D3: The e values corresponding to each preference vector in the two datasets IHDP and JOBS,
where T and 0.4, which derived according Theorem T
IHDP
JOBS
PREFERENCEVECTOR
(1.00. 0.00)
0.822
0.877
(0.98, 0.17)
0.824
0.868
(0,94, 0.34)
(0.86, 0.50)
0.820
0.841
(0.77. 0.64)
0.813
0.801 0.806
(0.64. 0.76)
0.724
0.798
0.850,0886
0.524
0,703
(0.34,
0,512
0.694
(0.17. 0.98)
0,523
0.667
(0.00. 1.00)
0.523
0.666
Table D4: The values corresponding the preference vectors the datasets IHDP and JOBS, where T 4 and r 0.5, which are derived according to Theorem
IHDP
JOBS
PREFERENCEVECTEE
-E
(1.00, 0.00)
0.820
0.865
(0.98. 0.17)
0.826
0.863
(0.94, 0.34)
0.821
0.869
(0.86, 0.50)
0.816
0.853
(0.77. 0.64)
0.805
0.816
(0.64, 0.76)
0.679
0.781
(0.50, 0.86)
0.522
0.737
(0.34. 0.94)
0.489
0.722
(0.17, 0.98)
0.490
0.723
(0.00, 1.00)
0.541
0.684
19

---

D.2 Sensitivity Analysis on Preference Vector
In the following, we show more experimental result with different numbers of preference vectors K (4,8,, 12} under JOBS datasets, in table D5-D7.
Table D5:
Comparison of our method (OURS) and linear weighting method (LW) with 4 pref- erence vectors on JOBS, with Short-Term Reward (S-REWARDS) and Long- Term Reward (L- REWARDS), AW and Variance (S-VAR and LAR)R) evaluation metrics. The missing ratio
r 0.2 and = 4. The best result bolded.
L.
PREFERENCE VECTOR
OURS
LW
OURS
LW
OURS
LW
OURS
LW
OURS
LW
0.00)
210.87. 0.50)
1606.920
1599.620
1226.861
1222,933
154.242
148.628
60.608
77.
78.282
92.699
300.50. 0.86)
1612.500
1601.260
1226.470
1213.741
156.837
144.852
58.438
82.862
87.381
94,363
4(0.00, 1.00)
1615.740
1596.360
1224.834
1223.110
157.639
147.087
$8.856
87.150
Table D6: Comparison of method (OURS) and linear weighting method (LW) with 8 pref- erence vectors on JOBS, with Short-Term Reward (S-REWARDS) and Long- Term Reward (L- REWARDS), AW and Variance (S-VAR and L- L-VRR) evaluation metrics. The missing ratio
r 0.2 and 4. The best result bolded.
L.
PREFERENCE VECTOR
OURS
LW
OURS
LW
OURS
LW
OURS
LW
OURS
LW
1(1.00, 0.00)
162.215
2(0.97, 0.22)
1610.820
1605.220
1228.604
1223.384
157.064
151.654
63.065
66.878
85.019
87.506
300.90, 0.43)
1606.260
1599. 940
1212.864
1226.675
146.914
150.659
60.809
70.123
95.023
97.185
410.78, 0.62)
1614,960
1604.
1226.162
1222.282
157.913
62.883
5(0.62, 0.78)
1612.320
1594,220
1226.816
1225.956
156.920
147.440
$9.959
77.331
81.906
89.562
610.43, 0.90)
1611.860
1221.652
154,108
60.059
90.661
710.22, 0.97)
1612.700
1596. 060
1215.533
1224.388
151.468
147.561
$6.013
89 441
86.439
92.825
8(0.00, 1.00)
1612.580
1592.260
1233.756
1227.061
160.520
147.012
58.113
83.188
88,058
98,805
Table D7: Comparison of our method (OURS) and linear weighting method (LW) with 12 pref- erence vectors on JOBS, with Short-Term Reward (S-REWARDS) and Long-Term Reward (L- REWARDS), AW and Variance (S-VAR and L-VAR) evaluation metrics. The missing ratio
0.2 and
The best result bolded.
JOBS
S-REWARDS
L-REWARDS
AW
S-VAR
L-VAR
PREFFRENCE VECTOR
OURS
1.W
QURS
I.W
OURS
I.W
OURS
I.W
OURS
1(1000, 0.00)
1610.800
1614.600
1231.786
1232.158
158.645
160.731
56.774
60.101
89.746
87.940
210.98, 0.14)
1609.720
1610.740
1224.605
1222.904
154.515
154.174
59.048
88.027
92.283
3(0.95, 0.28)
1613.520
1606. 320
1228.204
1226.660
158.214
153.842
59.244
65.024
84,400
79.787
1615.600
1223.297
1231.718
152.681
58.106
88,081
5(0.84, 0.54)
1614.140
1604, 1604.860
1218..55 585
1220.647
153.714
150.105
61.420
614
89.712
93.134
6(0.75. 0.65)
1615.240
1598, 960
1227.95
1225.251
158.949
149.4577
54.882
76.213
86.061
89.256
7(0.65, 0.75)
1616.380
1596.
1226.506
1218.845
158.795
61.503
8(0.54, 0.84)
1613.420
1598.460
1223.097
1229.293
155.610
151.228
58.566
83.
92.342
97.554
9(0.41. 0.91)
1612.940
1594.320
1222.586
1224.040
155.115
146.532
$7.262
84.387
87.325
93,589
10(0.28. 0.95)
1612.980
1596.880
1230.538
1218.671
159.111
145.127
61.465
$1.949
92.381
95,530
1100. 0.98)
1612.160
1591.760
1214.424
1224.345
150.644
145
58.148
86.692 692
80.732
90.234
12.0.00.1.00)
1224.288
157.978
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the abstract and the third and fourth paragraphs in Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the conclusion (especially the last sentence.)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification: In Section 2.3, we provide detailed discussion of the adopted assumptions. Additionally, we present the complete proofs in Appendices A and
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section 4, we provide a detailed description for the experimental datasets. In addition, provide the datasets and codes in supplemental material to ensure easy reproduction of all reported results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the supplemental material for datasets and codes in zip file to ensure easy reproduction all reported results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section for the detailed description simulating outcome and experi- mental details. In addition, provide the supplemental material for datasets and codes in zip file to ensure easy reproduction of all reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the variance for all experimental results section by replicating each experiment 50 times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA]
Justification: All experimental results can be easily reproduced on personal computer.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://ggggggaaaa...././///aaaaa
Answer: [Yes]
Justification:
All experiments are conducted on publicly available datasets.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the first paragraph of Section (Introduction), we outline various potential applications of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
All experiments are conducted on publicly available datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In Section 4, provide references for the datasets and the simulation setups of the lata-gnneg process.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: In Section 4, we provide references for the datasets and the simulation setups of the data-generating process. addition, we provide the supplemental material for datasets and codes in zip file to ensure easy reproduction of all reported results.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't use a crowdsourcing service.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "GPykjiAt7X", "reviewer": "Reviewer_ouKL", "summary": "This paper proposes a framework for solving multi-objective optimization problems: multi-objective optimization problems are divided into sub-problems in different regions by setting different preference vectors. The parameter optimization direction of the sub-problem can be easily solved by transforming it into a dual problem through the KKT condition, and a Pareto optimal solution of the original problem can be obtained by solving the sub-problem. This paper uses this framework to balance the optimal strategy learning under multiple short-term rewards and long-term rewards and achieves better and more stable performance than the traditional linear weighted method in the constructed experimental environment.", "questions": "1. How to deal with a situation where long-term rewards are missing? Should the data be ignored when solving the network?\nI think the optimization method proposed in this paper and the connection with related methods are very interesting, and I will improve my score as appropriate.", "limitations": "Have addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper reveals in detail the connection between the proposed method and the linear weighted method and the epsilon-constrained optimization method. Based on this connection, the epsilon-constrained optimization method can provide interpretability for the method in this paper.\n2. The method in this paper theoretically overcomes the suboptimality problem of the linear weighted method and avoids the situation where the epsilon-constrained optimization method does not have a feasible solution.\n3. This paper obtains better and more stable results than the epsilon-constrained optimization method in the optimal strategy learning problem under multiple short-term rewards and long-term rewards constructed by the author.", "weaknesses": "1. This paper mainly proposes an important multi-objective optimization algorithm and compares it with two existing algorithms in theory. However, the title of this paper seems to be just a specific application scenario of the algorithm. In what other scenarios can this algorithm be applied?\n2. The experimental part is mainly conducted in a constructed environment, and it is unclear how difficult it is in the field of causal inference.\n3. The v in line 171 is missing \\bar. In Appendix B, t in line 5 of Algorithm 1 should start from 0.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your approval of the idea and the novelty of this work and thank you for the helpful suggestions. Below, we hope to address your concerns and questions. \n\n> **W1**:  This paper proposes an important multi-objective optimization algorithm. But the title of this paper seems to be just a specific application scenario. In what other scenarios can this algorithm be applied?\n\n**Response:**  Thank you for the insightful comments. As the reviewer noted, we propose a general multi-objective optimization algorithm, making the proposed method applicable to various scenarios involving multi-objective balance.  Balancing multiple objectives is particularly critical in the field of trustworthy AI, where the goal is not only to achieve desirable prediction performance but also to address other important aspects such as fairness [1,2] and non-harm [3,4].\n\n> **W2**: The experimental part is mainly conducted in a constructed environment, and it is unclear how difficult it is in the field of causal inference.\n\n**Response:**  Thanks for the constructive comments. In the field of causal inference, the primary focus is on estimating  treatment effects. With several identifiability assumptions, various methods have been developed to achieve this goal using the observed data only. **However, it is impossible to evaluate a method's performance based solely on the observed data. The basic rationale is that the true treatment effects are always unknown in real-world data.** Thus, the experiments in almost all the articles in causal inference [5-9] are conducted in a constructed environment where the true treatment effects are known.  \n\n**Next, we give a detailed explanation.**  Consider the case of a binary treatment variable $A\\in\\{0,1\\}$, $X$ is covariates, and $Y$ is the outcome. Although $X$, $A$, and $Y$ are all observed variables, they cannot alone define treatment effects. To define treatment effects, we denote $Y(a)$ as the potential outcome that would be observed if $A$ were set to $a$ ($a$=0, 1). Thus, each individual has two potential outcomes: one is $Y(1)$ if the individual receives the treatment ($A=1$), and the other is $Y(0)$ if the individual don't receives the treatment ($A=0$). The observed outcome $Y_i = A_i Y_i(1) + (1-A_i) Y_i(0)$, that is,  the observed outcome is the potential outcome corresponding to the received treatment. With the notation of potential outcomes,  we define the individualized treatment effect as \n$$ ITE_i=Y_i(1)-Y_i(0),$$\nwhich measures the magnitude of the treatment effect for individual $i$. **Since each individual can only receive one treatment, we observe either $Y_i(0)$ or $Y_i(1)$, but not both. This  is known as the fundamental problem of causal inference [10]. As a result, we cannot obtain the true ITE from the observed data because we cannot simultaneously observe $Y_i(0)$ and $Y_i(1)$.**\n\nIn this article, we aim to find the optimal policy that maximizes rewards $V(\\pi)=E[\\pi(X)Y(1)+(1-\\pi(X))Y(0)]$, which is also a causal problem. Consistent with the above analysis, due to the fundamental problem of causal inference, for a given policy $\\pi$, the rewards cannot be evaluated with the observed data only. \n\n> **W3**: The v in line 171 is missing \\bar. In Appendix B, t in line 5 of Algorithm 1 should start from 0.\n\n**Response:** Thank you for pointing this out. We will revise it accordingly. \n\n> **Q1**: How to deal with a situation where long-term rewards are missing? Should the data be ignored when solving the network? \n\n**Response:**   Thank you for raising this point and we apologize for the lack of clarity. From a high-level perspective, the proposed policy learning approach consists of the following two steps:\n* Step 1: policy evaluation, estimating the short-term and long-term rewards $\\mathcal{V}(\\theta, s_i)$ and $\\mathcal{V}(\\theta, y_j)$ for a given policy; \n* Step 2: policy learning, solving the optimization problem (3)  based on the estimated values of $\\mathcal{V}(\\theta, s_i)$ and $\\mathcal{V}(\\theta, y_j)$.  \n\nThe missingness of long-term outcomes primarily affects Step 1, which involves the the identifiability and estimation of  $\\mathcal{V}(\\theta, s_i)$ and $\\mathcal{V}(\\theta, y_j)$ (see line 81 of the manuscript). Once $\\mathcal{V}(\\theta, s_i)$ and $\\mathcal{V}(\\theta, y_j)$ are estimated, Step 2 focuses on learning the optimal policy by developing advanced optimization algorithms.     \n\nFor Step 1, the previous work [11] is well-established. However, it does not address Step 2, as it only uses a simple linear weighting method.  In this article, we primarily focus on Step 2 by proposing a novel optimization algorithm. Additionally, we establish connections with other methods (e.g., $\\epsilon$-constraint) to guide the selection and provide interpretation of the preference vectors. For Step 1, we adopt the method of [11] and the associated results are presented in Appendix A.\n\n**References**\n\n---\n[1] Kusner et al. Counterfactual fairness. NeurIPS 2017\n\n[2] Ethan et al., Fairness-Oriented Learning for Optimal Individualized Treatment, JASA 2023\n\n[3] Kallus N., Treatment Effect Risk: Bounds and Inference, FAccT 2022\n\n[4] Li et al., Trustworthy Policy Learning under the Counterfactual No-Harm Criterion, ICML 2023\n\n[5] Johansson et al. Learning representations for counterfactual inference, ICML 2016\n\n[6] Shalit et al. Estimating individual treatment effect: generalization bounds and algorithms. ICML 2017\n\n[7] Shi et al. Adapting neural networks for the estimation of treatment effects. NeurIPS 2019\n\n[8] Yoon et al. GANITE: Estimation of individualized treatment effects using generative adversarial nets, ICLR 2018\n\n[9] Bica et al. Estimating the effects of continuous-valued interventions using generative adversarial networks. NeurIPS, 2020.\n\n[10] Holland. Statistics and causal inference. JASA 1986\n\n[11] Wu et al. Policy Learning for Balancing Short-Term and Long-Term Rewards. arXiv 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYRn964rBA", "reviewer": "Reviewer_xruw", "summary": "This paper studies the tradeoff between short-term and long-term rewards. The authors formulate the policy learning problem as a multi-objective optimization problem and propose a decomposition-based Pareto policy learning method. I only had experience in reinforcement learning in robotics five years ago. I tried my best to understand the paper, but I am not sure about my rating and comments.", "questions": "Please see the weakness.", "limitations": "Yes", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper studies a quite interesting and important problem, and the proposed methods seem effective on these two benchmarks.\n- The paper is well-organized, the division is relatively easy to follow, and the proposed method is well-motivated.", "weaknesses": "- Only the linear weighting method is used as the baseline. I am wondering if there are any other methods that can be used for comparison. If not, why? Since both IHDP and JOBS are widely used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your comments and thank you for the helpful suggestions. Below, we hope to address your concerns and questions. \n\n> **W1**: - Only the linear weighting method is used as the baseline. I am wondering if there are any other methods that can be used for comparison. If not, why? Since both IHDP and JOBS are widely used.\n\n**Response:**  Thanks for your comments. From a high-level perspective, the proposed policy learning approach consists of the following two steps:\n\n* Step 1: policy evaluation, estimating the short-term and long-term rewards $\\mathcal{V}(\\theta, s_i)$ and $\\mathcal{V}(\\theta, y_j)$;\n\n* Step 2: policy learning, solving the optimization problem (3) based on the estimated values of $\\mathcal{V}(\\theta, s_i)$ and $\\mathcal{V}(\\theta, y_j)$. \n\nFor Step 1, the previous work [1] is well-established. However, for Step 2, it only uses a simple linear weighting method. In this article, we primarily focus on Step 2 and adopt the method of [1] in Step 1, given in Appendix A.  \n\n**As discussed in the second paragraph of the Introduction (lines 32-35), the policy learning problem of balancing short and long-term rewards remains largely unexplored. There is only a single work [1] that addresses this problem, using a simple linear weighting method. Therefore, our work primarily compares with this approach.**\n\nIn addition, we also tried to consider the epsilon constraint method, but it can be converted into a linear weighted method through certain transformations, so we did not choose it as an additional baseline. \n\nNevertheless, to make a more comprehensive comparison, we added two extra estimation methods in Step 1, and compare our proposed optimization method with the linear weighting method in Step 2.  Specifically, the two extra estimators are given as \n\n* OR estimator \n$$\\hat{\\mathbb{V}}(\\pi;s)^{OR}=\\frac{1}{n}\\sum_{i=1}^{n} [\\pi(X_{i}) \\hat{\\mu}_{1}(X_i)+(1-\\pi(X_i))\\hat{\\mu}_0(X_i)]$$ \n \n$$\\hat{\\mathbb{V}}(\\pi;y)^{OR}=\\frac{1}{n}\\sum_{i=1}^{n}[\\pi(X_{i}) \\hat{\\tilde{m}}_1(X_i,S_i)+(1-\\pi(X_i)) \\hat{\\tilde{m}}_0(X_i,S_i)]$$\n\n* DR estimator \n$$\\hat{\\mathbb{V}}(\\pi;s)^{DR}=\\frac{1}{n}\\sum_{i=1}^{n}\\Big[\\pi(X_i)\\left(\\frac{A_i(S_i-\\hat{\\mu}_1(X_i)}{\\hat{e}(X_i)}+\\hat{\\mu}_1(X_i)\\right)+(1-\\pi(X_i))\\left(\\frac{(1-A_i)(S_i-\\hat{\\mu}_0(X_i)}{1-\\hat{e}(X_i)}+\\hat{\\mu}_0(X_i)\\right)\\Big]$$ \n\n$$\\hat{\\mathbb{V}}(\\pi;y)^{DR}=\\frac{1}{n}\\sum_{i=1}^{n}\\Big[\\pi(X_i)\\left(\\frac{A_i(Y_i-\\hat{\\tilde{m}}_1(X_i,S_i)}{\\hat{e}(X_i)}+\\hat{\\tilde{m}}_1(X_i,S_i)\\right)+(1-\\pi(X_i))\\left(\\frac{(1-A_i)(Y_i-\\hat{\\tilde{m}}_0(X_i,S_i)}{1-\\hat{e}(X_i)}+\\hat{\\tilde{m}}_0(X_i,S_i)\\right)\\Big]$$ \nThe associated experimental results on JOBS are presented below.\n\n| JOBS| S-Rewards | | L-Rewards | | $\\Delta{W}$ | |  S-VAR | |  L-VAR  | |\n|:-----------------:|:---------:|:--------:|:---------:|:--------:|:---------:|:-------:|:------:|:-------:|:-------:|:------:|\n| Preference Vector | OURS | LW | OURS | LW | OURS | LW |OURS | LW | OURS | LW |\n| (1.00, 0.00)| 1612.70 | **1614.44** | **1226.81** | 1212.77 | **157.11** | 150.96 | 60.53 | **60.12** | 102.07 | **87.59**  |\n| (0.98, 0.17) | **1611.94** | 1604.74 | **1220.29** | 1216.57 | **153.47** | 148.01 | **60.19** | 65.98 | 100.92 | **93.77** |\n| (0.94, 0.34) | **1613.46** | 1597.26 | **1220.38** | 1215.91 | **154.27** | 143.94 | **62.48** | 77.57 | **86.22** | 87.80 |\n| (0.86, 0.50) | **1614.34** | 1595.88 | **1228.16**| 1218.13 | **158.60** | 144.36 | **58.28** | 82.76 | 91.15 | **90.15** |\n|(0.76, 0.64)| **1616.20** | 1597.76 | 1219.02 | **1219.98** | **154.96**  | 146.22 | **56.09** | 88.02 | **90.36** | 92.34 |\n| (0.64, 0.76)|**1614.58** | 1592.96 | **1219.77** | 1217.16 | **154.53**  | 142.41 | **56.96** | 92.09 | **86.43** | 90.10 |\n| (0.50, 0.87)| **1614.64** | 1586.78 | 1212.08| **1223.47** | **150.71**  | 142.48 | **60.22** | 104.24 | **88.82** | 96.13 |\n| (0.34, 0.94) | **1611.14** | 1587.86 | 1212.31 | **1222.62** |**149.08** | 142.59 | **52.62** | 109.01 | **91.18** | 91.67 |\n| (0.17, 0.98) | **1612.24** | 1580.14 | 1222.36 | **1226.82** |**154.65** | 140.83 | **54.33** | 109.42 | **87.33** | 95.10 |\n| (0.00, 1.00) | **1613.48** | 1586.04 | 1222.03 | **1227.29** |**155.11** | 144.02 | **60.04** | 107.85 | **87.18** | 90.78 | \n\n\n| JOBS | S-Rewards |    | L-Rewards |    | $\\Delta{W}$ |  |  S-VAR |    |  L-VAR |    |\n|:---------------:|:---------:|:--------:|:---------:|:--------:|:---------:|:-------:|:------:|:-------:|:------:|:------:|\n| Preference Vector |OURS| LW |OURS| LW |OURS |LW | OURS | LW | OURS | LW |\n|(1.00, 0.00)|1607.82|**1612.26**|**1230.10**|1224.90| **156.31**|155.93| 63.43| **59.50**|90.06| **87.42**|\n|(0.98, 0.17)|**1616.08**|1598.50|**1230.45**|1223.66| **160.61** |148.43|**58.27**|65.18| 95.92| **90.49**|\n|(0.94, 0.34)| **1612.72**|1598.14|1218.20|**1218.87**| **152.81** |145.86|**57.97**|80.05|85.47| **84.29**|\n|(0.86, 0.50)|**1610.58**|1592.74|1219.66|**1224.71**| **152.47** |146.08|**57.60**|84.19|**80.43**| 95.71|\n|(0.76, 0.64)|**1608.50**|1594.00|**1228.59**|1221.82| **155.90** |145.26|**57.27**|89.49|**81.03** | 97.67  |\n|(0.64, 0.76)|**1616.34**|1592.82|1229.27|**1234.92**| **160.16** |151.22|**58.65**|87.72| **90.82** | 95.56  |\n|(0.50, 0.87)|**1612.54**|1593.30|1217.99|**1227.72**| **152.62** |147.86|**57.13**|101.56| **88.11** | 96.09  |\n|(0.34, 0.94)|**1613.38**|1584.02|**1224.66**|1220.44| **156.37** |139.58|**58.84**|104.60| **90.69** | 93.02  |\n|(0.17, 0.98)|**1612.54**|1591.02|**1228.40**|1224.59| **157.82** |145.16|**56.88**|110.24| 83.45  | **82.22**  |\n|(0.00, 1.00)|**1613.74**|1588.70|1218.97|**1237.01**| **153.71** |150.21|**58.38**|106.48| **90.94**  | 99.12  |\n\n\nFrom the experimental results, we can see that our method achieves better performance on the evaluation metric $\\Delta{W}$ (the most important evaluation metric), and our method also performs better in terms of overall performance.\n\n\n[1] Wu et al. Policy Learning for Balancing Short-Term and Long-Term Rewards. arXiv:2405.03329, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "boI9Q1izTS", "reviewer": "Reviewer_hVvL", "summary": "This paper attempts to address the challenge of learning the optimal policy for balancing multiple long-term and short-term rewards. The authors point out that the existing linear weighting method leads to a sub-optimal policy. To address this limitation, the authors propose formulating formulate the problem as a multi-objective optimization problem. They utilize the Lagrange algorithm to use preference vectors to solve the formulated multi-objective optimization problem and aim to learn the policy to meet Pareto optimization. In order to decide the preference vectors, the authors propose establishing the connection between the optimization problems and the ε-constraint problem. Experiments on IHDP and JOBS demonstrate the efficacy of the proposed method.", "questions": "Please see the comments above.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe multi-object problem is practical in both reinforcement learning and other optimization scenarios. The paper provides a good summary of the limitations of the existing linear weighting method and introduces a novel perspective on solving the problem by resorting to the Lagrange algorithm and Pareto optimization.\n2.\tThe author has a solid mathematical foundation and is able to provide detailed mathematical descriptions and solutions to the proposed optimization problems.", "weaknesses": "1. The authors point out that the linear weighting method is suboptimal. However, there is no explanation in the method section or corresponding experiments to demonstrate that the proposed method (i.e. DPPL) is optimal.\n\n2. In line 38, the authors claim that when some of the rewards are interrelated, the linear weighting method can only achieve a suboptimal solution. The claim may not be rigorous as the linear weighting method might be able to model the relationship among the rewards. More explanation and experiments are required.\n\n3. In line 95, the definition of Pareto optimality, the condition for Pareto optimality by the author is to find the $\\theta$ that makes all $\\bar{\\mathcal{V}}$ optimal. However, is it possible that the $\\theta$ is not optimal for some $\\bar{\\mathcal{V}}$ but is optimal for the overall $\\bar{\\mathcal{V}}$?\n\n4. Some mathematical symbols and proprietary terms in the paper are not explained clearly. For example, what does the $e$ in line 110 mean? What does MOP represent? Does MOP represent multi-objective problems? What do $v$ and $R_{+}$ mean in line 171? What does the KKT condition mean? Is it the KKT condition in the Lagrange algorithm? What is the difference between the two descent directions $d_{rt}$ and $d_t$? There are many similar situations in the paper. I suggest providing necessary explanations for each noun and symbol that appears for the first time.\n\n5. In section Simulating Output and section Experimental Details, many parameters are defined by the authors themselves, but most of them do not have reasons or ablation experiments. For example, why is the number of preference vectors 10? In Line 253 to Line 254, why are some parameters truncated normal distributions and some Gaussian distributions?\n\n6. In Table 1 on the L-REWARDS metric, the proposed method is comparable to the linear weighting method. However, the authors claim that for most of the preference vectors, DPPL's solutions have better performance.\n\n7. In Figure 1, it seems that the effect on the $\\delta{w}$ from the missing rate and T is not obvious for either the proposed method or LW. More explanation is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Below, we hope to address your concerns and questions.\n\n>**W1** No enough explanation/experiments to demonstrate that the proposed method is optimal.\n\n> **W2**: When some of the rewards are interrelated, the linear weighting method can only achieve a suboptimal solution. The claim may not be rigorous. More explanation and experiments are required.\n\n**Response:** Thanks for your comments. Since W1 and W2 are similar, we response them together.  \n\n**First, we explain why the proposed method is optimal. In Lemma 2 of Section 3.2 (lines 191-194), we show that the proposed method achieves Pareto optimality.** Specifically, a Pareto optimal solution is obtained by solving the optimization problem (5) (lines 181-182) using an iterative gradient-based update rule. Lemma 2 demonstrates that this iterative rule converges to the Pareto optimal solutions.\n\n**Second, we explain why the linear weighting (LW) method is suboptimal**.\n - Previous studies [1-4] highlight the limitations of LW method in multi-objective optimization. LW method is restricted by assumptions of linearity and convexity, potentially ignoring complex interactions among objectives.\n- When multiple objectives have nonlinear relationships, the Pareto set (the collection of all Pareto optimal solutions, see line 100) may not be convex. Due to its linear restriction, the LW method may not effectively identify Pareto optimal solutions within non-convex regions, leading to suboptimal solutions. \n- As illustrated in Figure 2 of [5], when dealing with two objective functions, the LW method may struggle to identify Pareto-optimal solutions when the Pareto set is nonconvex.\n\n**Finally, we conducted an extra experiment to further illustrate the limitations of LW method and the optimality of our method.** We consider two objectives \n$min_x F(x)=[f_1(x),f_2(x)]^T,$ \n$$f_1(x)=1-\\exp{(-\\sum_{i=1}^d(x_d-\\frac1{\\sqrt{d}})^2)}, f_2(x)=1-\\exp{(-\\sum_{i=1}^d(x_d+\\frac1{\\sqrt{d}})^2)},$$\nwhere $x$ is a 20-dimensional covariates and $x_d$ is the $d$-th element of $x$. For the optimization problem above, we know the true Pareto set. For clarity, we present 10 uniformly selected Pareto solutions in Table 1 below, from which we can see that the Pareto set is concave rather than convex.\n\n**Table 1**\n|id|f1_pf|f2_pf|\n|-|-|-|\n|0|0.982|0.000|\n|1|0.960|0.041|\n|2|0.921|0.153|\n|3|0.854|0.313|\n|4|0.754|0.486|\n|5|0.617|0.647|\n|6|0.452|0.777|\n|7|0.279|0.870|\n|8|0.126|0.930|\n|9|0.026|0.966|\n\nNext, we use both the proposed method and the LW method to solve the above optimization problem. Here we select ten preference vectors (they also are the weight vectors in the LW method). The results are shown in Table 2 below,  where $(f_{1, LW},f_{2, LW})$ and $(f_{1, our},f_{2,our})$ represents the solution obtained by the LW method and our method, respectively. **we can see that the LW method only finds solutions at the endpoints of the Pareto set (i.e., in the convex part), while our method successfully locates all Pareto optimal solutions within the whole region.** This indicates that the LW method cannot achieve the Pareto optimal solutions.\n\n**Table 2**\n|PreferenceVector|$f_{1,LW}$|$f_{2,LW}$||$f_{1,our}$|$f_{2,our}$|\n|-|-|-|-|-|-|\n|(1.00,0.00)|0.002|0.980||0.953|0.060|\n|(0.98,0.17)|0.000|0.981||0.918|0.160|\n|(0.94,0.34)|0.000|0.981||0.823|0.374|\n|(0.86,0.50)|0.001|0.980||0.736|0.511|\n|(0.76,0.64)|0.001|0.979||0.632|0.632|\n|(0.64,0.76)|0.979|0.001||0.632|0.632|\n|(0.50,0.87)|0.007|0.974||0.511|0.736|\n|(0.34,0.94)|0.981|0.000||0.334|0.844|\n|(0.17,0.98)|0.981|0.000||0.228|0.892|\n|(0.00,1.00)|0.979|0.002||0.055|0.955|\n\n> **W3**: For Pareto optimality, is it possible that $\\theta$ is not optimal for some $\\mathcal{\\bar V}$ but is optimal for the overall $\\mathcal{\\bar V}$? \n\n**Response:** Thanks for your comments. We kindly remind the reviewer that there might be a misunderstanding regarding the definition of Pareto optimality. Below, we provide a detailed explanation. A solution is Pareto optimal if it is impossible to improve on objective without worsening other objectives. **Typically, multi-objective problems have numerous Pareto optimal solutions, and all of them form the Pareto set. These solutions represent the best trade-offs among different objectives, rather than optimality for a specific single objective.** Thus, it is possible that a Pareto optimal solution achieves the optimal solution in some objectives but not for others. \n\n> **W4**: Some mathematical symbols and proprietary terms in the paper are not explained clearly.\n\n**Response:** Thanks for your comments and we apologize for any lack of clarity. Below, we provide a detailed explanation of the mathematical symbols:\n- We define $e(x) \\triangleq P(A=1|X =x)$ (see line 110).\n- The MOP represents multi-objective optimization (see line 91).\n- In line 171, $v$ is an $M$-dimensional vector in the positive real space $R_+^M$, where $M$ is the number of objectives.\n- The Karush-Kuhn-Tucker (KKT) conditions are a set of necessary conditions that characterize the solutions to constrained optimization problems. The KKT conditions can be seen as a set of constraints derived from the Lagrangian function that must be met for a solution to be considered optimal in a constrained optimization problem.  \n- Both $d_{r_t}$ and $d_{t}$ represent the descent direction at $t$-th iteration, where the former is for seeking the initial solution, while the latter is for finding the Pareto optimal solution. \n\n---\n[1] Miettinen. Nonlinear multiobjective optimization, 2012\n\n[2] Marler et al. The weighted sum method for multi-objective optimization: new insights, 2010\n\n[3] Censor. Pareto optimality in multiobjective problems,1977\n\n[4] Ngatchou et al. Pareto multi objective optimization, 2005\n\n[5] Shim et al. Pareto‐based continuous evolutionary algorithms for multiobjective optimization, 2002\n\n(Regarding the response to W5-W7, please see global review.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1RUVaV6NOa", "reviewer": "Reviewer_B23s", "summary": "This paper introduces a new way to balance multiple rewards with some long-term rewards potentially missing. It does so by using Pareto Policy Learning of optimizing each reward subject up to the tradeoff frontier. This can be more practical than simple linear weighting since the linear weighting strategy applies the constant weight regardless of the amount of conflict between pairs of objectives. Empirically, the papers show that the approach is superior to linear on two synthetic tasks with some real data. Overall I think the paper is promising and adding more realistic empirical evaluation can add values to the current state of the paper.", "questions": "- The separation between short-term and long-term reward is practically meaningful, but mathematically the only difference is that one reward can be missing and the other is fully observable, since the Pareto Policy Learning treats all objectives the same. Do we necessarily need these separate definitions? Can we put everything as a long-term reward that can be missing sometimes?\n- How are the weighting and preference vectors chosen? Have the author considered running a sweep over different configurations to compare against linear weighting? Apologies if I overlook this detail from the paper.", "limitations": "- The limitation of the overall framework are mentioned but there is no much detail perhaps due to space constraint.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Learning to combine multiple rewards is an important and well-motivated question, and has wide ranging implications.\n- The method proposed is mathematically sound. The paper shows theoretically that the input parameters can be interpreted as a form of worst case value on each objective. \n- The paper explains how the most popular approach of linear weighting can fall short, derives the method through first principles, and empirically demonstrates that the proposed method is superior.", "weaknesses": "- The main weakness of the paper is that the experimentation is rather limited. The experiment uses partial real data with synthetic generation of short-term and long-term rewards. For example, in robotic planning, the authors could show how their approach helps balance the long-term reward (e.g. goal reaching) / short-term reward (e.g. minimizing jerk). This is just an example, but including other more real-world planning and RL problems would seem beneficial.\n- It seems that compared to linear weighting, the proposed method seeks more short-term reward but is not necessarily better in terms of long-term reward. It may not be a weakness but reading the table does strike me that the method is more “short-sighted.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for approving our work and for the helpful suggestions. Below, we address your concerns and questions. \n\n>**W1**: The experiment uses partial real data with synthetic generation of short-term and long-term rewards.\n\n**Response:** Thanks for your comments. We fully agree that applying the proposed method to other domains, such as robotic planning and RL, would be beneficial. \n\nHowever, **we would like to clarify that our method focuses on policy learning in causal inference, which has peculiarities that cannot be correctly evaluated using real-world data**. Here are the reasons:\n\nIn causal inference, the primary focus is on estimating treatment effects. Various methods achieve this using observed data under several identifiability assumptions. However, **it is impossible to evaluate a method's performance based solely on observed data because the true treatment effects are always unknown in real-world data. Therefore, almost all experiments in causal inference [1-5] are conducted using synthetic data where the true treatment effects are known.** \n\n**Next, we give a detailed explanation.** Consider a binary treatment $A\\in\\{0,1\\}$, covariates $X$, and outcome $Y$. Although $X$, $A$, and $Y$ are observed variables, they cannot alone define treatment effects. To define treatment effects, we denote $Y(a)$ as the potential outcome if $A$ were set to $a$ ($a$=0, 1). Thus, each individual has two potential outcomes: $Y(1)$ and $Y(0)$. The observed outcome $Y_i = A_i Y_i(1) + (1-A_i) Y_i(0)$, i.e., the observed outcome is the potential outcome corresponding to the received treatment. Then individualized treatment effect is defined as  \n$$ ITE_i=Y_i(1)-Y_i(0),$$\nwhich measures the treatment effect for individual $i$. **Since each individual can only receive one treatment, we observe either $Y_i(0)$ or $Y_i(1)$, but not both. This is the fundamental problem of causal inference. Thus, we cannot obtain the true ITE from the observed data because we cannot simultaneously observe $Y_i(0)$ and $Y_i(1)$.** In this article, we study the policy learning problem, which is also a causal problem. Consistent with the above analysis, due to the fundamental problem of causal inference, the rewards for a given policy cannot be evaluated using observed data alone.\n\n> **W2:** It seems that compared to linear weighting, the proposed method seeks more short-term reward but is not necessarily better in terms of long-term reward.\n\n**Response:** Thanks for your insightful comments. In this article, **we aim to develop a policy learning approach that balances both short- and long-term rewards, rather than focusing solely on maximizing either of them.** \n\nTo evaluate the proposed method, we mainly use three metrics: S-REWARD, L-REWARD, and $\\Delta W$. S-REWARD measures the short-term reward induced by the learned policy,  L-REWARD measures the long-term reward, and $\\Delta W$ measures the balanced welfare, indicating the policy's ability to trade off between multiple rewards. Clearly, **to evaluate an approach that balances both short-term and long-term reward, $\\Delta W$ is the most important metric.**\n\nIn the experimental results, the proposed method may not always perform better on  L-REWARD, but it shows superior performance for $\\Delta W$. **This indicates that the linear weighting approach does not achieve a good balance, as it pursues long-term rewards at the expense of short-term rewards. In contrast, our approach proves to be Pareto optimal, pursuing long-term rewards while also maximizing short-term rewards.** \n\n> **Q1**: Do we necessarily need these separate definitions? Can we put everything as a long-term reward that can be missing sometimes? \n\n**Response:**  Thank you for raising this point. Yes, both long-term and short-term outcomes can be missing, and the proposed policy learning method remains applicable. However, this will bring extra difficulty in identifying and estimating long and short-term rewards. From a high-level perspective, the proposed policy learning approach consists of two steps:\n\n* Step 1: policy evaluation, estimating the short and long-term rewards; \n* Step 2: policy learning, solving optimization problem (3) based on the estimated rewards.\n\nThe missingness of short and long-term outcomes mainly affects Step 1, which involves identifying and estimating the rewards. Once the rewards are estimated, Step 2 focuses on learning the optimal policy through advanced optimization algorithms. \n\nIn Step 1, when both outcomes are missing, the estimation method of [6] may be invalid. Developing novel methods for identifying and estimating rewards in this context is an interesting direction, and we leave it for future work. \n\n> **Q2**: How are the weighting and preference vectors chosen?\n\n**Response:**  Thanks for your comments. Preference vectors (PVs) are used to quantify an individual's preference for different objectives. **In our article, we randomly generate 10 unit PVs (see lines 270-272), ensuring the consistency and comparability of the preference measures.** Each component of the PV represents the importance of the decision maker's preference for different objectives. Each PV is considered as a weight vector in the linear weighting method, see lines 275-276.  **In addition, we added experiments for different number of PVs, see response W5 to reviewer hVvL.**\n\n[1] Johansson et al. Learning representations for counterfactual inference, ICML 2016\n\n[2] Shalit et al. Estimating individual treatment effect: generalization bounds and algorithms.ICML 2017\n\n[3] Shi et al. Adapting neural networks for the estimation of treatment effects. NeurIPS 2019\n\n[4] Yoon et al. GANITE: Estimation of individualized treatment effects using generative adversarial nets, ICLR 2018\n\n[5] Bica et al. Estimating the effects of continuous-valued interventions using generative adversarial networks. NeurIPS 2020\n\n[6] Wu et al. Policy Learning for Balancing Short-Term and Long-Term Rewards. arXiv 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewer hVvL, we provide the responses to W1-W4 below your Official Review. Here, we further response W5-W7. 

> **W5**: In experiment, why choose 10 preference vectors? why are some parameters truncated normal distributions. 

**Response:** Thanks for your comments. **We would like to clarify that our data generation mechanism follows from previous works [6-8] for ease of comparison. In addition, we further performed two more experiments with varying numbers of preference vectors and different truncation thresholds for normal distributions.**
 
First, we generate $K$ unit preference vectors (the same as the article), but with $K$=4 and 12. The results, shown in Tables 3-4 below, indicates that the proposed method stably performs better. 

**Table 3, $K=4$**
|JOBS|S-Rewards||L-Rewards||$\Delta{W}$||S-VAR||L-VAR||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|PreferenceVector|OURS|LW|OURS|LW|OURS|LW|OURS|LW|OURS|LW|
|(1.00,0.00)|**1616.5**|1613.9|1226.5|**1232.2**|158.9|**160.4**|60.2|**57.8**|94.8|**92.3**|
|(0.87,0.50)|**1606.9**|1599.6|**1226.9**|1222.9|**154.2**|148.6|**60.6**|77.8|**78.3**|92.7|
|(0.50,0.86)|**1612.5**|1601.3|**1226.5**|1213.7|**156.8**|144.9|**58.4**|82.9|**87.4**|94.4|
|(0.00,1.00)|**1615.7**|1596.4|**1224.8**|1223.1|**157.6**|147.1|**58.9**|86.3|**86.4**|87.2|

**Table 4, $K=12$**
|JOBS|S-Rewards||L-Rewards||$\Delta{W}$||S-VAR||L-VAR||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|PreferenceVector|OURS|LW|OURS|LW|OURS|LW|OURS|LW|OURS|LW|
|(1.00,0.00)|1610.8|**1614.6**|1231.8|**1232.2**|158.6|**160.7**|**56.8**|60.1|89.8|**87.9**|
|(0.98,0.14)|1609.7|**1610.7**|**1224.6**|1222.9|**154.5**|154.2|**59.1**|60.9|**88.0**|92.3|
|(0.95,0.28)|**1613.5**|1606.3|**1228.2**|1226.7|**158.2**|153.8|**59.3**|65.0|84.4|**79.8**|
|(0.91,0.41)|**1615.6**|1598.9|1223.3|**1231.7**|**156.8**|152.7|**58.1**|70.9|98.6|**88.1**|
|(0.84,0.54)|**1614.1**|1604.9|1218.6|**1220.7**|**153.7**|150.1|**61.4**|65.4|**89.7**|95.1|
|(0.75,0.65)|**1615.2**|1599.0|**1228.0**|1225.3|**159.0**|149.7|**54.9**|76.2|**86.1**|89.3|
|(0.65,0.75)|**1616.4**|1596.2|**1226.5**|1218.8|**158.8**|144.9|**61.5**|77.3|**91.9**|95.6|
|(0.54,0.84)|**1613.4**|1598.5|1223.1|**1229.3**|**155.6**|151.2|**58.6**|83.1|**92.3**|97.6|
|(0.41,0.91)|**1612.9**|1594.3|1222.6|**1224.0**|**155.1**|146.5|**57.3**|84.4|**87.3**|93.6|
|(0.28,0.95)|**1613.0**|1596.9|**1230.5**|1218.7|**159.1**|145.1|**61.5**|82.0|**92.4**|95.5|
|(0.14,0.98)|**1612.2**|1591.8|1214.4|**1224.3**|**150.6**|145.4|**58.2**|86.7|**80.7**|90.2|
|(0.00,1.00)|**1613.0**|1592.4|**1228.2**|1224.3|**158.0**|145.7|**63.8**|87.6|**84.5**|87.6|

Second, we varied the truncation thresholds in normal distributions, setting $\omega_0\sim\mathcal{N}_{[-2,2]}(0,1)$. The corresponding results are give in Table 5, which show that our proposed method consistently performs better.

**Table 5**
|JOBS|S-Rewards||L-Rewards||$\Delta{W}$||S-VAR||L-VAR||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|PreferenceVector|OURS|LW|OURS|LW|OURS|LW|OURS|LW|OURS|LW|
|(1.00,0.00)|1612.9|**1617.4**|**1210.7**|1208.7|147.9|**149.2**|**52.5**|54.4|**101.7**|109.4|
|(0.98,0.17)|**1615.6**|1605.9|1210.3|**1211.1**|**149.1**|144.6|**52.3**|56.1|**95.9**|96.7|
|(0.94,0.34)|**1609.5**|1607.9|**1215.2**|1199.3|**148.5**|139.7|59.1|**55.0**|**104.0**|104.3|
|(0.86,0.50)|**1615.1**|1606.3|1203.8|**1204.7**|**145.6**|141.6|**55.0**|64.1|106.9|**104.6**|
|(0.76,0.64)|**1614.0**|1609.5|1211.4|**1212.1**|**148.8**|146.9|**57.7**|70.8|**91.7**|92.5|
|(0.64,0.76)|**1611.1**|1609.5|**1213.3**|1196.2|**148.3**|139.0|**52.1**|67.5|**100.5**|105.8|
|(0.50,0.87)|**1615.6**|1610.7|**1209.0**|1202.9|**148.4**|143.0|**57.5**|75.1|105.9|**94.5**|
|(0.34,0.94)|**1615.8**|1606.8|**1211.8**|1198.7|**150.0**|138.9|**53.3**|74.1|**96.3**|101.1|
|(0.17,0.98)|**1617.3**|1600.1|**1215.2**|1210.8|**152.4**|141.6|**54.9**|78.3|112.4|**98.1**|
|(0.00,1.00)|**1614.1**|1605.5|1207.7|**1210.0**|**147.0**|143.9|**56.0**|72.2|**94.9**|98.1|


> **W6:** For L-REWARDS in In Table 1, the proposed method is comparable to the LW method. 

**Response:** In this article, **we aim to develop a policy learning method that balances both short and long-term rewards, rather than focusing solely on maximizing either of them.** 

To evaluate the proposed method, we mainly use three metrics: S-REWARD, L-REWARD, and $\Delta W$. S-REWARD and  L-REWARD measures the short and long-term rewards, and $\Delta W$ measures the balanced welfare, indicating the policy's ability to trade-off between multiple rewards. Clearly, **to evaluate an approach that balances both short and long-term reward, $\Delta W$ is the most important metric (lines 283-284).**

In Table 1, the proposed method may not always perform better on L-REWARD, but it shows superior performance for $\Delta W$. **This indicates that the LW method does not achieve a good balance, as it pursues long-term rewards at the expense of short-term rewards. In contrast, our method proves to be Pareto optimal, pursuing long-term rewards while also maximizing short-term rewards.** 


> **W7:** More explanation is needed for the influence of $r$ and $T$ in experiments. 

**Response:** The proposed method consists of two parts: (1) estimation of long-term and short-term rewards;(2) learning the optimal policy using DPPL method. In our experimental, different missing rates $r$ and time steps $T$ mainly affect the generation of long-term outcomes, thus impacting the estimation of long-term rewards, i.e., the first part. However, since the proposed DPPL method focuses on the second part. Thus, we may not observe a significant influence of $r$ and $T$ on the results. This indicates the stability of the proposed method.

---
[6] Wu et al. Policy Learning for Balancing Short-Term and Long-Term Rewards. arXiv, 2024

[7] Cheng et al. Long-term effect estimation with surrogate representation, WSDM, 2021.

[8] Li et al. Trustworthy policy learning under the counterfactual no-harm criterion, ICML, 2023.

Meta-review: This paper proposes a method for learning an optimal policy with both long-term and short-term rewards, which may be missing in the causal inference context. The idea is to formulate a multi-objective optimization problem and obtain a set of Pareto optimal policies by solving a series of subproblems based on preference vectors.

Many reviewers were somewhat misled by the title and the abstract into thinking the paper addressed a general long vs. short trajectory setting in reinforcement learning. I suggest the authors update the title and abstract to explicitly state that the main focus of the paper is on policy learning in causal inference. Additionally, as reviewer ``hVvl`` pointed out, I recommend making the subsection on the suboptimality of the linear weighting method self-contained, rather than referring to [1-4], as general RL readers might not be familiar with the topic in causal inference.

Overall, my recommendation is to accept the paper. I suggest the authors incorporate the feedback in the final submission.

[1] Miettinen. Nonlinear Multiobjective Optimization, 2012
[2] Marler et al. The Weighted Sum Method for Multi-Objective Optimization: New Insights, 2010
[3] Censor. Pareto Optimality in Multiobjective Problems, 1977
[4] Ngatchou et al. Pareto Multi-Objective Optimization, 2005
Decision: Accept (poster)