Learning the Optimal Policy for Balancing Short-Term and Long-Term Rewards

Authors: qinwei  yang
xueqing  liu
yan  zeng
ruocheng  guo
yang  liu
peng  wu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:37:00

Abstract: Learning the optimal policy to balance multiple short-term and long-term rewards has extensive applications across various domains. Yet, there is a noticeable scarcity of research addressing policy learning strategies in this context. In this paper, we aim to learn the optimal policy capable of effectively balancing multiple short-term and long-term rewards, especially in scenarios where the long-term outcomes are often missing due to data collection challenges over extended periods. Towards this goal, the conventional linear weighting method, which aggregates multiple rewards into a single surrogate reward through weighted summation, can only achieve sub-optimal policies when multiple rewards are related. Motivated by this, we propose a novel decomposition-based policy learning (DPPL) method that converts the whole problem into subproblems. The DPPL method is capable of obtaining optimal policies even when multiple rewards are interrelated. Nevertheless, the DPPL method requires a set of preference vectors specified in advance, posing challenges in practical applications where selecting suitable preferences is non-trivial. To mitigate this, we further theoretically transform the optimization problem in DPPL into an $\varepsilon$-constraint problem, where $\varepsilon$ represents the minimum acceptable levels of other rewards while maximizing one reward. This transformation provides intuitive into the selection of preference vectors. Extensive experiments are conducted on the proposed method and the results validate the effectiveness of the method.

Main paper: 
 introduction: 
 Learning an optimal policy for balancing multiple short-term and long-term rewards holds extensive applications across various domains. For instance, content providers can optimize recommendations to avoid short-term clickbait strategies, ensuring sustained user engagement and revenue growth [1] . IT companies can design web pages catering to immediate user preferences while enhancing longterm engagement and satisfaction [2] . Economists explore the effects of early childhood interventions on lifetime earnings, seeking optimal policies (e.g., class size) maximizing short-term test scores and long-term earnings simultaneously [3] . Policymakers can improve job training program design, considering both immediate income impacts and subsequent employment status improvements [4, 5] . Medical practitioners can refine drug prescriptions, considering short-term alleviation and long-term outcomes in chronic diseases like Alzheimer's and AIDS [6] . Marketing professionals can optimize incentive strategies to positively influence customer behavior in both short and long terms [7] .Despite the importance of balancing multiple short-term and long-term rewards, policy learning methods in this area remain largely unexplored. Recent literature [8] employs a linear weighting method to achieve this goal. It combines multiple rewards into a single surrogate reward by weighted summation, which is optimized to learn the optimal policy. However, this strategy has several limitations. First, it can only find optimal solutions in convex regions of objective space and cannot obtain the optimal solutions in non-convex regions [9] . Second, it achieves the optimal solution only when the rewards are independent of each other. When some of the rewards are interrelated, it can only achieve sub-optimal solutions [10] . Consequently, although the linear weighting method is easy to implement, the optimality of its solution cannot be guaranteed when balancing multiple objectives.In this article, we propose a principled policy learning approach for balancing multiple long-term and short-term rewards (objectives). Specifically, we first formulate it as a multiple-objective problem (MOP) and aim to seek the Pareto optimal solutions (policies). A solution is Pareto optimal if improving one objective necessitates worsening other objectives. Then, we propose a novel decomposition-based policy learning (DPPL) method, which involves (1) introducing a set of preference vectors, (2) dividing the whole optimization problem into several subproblems based on the preference vectors, and (3) ultimately achieving different Pareto solutions for the objectives by solving these subproblems. Compared with the linear weighting method, it can obtain Pareto optimal solutions in non-convex regions and is applicable to cases where multiple objectives are interrelated.While the proposed DPPL method can find Pareto optimal policies, it necessitates specifying a set of preference vectors in advance. In practical applications, decision-makers may encounter the challenge of determining which preference vector to choose. To mitigate this concern, we further theoretically transform the optimization problem in DPPL into an ε-constraint problem. This transformation can assist decision-makers in better understanding and selecting preference vectors.The contributions of this paper are summarized as follows.• We formulate the policy learning problem of balancing multiple long-term and short-term rewards as a multi-objective optimization problem and propose a decomposition-based Pareto policy learning (DPPL) method to obtain a set of Pareto optimal policies.• We theoretically establish the connection between the DPPL method and the ϵ-constraint problem, offering an intuitive interpretation of preference vectors and guiding their selection.• We conduct extensive experiments to demonstrate the effectiveness of the proposed method. 
problem formulation: 
 Throughout, we employ bold letters for vectors, uppercase letters for random variables, and lowercase letters for their realization values.We introduce notations to delineate short-term and long-term causal effects. Let A denote the binary treatment indicator, where A = 1 represents the treated group and A = 0 represents the control group. X represents the features observed, S = (S 1 , ..., S I ) ∈ R I and Y = (Y 1 , ..., Y J ) ∈ R J represent the vector of short-term and long-term outcomes, respectively. Both short-term and long-term outcomes are observed after the treatment A, and associations among them may exist.Utilizing the potential outcome framework [11] , we denote S(a) = (S 1 (a), ..., S I (a)) and Y (a) = (Y 1 (a), ..., Y J (a)) for a = 0, 1 as the potential short-term and long-term outcomes under treatment A = a, respectively. We assume that larger short-term and long-term outcomes are preferable. The observed short-term and long-term outcomes S and Y correspond to the potential outcomes of the actual treatment, that is, S = S(A) and Y = Y (A).In real-world applications, long-term outcomes often suffer from missing due to prolonged follow-up periods and budget constraints. In contrast, collecting short-term outcomes is more manageable. Therefore, we presume that all short-term outcomes S are observable, while long-term outcomes Y may be subject to missing. Let R = (R 1 , ..., R J ) ∈ {0, 1} J denote the indicator for observing the long-term outcome Y , where R j = 1 indicates that Y j is observed and R j = 0 indicates that Y j is missing. The missingness of Y would lead to identifiability and estimation problems [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] .In this article, we aim to learn the Pareto optimal policy for balancing multiple correlated shortterm and long-term rewards, which has a wide range of application scenarios [1, 6, 8, 24] . Let π : X → {0, 1} be a policy that maps from the individual context X = x to the treatment space {0, 1}. For a given policy π(θ) = π(X, θ) parameterized by θ, the policy values for the i-th short-term outcome S i and the j-th long-term outcome Y j are defined as,V(θ; s i ) = E[π(θ)S i (1) + (1 -π(θ))S i (0)], i = 1, ..., I V(θ; y j ) = E[π(θ)Y j (1) + (1 -π(θ))Y j (0)], j = 1, ..., J,which are the i-th short-term reward and the j-the long-term reward induced by the policy π(θ).Conventionally, we convert maximization problems to minimization problems. Let V(θ; s i ) = -V(θ; s i ), V(θ; y j ) = -V(θ; y j ). The trade-off among multiple correlated long-term and short-term rewards can be formulated as a multi-objective optimization (MOP) problem given byEQUATIONwhere M = I + J and the symbol ≜ means 'denoted as'. Generally, there is no single solution that can simultaneously optimize all objectives in problem ( 1) and thus we resort to the Pareto optimality. This concept is employed to define the optimal solutions for the MOP problem. Definition 1. (Pareto optimality) (a) Pareto dominance. For two points θ 1 , θ 2 . θ 1 dominates θ 2 if and only if Vm (θ 1 ) ≤ Vm (θ 2 ), ∀ m ∈ {1, ..., M } and Vm ′ (θ 1 ) < Vm ′ (θ 2 ), ∃ m ′ ∈ {1, ..., M } (b) Pareto optimality. θ * is a Pareto optimal point if there is no other solution θ that dominates θ * .Pareto optimality refers to a condition where improving one objective comes at the expense of worsening other objectives. The collection of Pareto optimal solutions is called the Pareto set. Our goal is to derive the set of Pareto optimal solutions (or Pareto optimal policies), each of them providing a distinct optimal trade-off among all objectives.The long-term and short-term rewards are causal parameters that cannot be identified without imposing causal assumptions [25] [26] [27] . Therefore, before seeking the Pareto optimal solutions for balancing multiple long-term and short-term rewards, it is necessary to consider the identification and estimation of long-term and short-term rewards. The proposed method is based on Assumptions 1 and 2 below. Assumption 1 (Strong Ignorability).(a) (S(a), Y(a)) ⊥ ⊥ A | X for a = 0, 1; (b) 0 < e(x) ≜ P(A = 1 | X = x) < 1 for all x.Assumption 1(a) suggests that, given the feature X, treatment assignment A is independent of the potential outcomes S(a) and Y (a). This implies that confounding bias between the treatment A and the short/long-term outcomes (S(a), Y (a)) can be eliminated by conditioning on X [28] . Assumption 1(b) ensures that for the subpopulation of X = x, units with both A = 1 and A = 0 exist. These assumptions are widely used in causal inference [11, 27, [29] [30] [31] [32] [33] [34] [35] .In addition to confounding bias, we also need to address the selection bias induced by the missingness of long-term outcomes [8] . Thus, we further invoke the Assumption 2. Assumption 2 (Missing Mechanism of Long-term Outcome). For a = 0, 1 and j = 1, ..., J,(a) R j ⊥ ⊥ Y j (a) | X, S(a), A = a; (b) 0 < r j (x, a, s) ≜ P(R j = 1 | X = x, A = a, S = s).Assumption 2(a) can be reformulated as R j ⊥ ⊥ Y j | (X, S, A), which means that R j relies only on the observed variables (X, A, S). This assumption also ensures that P(Y j = y|X, S, A, R j = 1) = P(Y j = y|X, S, A, R j = 0). This implies that we can utilize the available data to draw conclusions about the missing long-term outcome. Assumption 2(b) assumes that the long-term outcome for each unit has a non-zero probability of being observed. Assumptions 1 and 2 ensures the identifiability of V(θ; s i ) and V(θ; y j ), as shown in Lemma 1.Lemma 1 (Identifiability of Short-term and Long-term Rewards). For i = 1, ..., I and j = 1, ..., J, (a) under Assumptions 1, the i-th short-term reward V(θ; s i ) is identifiable.(b) under Assumptions 1-2, the j-th long-term reward V(θ; y j ) is identifiable.When we have access to only one short-term outcome and one long-term outcome, Lemma 1 reduces to the identifiability result presented in [8] . In this article, our focus is on achieving the Pareto optimal policy for multiple short-term and long-term rewards. Therefore, for the estimation of V(θ; s i ) and V(θ; y j ), we defer it to Appendix A.3 Pareto Policy Learning for Balancing Short-Term and Long-Term RewardsIn this section, we aim to learn Pareto optimal policies for the MOP problem (1). Section 3.1 gives the motivation for this work and Section 3.2 introduces the proposed policy learning approach. In Section 3.3, we theoretically establishe the connection between the linear weighting method, the MOP problem for a given preference vector, and the ε-constraint problem. This connection offers an intuitive interpretation and guides practitioners in selecting the preference vector. 
motivation: 
 For seeking the optimal policy for balancing short-term and long-term rewards, previous work [8] adopted the linear weighting method. Specifically, the authors formulate the goal asEQUATIONwhere ω m is the pre-specified weight for the m-th objective. The objective function in the optimization problem ( 2) is merely a linear combination of multiple objectives from the MOP problem (1) . Due to its intuitiveness and simplicity, the traditional linear weighting method is commonly used for solving MOP or multi-task learning problems [36] [37] [38] .The linear weighting method simply combines multiple objectives into a single surrogate objective through weighted summation. While simple, it has several limitations. First, the optimal solution is found only in convex regions and not in non-convex regions [9] . Second, an optimal solution can only be achieved if the objectives are independent of each other. That is, if some objectives are interrelated, only a suboptimal solution can be obtained [10] . Thus, it does not guarantee the superiority of the solution or its solution may deviate from the Pareto optimal solution.To overcome the limitations of the linear weighting method in [8] , we first introduce a decompositionbased multi-objective optimization algorithm to achieve the Pareto optimal policy. However, this algorithm relies on pre-specified preference vectors, which are used to express a decision maker's degree of preference for multiple conflicting objectives. In practice, the explanation and selection of preference vectors is a challenging problem. To further tackle this issue, we establish a theoretical relationship between preference vectors and the ε-constraint method [39] . This relationship provides a clear interpretation on preference vectors, assisting in selecting more suitable ones.We introduce the decomposition-based Pareto policy learning (DPPL) method, which can generate the Pareto set containing policies that are optimum from a trade-off perspective. The main idea of the DPPL method is to first decompose the original MOP problem into several constrained subproblems based on a predefined set of preference vectors, and then obtain a set of Pareto optimal policies by solving these subproblems in parallel [40] .For obtaining the Pareto optimal policy for balancing M short-term and long-term objectives, first, we are given a set of K preference vectors {u 1 , u 2 , ..., u K } in R M + . Each element of a preference vector specifies the importance of the corresponding short-term or long-term reward. For each preference vector u k , the corresponding subproblem is given asEQUATIONwhere G k ′ (θ t ) ≤ 0 means that objective space2 of the subproblem is restricted in the subregion Ω k , which is defined byΩ k = {v ∈ R M + |u T k ′ v ≤ u T k v, ∀ k ′ = 1, ..., K}.Geometrically speaking, Ω k represents the set of v that forms the smallest acute angle with u k , which means that the optimal solution of the subproblem can be obtained by only searching the subregion. The preference vectors divide the objective space into different subregions.Solving the subproblem (3) involves the following two steps:• Step (a). Find a reasonable initial solution θ 0 . Specifically, we first randomly generate a solution θ r in the full decision space 3 , and then iteratively update it with the rule θ rt+1 = θ rt + η r d rt , where η r is the step size. For a given θ rt , the descent direction d rt is updated by solving (4).(d rt , α rt ) = arg minEQUATIONwhereI(θ rt ) = {k ′ |G k ′ (θ rt ) ≥ 0, k ′ = 1, ..., K} is index set of all activated constraints, which means V(θ rt ) not in Ω k . The problem (4) aims to find the descent direction d rt for each iteration t and then obtain the initial solution θ 0 such that V(θ 0 ) in Ω k . • Step (b). Solving the subproblem (3). The descent direction d t for the t-th iteration is obtained byEQUATIONwhere I ϵ (θ) = {k ′ |G k ′ (θ) ≥ -ϵ}, and the threshold ϵ is a slack variable used to deal with the solutions near the constraint boundary. We further transform it into a dual problem which will greatly reduce the dimension of decision space. Based on the KKT conditions, we haved t = -( M m=1 λ m ∇ Vm (θ t ) + k ′ ∈Iϵ(θ) β k ′ ∇G k ′ (θ t )). Therefore, the dual problem is given asEQUATIONwhere λ m ≥ 0 and β k ′ ≥ 0 are the Lagrange multipliers for the linear inequality constraints.Step (a) is to find an initial solution θ 0 that is restricted in a subregion of the subproblem (3), and once a feasible solution is found or a predetermined number of iterations is reached, the step stops.For an given initial solution θ 0 , Step (b) is to find the optimal solution θ * for the subproblem (3). We summarize the proposed policy learning approach in Appendix B. Lemma 2 ([41] ). Let (d t , α t ) be the solution to the t-th iteration of problem (5) .EQUATIONLemma 2(a) implies that at the t-th iteration, no direction (d t = 0) can simultaneously improve the performance for all objectives, confirming that the solution θ t satisfies Pareto optimality. Lemma 2(b) suggests that if θ t does not meet Pareto optimality, then the descent direction d t ̸ = 0 serves as the descent direction for all objectives, such that the solution of the next iteration is closer to the Pareto optimal solution. Thus, Lemma 2 demonstrates that we always attain Pareto optimal solutions for each subproblem using the update rule θ t+1 = θ t + η r d t . By solving all subproblems, we can acquire a diverse set of Pareto optimal solutions (or policies) confined to different subregions, even when the multiple objectives are correlated.The DPPL method in Section 3.2 requires a set of pre-specified preference vectors, posing challenges in practical applications where selecting suitable preference vectors is non-trivial. To mitigate this problem, we provide a practical method for decision-makers to select appropriate preference vectors by theoretically establishing the connection between the DPPL method and the ε-constraint problem.We first give a brief introduction to the ε-constraint problem [10] , which is defined as follows,EQUATIONwhere ε m is pre-specified threshold. Compared to the MOP problem (1) and the linear weighting objective (2), a notable advantage of the ε-constraint problem is its interpretation on the threshold ε m , which represents the maximum acceptable value (i.e., the acceptable worst-case scenario) for the m-th objective. In contrast, the weights and the preference vectors in problems ( 1) and ( 2) are not straightforward for relating the resulting values of objectives. Thus, if we can establish the connection between ε = (ε 1 , ..., ε M ) and the preference vector u k , then we can provide powerful guidance for choosing appropriate preference vectors. Theorem 1. For the preference vector u k = (u k1 , ..., u kM ) in problem (1), the weights ω = (ω 1 , ...., ω M ) in problem (2), and the thresholds ε in problem (8), the following statements hold:(a) the connection between ε and ω is given asEQUATIONwhere τ m (X) is the conditional average causal effects for m-th short/long-term outcome,τ m (X) = E[S i (1) -S i (0)|X], if ω m is the weight of V(θ, s i ), E[Y j (1) -Y j (0)|X], if ω m is the weight of V(θ, y j ),h m (X) = E[S i (0)|X], if ω m is the weight of V(θ, s i ), E[Y j (0)|X, S, R j = 1], if ω m is the weight of V(θ, y j ).(b) the connection between ω and u k is given asEQUATIONwhere λ m and β k ′ are defined in Eq. ( 6), andI ϵ (θ) = {k ′ |G k ′ (θ) ≥ -ϵ} defined in Eq. (4).Theorem 1 (see Appendix C for proofs) establishes a link between the preference vector u k and ε through ω in scenarios involving multiple long-term and short-term objectives. Specifically, Theorem 1(a) shows how to estimate the threshold ε for given weights ω, and Theorem 1(b) shows how to assign weights ω via preference vectors u k . This means that for the subproblem determined by preference vectors u k , we can ascertain the maximum acceptable threshold ε based on Theorem 1, thereby offering an intuitive interpretation of the preference vector u k .There are several practical implications with Theorem 1. On one hand, it assists decision-makers in better understanding and selecting preference vectors in practical applications. In practice, we can initially pre-specify a set of preference vectors {u 1 , u 2 , ..., u K } in R M + , then derive the weights ω corresponding to each preference vector u k through Eq. (10) , and finally substitute the obtained weight ω into Eq. ( 9) to calculate the threshold ε. Leveraging the intuitive interpretability of the threshold ε, decision-makers can select the appropriate preference vectors according to their specific requirements. On the other hand, it also provides guidance for specifying ε in the ε-constraint problem (8) . Inappropriate selection of ε for this problem may result in an empty feasible region, yielding empty solutions. By utilizing a set of preference vectors, we can efficiently screen out some reasonable choices of ε and reduce the cumbersome trial-and-error process of testing different ε.In conclusion, by establishing the connection between the DPPL method and the ε-constraint problem, we can harness the advantages of both methods while mitigating their respective weaknesses. 
experiments: 
 Datasets. Following the previous studies [8] , we use two widely used datasets: IHDP and JOBS, for evaluating the performance of the proposed method. The IHDP dataset explores the effectiveness of high-quality home visiting in promoting children's future cognitive development and covers a sample of 747 units, including 139 treated and 608 controlled. In addition, the dataset has 25 characteristics that provide a comprehensive picture of the children and their mothers. The second dataset, JOBS, explores the effects of job training on income and employment status. It consists of 2,570 units (237 treated, 2,333 controlled), with 17 covariates. Note that each unit in both datasets has only one observed outcome from a single treatment, and neither dataset collects long-term outcomes.Simulating Outcome. Consider the case of one long-term reward and one short-term reward. Following the previous data-generation mechanisms [1, 42] , for the n-th unit (n = 1, ..., N ), we simulate the potential short-term outcomes S(0) and S(1) as follows:S n (0) ∼ Bern(σ(w 0 X n + ϵ 0,n )), S n (1) ∼ Bern(σ(w 1 X n + ϵ 1,n )),where σ(•) is the sigmoid function, w 0 ∼ N [-1,1] (0, 1) follows a truncated normal distribution, w 1 ∼ Unif(-1, 1) follows a uniform distribution, ϵ 0,n ∼ N (µ 0 , σ 0 ) and ϵ 1,n ∼ N (µ 1 , σ 1 ). We set µ 0 = 1, µ 1 = 3 and σ 0 = σ 1 = 1 for the IHDP dataset, and we set µ 0 = 0, µ 1 = 2 and σ 0 = σ 1 = 1 for the JOBS dataset. For generating long-term potential outcomes Y (0) and Y (1), we introduce the time step t: we set the initial value at time step 0 as:EQUATION) according to the following equation and we eventually regard the outcome at the last time step T as the long-term outcome,EQUATION.Y t,n (0) ∼ Bern(σ(β 0 X n )+C t-1 t ′ =0 Y t ′ ,n (0))+ϵ 0,n , Y t,n (1) ∼ Bern(σ(β 1 X n )+C t-1 t ′ =0 Y t ′ ,n (0))+ϵ 1,n ,where β 0 is randomly sampled from {0, 1, 2, 3, 4} with probabilities {0.5, 0.2, 0.15, 0.1, 0.05}, 1) , and C = 1/T is a scaling factor. For ϵ 0,n and ϵ 1,n , we set µ 0 = µ 1 = 0, σ 0 = 1 and σ 1 = 3 for the IHDP dataset and set µ 0 = µ 1 = 0, σ 0 = 1 and σ 1 = 1 for the JOBS dataset.β 1 ∼ 4 • N [0,4] (0,Assumption 2 shows that observing indicator R depends on the feature X, the treatment A, and shortterm outcome S. For a given missing rate r, we select the missing indexes for Y and derive the missing indicator R according to the following criterion: calculate the Experimental Details. In this paper, preference vectors are used to quantify an individual's preference for different objectives in the multi-objective optimization problem. For the case of two-objective, we randomly generate 10 unit preference vectors (u 1 , u 2 , • • • , u 10 ), where u k = (u k1 , u k2 ), u k1 = cos(t k ), u k2 = sin(t k ), t k ∈ (0, 1), which implies that the L 2 -norm of the preference vectors is 1, ensuring the consistency and comparability of the preference measures. u k1 and u k2 are the preferences for the short-term objective and the long-term objective, respectively. Each component of the preference vector u k represents the strength or importance of the decision maker's preference for different objectives. Preference vectors are used as weights in the linear weighting method, whereas our method uses them to divide the original problem (1) into several subproblems. Evaluation Metrics. We measure the performance of our proposed method by three metrics: long and short-term rewards, the variance of long and short-term rewards, and the change in welfare. Formally, the short-term reward of the learned policy πm n = 1/D D d=1 (X nd + s n ), n = 1, • • • , N,(X, θ) is V(θ; s) = N n=1 [π(X n , θ)S n (1) + (1 - π(X n , θ))S n (0)], the long-term reward is V(θ; y) = N n=1 [π(X n , θ)Y n (1)+(1-π(X n , θ))Y n (0)].Similar as [42, 43] , the welfare changes are defined as∆W s = N n=1 [(S n (1) -S n (0)) • π(X n , θ)] for the short-term reward, ∆W y = N n=1 [(Y n (1) -Y n (0)) • π(X n , θ)]for the long-term reward, ∆W = 0.5∆W s + 0.5∆W y for the overall balanced-base reward. Among these metrics, ∆W is the most critical here, as it directly measures the balance reward achieved by the learned policy.Policy learning with short-term and short-term reward. We choose MLP as the policy model π(θ), and we average over 50 independent trials of policy learning with the short-term and long-term reward in IHDP and JOBS. We fix the missing ratio r = 0.2 and the time step T = 4. We measure the uncertainty of the model by calculating the variance of the long and short-term reward over 50 experiments, and a smaller variance means a more stable model performance.Performance Comparison. From our previous analyses, the linear weighting method generally achieves the sub-optimal policies. The proposed DPPL method can generate a set of Pareto optimal policies. First, for the long-term reward, the short-term reward, and ∆W , it is not surprising to observe that for most of the preference vectors, DPPL's solutions have better performance. Second, for the variance, our method performs more stable in 50 experiments. Because we will divide the original problem into several subproblems according to preference vectors, and then solve the subproblems in a relatively small subregion to obtain the Pareto optimal solution, whereas the linear weighting method searches the entire objective space. The associated results are displayed in Table 1 . More experimental results with missing ratio r = 0.3 are given in Appendix D. Sensitivity Analysis. We perform the sensitivity analysis of missing ratio r and time step T on JOBS.Our method achieves better performance in all missing rates r = [0.2, 0.3, 0.4, 0.5] with T = 4, and r = 0.2 with time step T = [4, 6, 8, 10] . Our method stably outperforms the linear weighting method under varying r and T , even in scenarios with a high missing ratio or a large time step. This further illustrates the effectiveness of our method. The associated results are displayed in Figure 1 . 2 . This value of -ε is the minimum value of the short-term reward that the decision maker can accept while maximizing the long-term reward. Our results show that as the second component of the preference vector increases, the value of -ε shows a decreasing trend. In essence, this signifies that a decision-maker who emphasizes the long-term reward must necessarily loosen constraints on the short-term reward. In practice, decision makers can determine the threshold based on their specific needs for the short-term reward, and then select the most appropriate preference vector from the set of pre-specify preference vectors with the help of the intuitive interpretability of the threshold according to Table 2 . More experimental results with different missing ratios {0.3, 0.4, 0.5} are provided in Appendix D. Recently, there has been increasing interest in using short-term surrogates to identify and estimate long-term causal effects, such as [4, 5, 7, 13, 44, 45] . In contrast to these previous works focusing on long-term causal effects, this paper aims to balance multiple short-term and long-term causal effects.Trustworthy policy learning. Trustworthy policy learning ensures that the learned policies or models are reliable and dependable for practical applications. Traditional policy learning aims to identify individuals who would maximize the utility function based on their features if treated [46] . Recently, trustworthy policy learning has focused on ensuring that the learned policy adheres to principles such as beneficence, non-maleficence, autonomy, justice, no-harm, and explicability [42, [47] [48] [49] . Various counterfactual-based metrics have been suggested to assess a policy's trustworthiness [42, [50] [51] [52] [53] . In this paper, we complement this series of work by developing a principled policy learning approach that can effectively balance multiple rewards.Multi-objective optimization (MOP). MOP aims to find compromises or trade-offs among multiple possibly contrasting objectives. It is widely used in the field of machine learning such as multi-task learning [40, 54] , neural architecture search [55] , and multi-objective reinforcement learning [56] [57] [58] . We extend these works to a new setting by learning the optimal policy for balancing multiple long-term and short-term rewards. Additionally, we provide a practical method for interpreting and selecting preference vectors with theoretical guarantees. 
conclusion: 
 In this paper, we focus on learning the optimal policy for balancing multiple long-term and shortterm rewards. We reveal the limitations of the previous linear weighting method, which usually results in sub-optimal policies in practice. To address these limitations, we formulate the policy learning problem as a multi-objective optimization problem and then propose the novel DPPL method to learn optimal policies. The DPPL method obtains a set of Pareto optimal policies by solving a series of subproblems based on pre-specified preference vectors, effectively balancing multiple objectives. Furthermore, we theoretically establish the connection between the optimization subproblems in the DPPL method and the ε-constraint problem. This connection aids decision-makers in better understanding and selecting preference vectors. We conducted extensive experiments on two benchmark datasets which validate the effectiveness of our proposed method. A limitation of this work is that it focuses on discrete treatments in identification and estimation (Section 2.3). In some application scenarios, continuous treatments (e.g., price) are of interest. Further investigation is required to extend the proposed method to accommodate such cases.[57] Nan Xu, Nitin Kamra, and Yan Liu. Treatment recommendation with preference-based reinforcement learning. In 2021 IEEE international conference on big knowledge (ICBK), pages 1-8. IEEE, 2021.[58] Jiangjiao Xu, Ke Li, and Mohammad Abusara. Preference based multi-objective reinforcement learning for multi-microgrid system optimization problem in smart grid. Memetic Computing, 14(2):225-235, 2022.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Section 4 for the detailed description of simulating outcome and experimental details. In addition, we provide the supplemental material for datasets and codes in a zip file to ensure easy reproduction of all reported results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report the variance for all experimental results in section 4, by replicating each experiment 50 times.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In the first paragraph of Section 1 (Introduction), we outline various potential applications of our work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: All experiments are conducted on publicly available datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: In Section 4, we provide references for the datasets and the simulation setups of the data-generating process.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: In Section 4, we provide references for the datasets and the simulation setups of the data-generating process. In addition, we provide the supplemental material for datasets and codes in a zip file to ensure easy reproduction of all reported results.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: We don't use a crowdsourcing service.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: For a given policy π(θ), the policy values for the short-term outcome S i and the long-term outcome Y j are defined asV(θ; s i ) = E[π(θ)S i (1) + (1 -π(θ))S i (0)], i = 1, ..., I V(θ; y j ) = E[π(θ)Y j (1) + (1 -π(θ))Y j (0)], j = 1, ..., J,Under Assumptions 1-2, the short-term reward V(θ; s i ) and long-term reward V(θ; y j ) are identified as V(θ;s i ) = E[π(θ)µ i1 (X) + (1 -π(θ))µ i0 (X)], V(θ; y j ) = E[π(θ mj1 (X, S) + (1 -π(θ)) mj0 (X, S)].whereµ ia (X) = E[S i |X, A = a] , mja (X, S) = E[Y j |X, S, A = a, R j = 1] for a = 0, 1.The identifiability results are derived using a similar approach to that outlined in Section 5 of [8] . In addition, for estimating the V(θ; s i ) and V(θ; y j ), [8] proved the efficient bounds of V(θ; s i ) and V(θ; y j ), which we list them below for the sake of self-containedness.Lemma A.1 (Efficiency Bounds of V(θ; s i ) and V(θ; y j ), [8] ). Let Z = (X, A, S, Y ), under Assumptions 1-2, we have that (a) the efficient influence function of V(θ; s i ) is ϕ si -V(θ; s i ), whereϕ si = ϕ si (Z; e, µ i0 , µ i1 ) = {π(θ)µ i1 (X) + (1 -π(θ))µ i0 (X)} + π(θ)A(S i -µ i1 (X)) e(X) + (1 -π(θ))(1 -A)(S i -µ i0 (X)) 1 -e(X) ,and e(X) = P(A = 1|X) is propensity score. The associated semiparametric efficiency bound of V(θ; s i ) is Var(ϕ si ).(b) the efficient influence function of V(θ; y j ) is ϕ yj -V(θ; y j ), where ϕ yj = ϕ yj (Z; e, r j , m j0 , m j1 , mj0 , mj1 )= {π(θ)m j1 (X) + (1 -π(θ))m j0 (X)} + π(θ)AR j (Y j -mj1 (X, S)) e(X)r j (1, X, S) + π(θ)A( mj1 (X, S) -m j1 (X)) e(X)+ (1 -π(θ))(1 -A)R j (Y j -mj0 (X, S))(1 -e(X))r j (0, X, S)+ (1 -π(θ))(1 -A)( mj0 (X, S) -m j0 (X)) 1 -e(X) , m ja (X) = E[Y j |X, A = a, R j = 1]is the regression function for Y j , and r j (A, X, S) = P[R j = 1|X, S, A] is selection score. The associated semiparametric efficiency bound of V(θ; y j ) is Var(ϕ yj ).From Lemma A.1, for a given policy π(θ), it is natural to define the estimators of V(θ; s i ) and V(θ; y j ) asV(θ; s i ) = 1 N N n=1 ϕ si (Z n ; ê, μi0 , μi1 ), V(θ; y j ) = 1 N N n=1 ϕ yj (Z n ; ê, rj , mj0 , mj1 , mj0 , mj1 ).where N is the sample size. All of them can be identified from the observed data. And ê(x), μia (x), mja (x), mja (x, s), and rj (a, x, s) for a = 0, 1 are the estimators of e(x), µ ia (x), m ja (x), mja (x, s) and r j (a, x, s) respectively. for t = 1 to T do6: obtain λ (k) tm ≥ 0, β (k) tk ′ ≥ 0, ∀m = 1, ..., M, ∀k ′ ∈ I ϵ (θ) by solving subproblem (6) 7: calculate direction d (k) t = -( M i=m λ (k) tm ∇ Vm (θ (k) t )+ k ′ ∈I ϵ(θ) β (k) tk ′ ∇G k ′ (θ (k) t )) /d (k) t = -(λ k tm + k ′ ∈I ϵ(θ) β k tk ′ (u k ′ m -u km ))∇ Vm (θ k t ) (step b) 8:update the parameters θ(k) t+1 = θ (k) t + ηd (k) t 9:end for 10: end for 11: Output: The set of solutions for all subproblems with different trade-offs {θ When λ = 0, it is equivalent to finding an optimal policy for minimizing V(θ; y) alone, π * y (θ) = arg min π V(θ; y) = arg min π -E[π(θ)τ y (X)] = I(τ y (X) ≥ 0). When λ = ∞, it is equivalent to finding an optimal policy for minimizing the V(θ; s) alone, π * s (θ) = arg min π V(θ; s) = arg min π -E[π(θ)τ s (X)] = I(τ s (X) ≥ 0). We have the following theorem: Theorem C.1. For the weights ω in problem (A.5), and the thresholds ε in problem (A.1), the following statements hold:• When ε < -E[π * s (θ)S(1) + (1 -π * s (θ))S(0)],the solution of the constrained optimization problem is empty.• When ε ≥ -E[π * s (θ)S(1)+(1-π * s (θ))S(0)],the relationship between λ and α is described as follows:-λ = 0, if ε ≥ -E[π * y (θ)S(1) + (1 -π * y (θ))S(0)]. -λ is the solution of the equation -E[I(τ y (X) + λτ s (X) > 0) • τ s (X) + µ 0 (X)] = ε, if -E[π * s (θ)S(1) + (1 -π * s (θ))S(0)] < ε ≤ -E[π * y (θ)S(1) + (1 -π * y (θ))S(0)].It is important to note that for a given λ, we could solve the value of ε by solving the equation-E[I(τ y (X) + λτ s (X) > 0) • τ s (X) + µ 0 (X)] = ε,as the left side of the equation is a monotone function of λ and the solution is unique, and all the quantities such as τ s (X), τ y (X), and µ 0 (X) are identifiable.Proof. Initially, we recognize that ε cannot be too small so that no policy can satisfy the constraint of V(θ; s) ≤ ε. The optimal policy of minimizing only theV(θ; s) is π * s (θ) = I(τ s (X) ≥ 0). Thus, ε ≥ -E[π * s (θ)S(1) + (1 -π * s (θ))S(0)]. When ε ≥ -E[π * s (θ)S(1) + (1 -π * s (θ))S(0)].First, the optimal policy of minimizing only V(π; y) is given as π * y (θ) = I(τ y (X) ≥ 0). Then, ε ≤ -E[π * y (θ)S( 1) + (1 -π * y (θ))S(0)]. otherwise, the constraint will be invalid and the constrained optimization problem becomes an unconstrained optimization problem with λ = 0. 1) + (1 -π * y (θ))S(0)], we show that the optimal policy π * (θ) parameterized by θ * , for the constrained optimization problem min θ V(θ; y), s.t., V(θ; s) ≤ ε is obtained only when V(θ * ; s) = ε. Below, we prove it with the method of reduction to absurdity.Second, when -E[π * s (θ)S(1) + (1 -π * s (θ))S(0)] ≤ ε ≤ -E[π * y (θ)S(If -E[π * s (θ)S(1) + (1 -π * s (θ))S(0)] ≤ ε ≤ -E[π * y (θ)S(1) + (1 -π * y (θ))S(0)],then there are some units that satisfies {τ s (X) < 0, τ y (X) > 0} that not being assigned treatment by π * (θ); otherwise, the constraint V(θ; s) ≤ ε will be violated. Thus, we could find another treatment policy π * that assigns more treatment to the units with {τ s (X) < 0, τ y (X) > 0}, which yields a lower V(θ; y) but increases the V(θ; s). That is, π * will lead to a V(θ; s) closer to ε but has a lower V(θ; y) than π * , thus, π * is not the optimal policy, which contradicts its definition of π * . Thus, the constrained optimization problem becomesmin θ V(θ; y), s.t., V(θ; s) = ε.By introducing the Lagrange multiplier β, π * satisfies where ω m is the pre-specified weight for the m-th reward. We have the following theorem: Theorem 1. For the preference vector u k in problem (1), the weights ω in problem (2), and the thresholds ε in problem (8), the following statements hold:π * = arg min π V(θ; y) + β V(θ; s) = I(τ y (X) + βτ s (X) > 0), where β is the solution of V(θ * ; s) = ε, i.e., -E[I(τ y (X) + βτ s (X) > 0)τ s (X) + µ 0 (X)] = ε.(a) the connection between ε and ω is given as-E[I(τ l (X) + ω m ω l τ m (X) > 0) • τ m (X) + h m (X)] = ε m , for m = 1 • • • M, and m ̸ = l,where τ m (X) is the conditional average causal effects for m-th short/long-term outcome,EQUATIONh m (X) = E[S i (0)|X], if ω m is the weight of V(θ, s i ), E[Y j (0)|X, S, R j = 1], if ω m is the weight of V(θ, y j ),and I(•) is the indicator function.(b) the connection between ω and u k is given asω m = λ m + k ′ ∈Iϵ(θ) β k ′ (u k ′ m -u km ), for m = 1, • • • , M,where λ m and β k ′ are defined in Eq.( 6),I ϵ (θ) = {k ′ |G k ′ (θ) ≥ -ϵ}Proof. First, for the Theorem1(a), combining the TheoremC.1, more generally, for the ε-constraint problemmin θ Vl (θ), s. t. Vm (θ) ≤ ε m for all m = 1, . . . , M, m ̸ = l, (A.5)and the linear weighting optimization problemmin θ V(θ) = M i=m ω m Vm (θ), (A.6)where ω m is the pre-specified weight for the m-th reward. By mathematical induction, we have:-E[I(τ l (X) + ω m /ω l τ m (X) > 0)τ m (X) + h m0 (X)] = ε i , m = 1 • • • M, and m ̸ = l where τ m (X) = τ si = E[S i (1) -S i (0)|X], if ω m is the weight of V(θ, s i ), τ yj = E[Y j (1) -Y j (0)|X], if ω m is the weight of V(θ, y j ), h m0 (X) = µ i0 (X) = E[S i |X, A = 0], if ω m is the weight of V(θ, s i ), mj0 (X, S) = E[Y j |X, S, A = a, R j = 1],if ω m is the weight of V(θ, y j ), This completes the proof for Theorem 1(a)Second, for the Theorem (b), motivated by [40] , for constraint problemEQUATIONwe have∇G k ′ (θ t ) = (u k ′ -u k ) T ∇ V(θ t ) = M m=1 (u k ′ m -u km )∇ Vm (θ t ). (A.8)Base on KKT conditions, we haved t = -( M m=1 λ m ∇ Vm (θ t ) + k ′ ∈I ϵ(θ) β k ′ ∇G k ′ (θ t )), M m=1 λ m + k ′ ∈I ϵ(θ) β k ′ = 1, (A.9)where λ m ≤ 0 and β k ′ ≤ 0 are the Lagrange multipliers. Then, the dual problem is given asmax λm,β k ′ - 1 2 || M m=1 λ m ∇ Vm (θ t ) + k ′ ∈Iϵ(θ) β k ′ ∇G k ′ (θ t )|| 2 s.t. M m=1 λ m + k ′ ∈Iϵ(θ) β k ′ = 1, λ m ≥ 0, β k ′ ≥ 0, ∀m = 1, ..., M, ∀ k ′ ∈ I ϵ (θ).(A.10) Substituting Eq.(A.8) into Eq.A.9, we haved t = -( M m=1 λ m ∇ Vm (θ t ) + k ′ ∈Iϵ(θ) β k ′ ( M m=1 (u k ′ m -u km )∇ Vm (θ t ))) = -(λ m + k ′ ∈I ϵ(θ) β k ′ (u k ′ m -u km ))∇ Vm (θ t ) (A.11)For the problem(A.6), d t is the negative gradient direction. Thus, we haveV(θ) = M m=1 ω m Vm (θ), where ω m = λ m + k ′ ∈I ϵ(θ) β k ′ (u k ′ m -u km ), (A.12)where λ m and β k ′ is obtained from Eq.(A.10). This shows that the DPPL method can be transformed into the linear weighting method. This completes the proof for Theorem 1(b)In the following, we show more experimental result with missing ratio r = 0.3 under IHDP and JOBS datasets, in table D1 .In additional, We show the corresponding ε value for each preference vector with different missing ratio {0.3, 0.4, 0.5} under IHDP and JOBS datasets, in tables D2, D3 and D7. Justification: We provide the supplemental material for datasets and codes in a zip file to ensure easy reproduction of all reported results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.V(θ) is the objective vector, and the space spanned by the objective vectors is called the objective space Ω.The parameter vector θ represents the decision variable and the space spanned it is called the decision space. 


Decision: This paper proposes a method for learning an optimal policy with both long-term and short-term rewards, which may be missing in the causal inference context. The idea is to formulate a multi-objective optimization problem and obtain a set of Pareto optimal policies by solving a series of subproblems based on preference vectors.

Many reviewers were somewhat misled by the title and the abstract into thinking the paper addressed a general long vs. short trajectory setting in reinforcement learning. I suggest the authors update the title and abstract to explicitly state that the main focus of the paper is on policy learning in causal inference. Additionally, as reviewer ``hVvl`` pointed out, I recommend making the subsection on the suboptimality of the linear weighting method self-contained, rather than referring to [1-4], as general RL readers might not be familiar with the topic in causal inference.

Overall, my recommendation is to accept the paper. I suggest the authors incorporate the feedback in the final submission.

[1] Miettinen. Nonlinear Multiobjective Optimization, 2012
[2] Marler et al. The Weighted Sum Method for Multi-Objective Optimization: New Insights, 2010
[3] Censor. Pareto Optimality in Multiobjective Problems, 1977
[4] Ngatchou et al. Pareto Multi-Objective Optimization, 2005

Reviews: review: paper_summary: paper_summary: This paper proposes a framework for solving multi-objective optimization problems: multi-objective optimization problems are divided into sub-problems in different regions by setting different preference vectors. The parameter optimization direction of the sub-problem can be easily solved by transforming it into a dual problem through the KKT condition, and a Pareto optimal solution of the original problem can be obtained by solving the sub-problem. This paper uses this framework to balance the optimal strategy learning under multiple short-term rewards and long-term rewards and achieves better and more stable performance than the traditional linear weighted method in the constructed experimental environment.
strength_weakness: strength_weakness: 1. This paper mainly proposes an important multi-objective optimization algorithm and compares it with two existing algorithms in theory. However, the title of this paper seems to be just a specific application scenario of the algorithm. In what other scenarios can this algorithm be applied?
2. The experimental part is mainly conducted in a constructed environment, and it is unclear how difficult it is in the field of causal inference.
3. The v in line 171 is missing \bar. In Appendix B, t in line 5 of Algorithm 1 should start from 0.
questions: questions: 1. How to deal with a situation where long-term rewards are missing? Should the data be ignored when solving the network?
I think the optimization method proposed in this paper and the connection with related methods are very interesting, and I will improve my score as appropriate.
limitations: limitations: Have addressed.
paper_summary: paper_summary: This paper studies the tradeoff between short-term and long-term rewards. The authors formulate the policy learning problem as a multi-objective optimization problem and propose a decomposition-based Pareto policy learning method. I only had experience in reinforcement learning in robotics five years ago. I tried my best to understand the paper, but I am not sure about my rating and comments.
strength_weakness: strength_weakness: - Only the linear weighting method is used as the baseline. I am wondering if there are any other methods that can be used for comparison. If not, why? Since both IHDP and JOBS are widely used.
questions: questions: Please see the weakness.
limitations: limitations: Yes
paper_summary: paper_summary: This paper attempts to address the challenge of learning the optimal policy for balancing multiple long-term and short-term rewards. The authors point out that the existing linear weighting method leads to a sub-optimal policy. To address this limitation, the authors propose formulating formulate the problem as a multi-objective optimization problem. They utilize the Lagrange algorithm to use preference vectors to solve the formulated multi-objective optimization problem and aim to learn the policy to meet Pareto optimization. In order to decide the preference vectors, the authors propose establishing the connection between the optimization problems and the ε-constraint problem. Experiments on IHDP and JOBS demonstrate the efficacy of the proposed method.
strength_weakness: strength_weakness: 1. The authors point out that the linear weighting method is suboptimal. However, there is no explanation in the method section or corresponding experiments to demonstrate that the proposed method (i.e. DPPL) is optimal.

2. In line 38, the authors claim that when some of the rewards are interrelated, the linear weighting method can only achieve a suboptimal solution. The claim may not be rigorous as the linear weighting method might be able to model the relationship among the rewards. More explanation and experiments are required.

3. In line 95, the definition of Pareto optimality, the condition for Pareto optimality by the author is to find the $\theta$ that makes all $\bar{\mathcal{V}}$ optimal. However, is it possible that the $\theta$ is not optimal for some $\bar{\mathcal{V}}$ but is optimal for the overall $\bar{\mathcal{V}}$?

4. Some mathematical symbols and proprietary terms in the paper are not explained clearly. For example, what does the $e$ in line 110 mean? What does MOP represent? Does MOP represent multi-objective problems? What do $v$ and $R_{+}$ mean in line 171? What does the KKT condition mean? Is it the KKT condition in the Lagrange algorithm? What is the difference between the two descent directions $d_{rt}$ and $d_t$? There are many similar situations in the paper. I suggest providing necessary explanations for each noun and symbol that appears for the first time.

5. In section Simulating Output and section Experimental Details, many parameters are defined by the authors themselves, but most of them do not have reasons or ablation experiments. For example, why is the number of preference vectors 10? In Line 253 to Line 254, why are some parameters truncated normal distributions and some Gaussian distributions?

6. In Table 1 on the L-REWARDS metric, the proposed method is comparable to the linear weighting method. However, the authors claim that for most of the preference vectors, DPPL's solutions have better performance.

7. In Figure 1, it seems that the effect on the $\delta{w}$ from the missing rate and T is not obvious for either the proposed method or LW. More explanation is needed.
questions: questions: Please see the comments above.
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a new way to balance multiple rewards with some long-term rewards potentially missing. It does so by using Pareto Policy Learning of optimizing each reward subject up to the tradeoff frontier. This can be more practical than simple linear weighting since the linear weighting strategy applies the constant weight regardless of the amount of conflict between pairs of objectives. Empirically, the papers show that the approach is superior to linear on two synthetic tasks with some real data. Overall I think the paper is promising and adding more realistic empirical evaluation can add values to the current state of the paper.
strength_weakness: strength_weakness: - The main weakness of the paper is that the experimentation is rather limited. The experiment uses partial real data with synthetic generation of short-term and long-term rewards. For example, in robotic planning, the authors could show how their approach helps balance the long-term reward (e.g. goal reaching) / short-term reward (e.g. minimizing jerk). This is just an example, but including other more real-world planning and RL problems would seem beneficial.
- It seems that compared to linear weighting, the proposed method seeks more short-term reward but is not necessarily better in terms of long-term reward. It may not be a weakness but reading the table does strike me that the method is more “short-sighted.”
questions: questions: - The separation between short-term and long-term reward is practically meaningful, but mathematically the only difference is that one reward can be missing and the other is fully observable, since the Pareto Policy Learning treats all objectives the same. Do we necessarily need these separate definitions? Can we put everything as a long-term reward that can be missing sometimes?
- How are the weighting and preference vectors chosen? Have the author considered running a sweep over different configurations to compare against linear weighting? Apologies if I overlook this detail from the paper.
limitations: limitations: - The limitation of the overall framework are mentioned but there is no much detail perhaps due to space constraint.

Comments: title: Kindly Reminder
Kindly Reminder
Kindly Reminder
Kindly Reminder
Gratitude for Review and Invitation for Further Discussion
comment: Dear Reviewer B23s,

We are deeply grateful for the time and effort you have invested in reviewing our paper. We have endeavoured to address your questions as thoroughly as possible. As the discussion period draws to a close, we kindly inquire if there are any further concerns or questions that we might assist with.

Sincerely,

Authors #17045
Dear Reviewer hVvL,

We are deeply grateful for the time and effort you have invested in reviewing our paper. We have endeavoured to address your questions as thoroughly as possible. As the discussion period draws to a close, we kindly inquire if there are any further concerns or questions that we might assist with.

Sincerely,

Authors #17045
Dear Reviewer xruw,

We are deeply grateful for the time and effort you have invested in reviewing our paper. We have endeavoured to address your questions as thoroughly as possible. As the discussion period draws to a close, we kindly inquire if there are any further concerns or questions that we might assist with. 

Sincerely, 

Authors #17045
Dear Reviewer  ouKL,

We are deeply grateful for the time and effort you have invested in reviewing our paper. We have endeavoured to address your questions as thoroughly as possible. As the discussion period draws to a close, we kindly inquire if there are any further concerns or questions that we might assist with. 

Sincerely, 

Authors #17045
Dear Reviewers ouKL, xruw, hVvL, and B23s, 

We are deeply grateful to all reviewers who dedicated their valuable time and effort to evaluating our work.

As the rebuttal deadline approaches, we would like to express our appreciation once again and invite any additional questions or concerns you may have. Your insights are crucial to refining our research, and we welcome further discussion.

Sincerely