ROIDICE: Offline Return on Investment Maximization for Efficient Decision Making

Authors: woosung  kim
hayeong  lee
jongmin  lee
byung-jun  lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:14:34

Abstract: In this paper, we propose a novel policy optimization framework that maximizes Return on Investment (ROI) of a policy using a fixed dataset within a Markov Decision Process (MDP) equipped with a cost function. ROI, defined as the ratio between the return and the accumulated cost of a policy, serves as a measure of efficiency of the policy. Despite the importance of maximizing ROI in various applications, it remains a challenging problem due to its nature as a ratio of two long-term values: return and accumulated cost. To address this, we formulate the ROI maximizing reinforcement learning problem as a linear fractional programming. We then incorporate the stationary distribution correction (DICE) framework to develop a practical offline ROI maximization algorithm.
Our proposed algorithm, ROIDICE, yields an efficient policy that offers a superior trade-off between return and accumulated cost compared to policies trained using existing frameworks.

Main paper: 
 introduction: 
 In economics, Return on Investment (ROI) is a financial metric used to evaluate the profitability of an investment relative to its cost. The concept of ROI originates from the work of [5] and is widely regarded as a valuable metric by the majority of marketing managers [4] . ROI is calculated by dividing the profit generated from the investment by the cost of the investment, and it is a key indicator for evaluating the efficiency and effectiveness of various economic decisions and strategies [17] . Maximizing ROI is crucial as a decision with a high ROI yields a higher return at a relatively lower cost.While ROI is a compelling metric for optimization, ROI maximization for decision making has not been actively explored within the field of machine learning. Previous studies on ROI maximization [3] have focused on multi-armed bandit scenarios, where reward and cost are immediately accessible following a decision. When it comes to sequential decision-making setups like the Markov Decision Process (MDP) that additionally considers a cost function, optimizing a sequence of decisions or a policy to maximize the ratio between return and accumulated cost becomes far more challenging and remains mostly unexplored. This optimization contains two major difficulties: the fractional relationship between the return and accumulated cost of a policy, and the fact that both are longterm quantities collected after a sequence of decisions or actions of the agent. The issues make ROI maximization via existing policy optimization techniques not straightforward, as it includes computing the gradient of the ratio between the two expected values.For effective optimization of ROI, we refer to the dual formulation of V-LP [16] , which represents policy optimization in reinforcement learning (RL) as linear programming in terms of the stationary distribution. The stationary distribution of a policy is the discounted sum of probabilities of the state-action pairs visited by the agent following the policy. Leveraging the property of stationary distribution, which allows for easy estimation of both return and accumulated cost, we derive ROI-LP, linear programming problem for ROI maximization. In this paper, we focus specifically on the offline setting and aim to derive an offline ROI maximization algorithm. The offline setting, which involves optimizing a policy using a fixed dataset of pre-collected experiences without further environmental interactions (e.g., offline RL), has been actively studied recently due to its compelling use cases [2, 7, 9, 10, 11, 20] . Building on the derived ROI-LP, we incorporate a convex regularization to address the distribution shift inherent in offline learning. This leads to our proposed algorithm, ROIDICE, which achieves offline ROI maximization via stationary distribution correction estimation.Through a series of diverse experiments, we compare ROIDICE to algorithms from other offline policy optimization frameworks, including offline RL and offline constrained RL. We demonstrate that maximizing the ROI of a policy leads to a different, more efficient behavior compared to frameworks that focus solely on maximizing policy return. In summary, our contributions are threefold:• Formulation of ROI maximization framework in an MDP with a cost function.• Derivation of ROIDICE, an offline ROI maximization algorithm that optimizes the ROI of a policy using a fixed dataset of pre-collected experiences. • Comparison with existing offline policy optimization frameworks, showing that ROIDICE optimizes ROI and improves policy efficiency across various domains. 
backgrounds: 
 Markov Decision Process with a cost function We consider an infinite-horizon discounted Markov Decision Process (MDP) with a cost function, represented as a tuple M = ⟨S, A, T, r, p 0 , γ⟩. Here, S is the set of states, A is the set of actions, T (s ′ |s, a) denotes the transition probability from stateaction pair (s, a) to the next state s ′ , and r(s, a) is the reward function for state-action pair (s, a). p 0 (s) is the initial state distribution, and γ is the discount factor. Additionally, we introduce a cost function c(s, a) > 0 for all state-action pairs (s, a).An agent begins at an initial state following p 0 (s) and repeats the process of collecting reward r(s, a) and incurring cost c(s, a) by taking action a in state s and transitioning to the next state s ′ . The policy π(a|s) represents the distribution of actions to be taken by the agent in state s. The performance of the policy π can be evaluated with the expected return R π = E π [ ∞ t=0 γ t r(s t , a t )] and the expected accumulated cost C π = E π [ ∞ t=0 γ t c(s t , a t )], which are the expected cumulative sums of discounted rewards and costs over rollouts of the policy π.The main goal of reinforcement learning (RL) is to optimize policy π to maximize expected return R π collected by the agent. Constrained RL optimizes the policy to maximize its expected return R π while constraining its expected accumulated cost C π to be lower than a given threshold C threshold .Stationary Distribution and Linear Programming (LP) formulation of RL The stationary distribution of policy d π represents the discounted sum of probabilities of the state-action pair (s, a) visited by an agent following policy π, i.e., d π (s, a) := (1 -γ) ∞ t=0 γ t Pr(s t = s, a t = a). Using the stationary distribution, R π and C π can be expressed as linear combinations of the stationary distribution and the reward and the cost respectively: (s, a) . This characteristic allows the RL problem to be formulated as a linear programming (LP), known as the dual of V-LP in [16] .R π = 1 1-γ s,a d π (s, a)r(s, a) and C π = 1 1-γ s,a d π (s, a)cEQUATIONEQUATIONwhere(B * d)(s) = a d(s, a) and (T * d)(s) = s,ā T (s|s, ā)d(s, ā).Constraint (2), known as Bellman flow constraint, is a requirement that any stationary distribution must satisfy. The condition can be interpreted as ensuring that the probability of state-action pairs leaving state s matches the total probability of state-action pairs entering s, plus the probability of state s initiating a trajectory. Once the optimal stationary distribution d * is obtained, optimal policy π * can be extracted from it by π * (a|s) = d * (s, a)/ a ′ d * (s, a ′ ) ∀s, a.Offline RL and DICE-RL Framework In this paper, we assume an offline policy optimization setting where interaction between the agent and environment is not allowed. Instead, the agent is provided with a fixed dataset of experiences D = {(s i , a i , r i , c i , s ′ i )} N i=1 , consisting of N transition samples, to optimize its policy. A unique challenge in offline RL is balancing between policy optimization and distribution shift, as it is not possible to gather samples from the optimized policy.In [12] , regularized policy optimization is proposed to penalize return maximization (1) s.t. (B * d)(s) = (1 -γ)p 0 (s) + γ(T * d)(s) ∀swhere α is a hyper-parameter that controls the trade-off. In the constrained setting studied in [13] , a cost constraint s,a d(s, a)c(s, a) ≤ C threshold is added to the problem.Previous studies have leveraged the characteristics of the above convex optimization problem to develop practical offline RL algorithms. They follow a similar approach by applying a Lagrangian multiplier ν(s) to the Bellman flow constraint (2), reformulating the Lagrangian dual of the problem into the main loss function of the algorithms. We refer to this family of algorithms as the DICE-RL framework. Detailed explanations of the loss functions and the full derivations of related DICE-RL algorithms (OptiDICE [12] and COptiDICE [13] ) are provided in Appendix A.Linear-fractional Programming Linear-fractional programming is defined as maximizing a ratio of two affine functions over a polyhedron. Its standard form is given by:EQUATIONThe domain of f is restricted to ensure the denominator remains positive, i.e., {x|d T x + β > 0}, as a zero denominator would result in an infeasible solution.Assuming the region satisfying the constraints is non-empty and bounded, any linear-fractional programming can be transformed into equivalent linear programming using the Charnes-Cooper transformation [1] . The main idea of the transformation is to replace variable x with two new variables t ≥ 0 and y, such that t(d T x + β) = 1 and y = tx. This change of variables eliminates the denominator of f (x), resulting in the following linear programming problem:max y,t≥0 c T y + αt s.t. Gy ⪯ th, Ay = tb, d T y + βt = 1.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Section 6Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
roi maximization in linear programming form (roi-lp): 
 In this section, we start with a return on investment (ROI) maximization problem by substituting the objective of the dual of V-LP from return R π to ROI. This results in a linear-fractional programming problem, as the objective is the ratio of two linear functions representing return and accumulated cost.We then apply the Charnes-Cooper transformation to derive an equivalent linear programming (LP), ROI-LP.We begin by defining the ROI of policy π in a Markov Decision Process (MDP). The ROI of policy π is the ratio between the return R π and its accumulated cost R π , which can be expressed in terms of its stationary distribution d π : ROI(π) = Rπ Cπ = s,a dπ(s,a)r(s,a) s,a dπ(s,a)c(s,a) . Substituting the objective of the dual of V-LP (1) with ROI, we get our ROI maximization problem, which adheres to the standard form of linear-fractional programming in (3). EQUATIONROI-LP We apply the Charnes-Cooper transformation to reformulate the ROI maximization problem (4) into an equivalent linear programming problem, ROI-LP. This involves introducing a non-negative variable t ≥ 0 and replacing d(s, a) with d ′ (s, a) and t, which are defined as:EQUATIONand thus d(s, a) = d ′ (s, a)/t. Substituting d, ROI-LP, the linear programming formulations for maximizing ROI, is given by:EQUATIONwhere the equality constraint ( 6) is newly introduced to ensure that (5) is satisfied. Once the optimization on ROI-LP is complete, optimal stationary distribution d * (s, a) is obtained by dividing d ′ * (s, a) with t * . Remark. When c(s, a) is a fixed constant C for all states and actions, the optimal solution of ROI-LP becomes equivalent to that of the dual of V-LP. In this case, the optimal t becomes1 C as per (5) , since any stationary distribution satisfies s,a d(s, a) = 1. Consequently, the optimal d ′ * becomes d * from the dual of V-LP scaled by t. This special case illustrates that ROI maximization simplifies return maximization when the accumulated cost is constant, regardless of the policy.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: Appendix G Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
offline roi maximization: 
 In this section, we derive an offline ROI maximization algorithm that extends ROI-LP to also include the trade-off for distribution shifts. We formulate a convex optimization problem for offline ROI maximization and use it to derive ROIDICE, the first offline algorithm to optimize policy efficiency in terms of ROI given a fixed dataset.To formulate an offline ROI maximization problem, we add a convex regularization to ROI-LP representing a distribution shift. The approach is similar to the DICE-RL framework [13] , which incorporates an f -divergence between the stationary distribution of trained policy d and behavior policy d D into the dual of V-LP. Our regularized ROI maximization framework is given by:EQUATIONs,aEQUATIONwhere the design of f (x, t) will be provided later in this subsection. We denote a) , t). The primary difference between our proposed problem and DICE-RL framework lies in the convex regularization. The f -divergence D f (d||d D ) from DICE-RL framework is not directly applicable to d ′ (s, a), as d ′ (s, a) is not a valid probability distribution; it is derived by scaling the stationary distribution d(s, a) by t. However, naively incorporating D f ( d ′ t ||d D ) results in the loss of the convexity, as f ( x t ) of f -divergence is generally not convex with respect to t. To this end, we design a new convex regularization that measures the amount of distribution shift while not breaking the convexity of the problem. We begin by outlining three conditions that the regularization should satisfy.D f,t (d ′ , d D ) = s,a d D (s, a)f ( d ′ (s,a) d D (s,2. f (x, t) should be zero when x = t. 3. f (x, t) should be well-defined on x > 0 and t > 0.The first condition is that the problem must maintain convexity with respect to both x and t. The second condition is that it must effectively regularize the distribution shift. Specifically, D f,t (d ′ , d D ) should be minimized to zero when the stationary distribution d(s, a) = d ′ (s, a)/t matches d D (s, a). The last condition is to avoid infeasible optimization scenarios, e.g., negative values inside a log function. We introduce two convex functions as examples.f 1 (x, t) = 1 2 (x -t) 2 , f 2 (x, t) = 1 2 (x -t) 2 x ≥ t x t log( x t ) -x t + 1 0 < x < t From f (x) = 1 2 (x -1) 2 of χ 2 -divergence,we can obtain the first function f 1 (x, t) by shifting it by t -1. We introduce a softened variant of f 1 , denoted as f 2 , where a scaled version of the function f (x) = x log x -x + 1 (arising from the KL divergence) is seamlessly combined with the left segment of f 1 . Although f 2 (x, t) is not entirely convex with respect to t, it serves as a valid regularization term, as f 2 (x, t) = 0 holds exclusively when x = t. Remark. A unique characteristic of the proposed regularization D f,t (d ′ , d D ) is that its magnitude is affected not only by the degree of distribution shift but also by the trained policy's accumulated cost. We demonstrate this by comparing the magnitude ofD f,t (d ′ , d D ) and f -divergence D f ( d ′ t ∥d D ) that corresponds to typical regularizer of DICE-RL framework. In the case of χ 2 -divergence, D f,t (d ′ , d D ) = t 2 D f ( d ′ t ||d D ) as D f ( d ′ t ||d D ) = s,a d D (s, a) 1 2t 2 ( d ′ (s,a) d D (s,a) -t) 2 .Since t represents the inverse of the expected accumulated cost (5) , the strength of the regularization with D f,t (d ′ , d D ) becomes stronger than the actual f -divergence when the expected accumulated cost of the trained policy is less than 1.We are now prepared to derive ROIDICE, an offline algorithm for ROI maximization based on the Regularized ROI maximization framework. To enforce the constraints ( 7) and ( 8 We follow a similar derivation process as outlined by [12] and obtain a closed-form solution of w * ν,µ,t (s, a):EQUATIONEQUATIONwhere e ν (s, a) = r(s, a)+ γ s ′ T (s ′ |s, a)ν(s ′ ) -ν(s) and (f ′ ) -1 (y, t) is the inverse function of ∂ ∂x f (x, t)with respect to x. Loss functions of ν, µ and t are obtained by substituting w * ν,µ,t (s, a) into L(ν, µ, w, t):min ν E s∼p0 [t(1 -γ)ν(s)] + E (s,a)∼d D w * ν,µ,t (s, a)(e ν (s, a) -µc(s, a)) -αf (w * ν,µ,t (s, a), t) , min µ E (s,a)∼d D w * ν,µ,t (s, a)(e ν (s, a) -µc(s, a)) -αf (w * ν,µ,t (s, a), t) + µ, max t≥0 E s∼p0 [t(1 -γ)ν(s)] -E (s,a)∼d D αf (w * ν,µ,t (s, a), t) .After the minimization on ν, µ and t is complete, we apply policy extraction techniques from [12] to obtain the policy that generates optimal stationary distribution d * (s, a) = 1 t w * ν,µ,t (s, a)d D (s, a). In this paper, we adopt weighted behavior cloning method, whose loss function is given as,EQUATIONThe more detailed derivation and algorithm of ROIDICE can be found in Appendix B. ROIDICE Behavior 
experiments: 
 In this section, we demonstrate that the offline policy obtained from ROIDICE achieves a superior trade-off between return and accumulated cost compared to other RL approaches such as offline RL and constrained offline RL, in both finite and continuous domains.We test ROIDICE in a Random MDP setting similar to [12] . A tabular MDP with |S| = 50 and |A| = 4 is randomly generated and a behavior policy that achieves 90% of the optimal return is used to gather fixed datasets with {10, 20, 50, 100, 1000, 2000} trajectories. We estimate the empirical transition probability T (s ′ |s, a) with the given samples and use it to run a tabular version of ROIDICE. Details of the experiments are provided in the Appendix C. We compare ROIDICE with DICE-RL on other offline policy optimization schemes, OptiDICE [12] for offline RL and COptiDICE [13] for offline constrained RL.vs. Offline RL In Figure 1a , we compare the performance of ROIDICE with OptiDICE. While OptiDICE achieves the highest return, it incurs a higher accumulated cost than ROIDICE. This is expected, as RL does not account for the accumulated cost it incurs. In contrast, ROIDICE optimizes the policy's ROI, resulting in a highly efficient policy that delivers a substantial return at a relatively lower accumulated cost.vs. Offline Constrained RL In Figure 1b , we compare the performance of ROIDICE with COptiDICE. Offline constrained RL aims to maximize return while keeping the accumulated cost below a specified threshold C π ≤ C threshold . As demonstrated in Appendix A, COptiDICE optimizes its policy using a penalized reward function r(s, a) -λ c c(s, a), where λ c is the Lagrange multiplier for the cost constraint. To ensure satisfaction of the cost constraint, λ c is updated to increase when the constraint is violated. To set the cost thresholds for our experiment, we obtain the accumulated cost of the ROIDICE policy on the Maximum Likelihood Estimate (MLE) MDP and multiply it with [0.7, 0.9, 1.0, 1.1, 1.3]. The left side of the figure shows the true ROI performance of the agents, while the right side shows the ROI performance on the MLE MDP, representing the performance agents expect. The cost thresholds from the MLE MDP are based on the offline assumption which excludes the interaction with the true MDP.With a sufficient number of trajectories in the offline dataset, COptiDICE with a cost multiplier of 1.0, which corresponds to the same cost budget as the optimized ROIDICE policy, achieves the best ROI among the tested multipliers. Both increasing or decreasing the cost threshold result in reduced While it is theoretically possible to identify a policy with optimal ROI using a constrained RL algorithm, doing so requires multiple runs with different cost thresholds to determine the best one. This process can become computationally intractable and less accurate as the complexity of the problem increases. In contrast, a single run of ROIDICE is sufficient to find a policy with high ROI without the need to explicitly search for the right cost threshold.We evaluate our algorithm's performance across diverse domains using datasets from D4RL [6] (CC BY 4.0) and NeoRL [18] (CC BY 4.0). From the D4RL benchmark, we select three locomotion tasks (Hopper, Halfcheetah, Walker2D), and from NeoRL, we include a stock trading task (FinRL).To assess ROIDICE's capability to maximize Return on Investment (ROI), we conduct comparative analyses with various unconstrained or constrained offline RL algorithms. We utilize offline datasets from D4RL and NeoRL, which are collected using data collection policies that do not consider cost. Similar to the random MDP experiments, we include OptiDICE and COptiDICE for comparison. Additionally, we incorporate the Constrained Decision Transformer (CDT, [15] ) to represent a recent approach that leverages the power of generative models on RL.In the three locomotion environments, observations include the positional values and velocities of various body parts, with actions corresponding to the torques exerted between links. The common reward function used in recent benchmarks consists of three components: incentive for forward running speed, incentive for maintaining a healthy state, and penalty for applying torques on joints.In this experiment, we use the control penalty as the cost function, aiming to train the agent to strive for efficient energy consumption from an ROI perspective. We used only forward running speed as our reward function.FinRL, the time-driven stock trading simulator, enables us to replicate live stock markets using real market data. This allows us to assess our algorithm's proficiency in both resource management and decision-making amidst uncertainty. Observations include balances, daily stock amounts, closing prices, and various technical indicators for each of the 30 stocks in the trading pool. Agents use their balances to trade these stocks. We define a cost function based on trading volume and stock price, imposing a fee on each transaction. This requires the agent to strategically optimize its transactions, considering both stock price fluctuations and associated fees. For more details on each environment and experimental setting, see Appendix E.Table 1 presents our results for all four tasks. Each algorithm is trained for 100k steps and evaluated for a maximum of 1000 steps with 5 seeds across 10 episodes. The data quality levels are denoted as follows: for locomotion tasks, medium (m), medium-expert (m-e), and expert (e); for the financial task, medium (M), and high (H). Complete experimental results are provided in Appendix H.ROIDICE exhibits superior ROI performance compared to other algorithms across most tasks, particularly excelling in the Hopper environment where there is significant potential for ROI improvement. OptiDICE also demonstrates high ROI in environments other than Hopper due to the high ROI of the offline datasets themselves for Walker2D and Halfcheetah (see Figure 2 for dataset ROIs). However, compared to OptiDICE, ROIDICE shows a greater return with lower accumulated cost in a significant number of tasks (see Appendix H).The constrained offline RL algorithms are evaluated under two cost thresholds, specifically using the 80th and 50th percentiles of the accumulated costs from the offline dataset to set the cost conditions for COptiDICE (discounted with γ = 0.99) and CDT (undiscounted). This constraint setting ensures that the constrained offline RL algorithms make in-distribution decisions regarding cost constraints.In some domains, particularly where costs can be lowered with a slight loss in return, COptiDICE demonstrates a higher ROI compared to OptiDICE, but without fine-grained tuning of cost constraints, it does not reach the performance of ROIDICE. Conversely, CDT tends to overfit under the current experiment settings when the dataset lacks diversity. In most cases, CDT either fails to optimize a policy (resulting in negative ROI), or optimizes a policy with a very high return but also a very high accumulated cost (resulting in a small ROI).Figure 2 illustrates how ROIDICE effectively learns from diverse behaviors in the dataset. We compare the ROI of ROIDICE with that of Behavior Cloning (BC) that clones behaviors only from high ROI trajectories. We compute the ROI of trajectories in the dataset and select the top 20%, 50%, and 80% ROI trajectories to train BCn% agents. We can observe that in a significant number of experiment settings, ROIDICE outperforms BCn% agents. This demonstrates ROIDICE's capability to stitch together various segments from different trajectories to optimize a policy that behaves more efficiently than any other trajectory in the dataset.Compared to locomotion tasks, the financial task adds another challenge of high stochasticity. Despite this, ROIDICE demonstrates strong performance on the financial task with both medium and highquality datasets, as shown in Table 1 . OptiDICE, although achieving high returns, shows relatively low ROI performance due to high transaction fees (cost). The high ROI observed from CDT is actually coincidental, as CDT fails to meet the given cost constraint due to overfitting; its return maximization policy accidentally results in an efficient policy with high ROI.Qualitative Behavior Examples Figure 3 illustrates the different behaviors between ROIDICE and OptiDICE in the Hopper task at the same time steps, selected to clearly represent these differences.Recall that in the locomotion task, the cost corresponds to the amount of torque applied, i.e., the energy consumed forcing it to move forward. OptiDICE focuses solely on maximizing return, resulting in the agent making large, far-jumping movements, even applying torque to the joint after the jump, which is inefficient for obtaining additional rewards. In contrast, ROIDICE finds the optimal behavior to increase return while reducing energy consumption. Consequently, the agent attempts to hop far but avoids inefficient actions that decrease ROI. 
conclusion: 
 In this paper, we propose a novel policy optimization framework designed to maximize the Return on Investment (ROI) of the policy, the ratio between the return and the accumulated cost. Maximizing ROI is particularly relevant in practical scenarios: the fuel efficiency (km/L) of autonomous vehicles, managing the trade-off between decision quality and planning time in meta-controllers for real-time decision-making, or balancing generation quality and inference time in large language models (LLMs). The ROI maximization is approached through linear-fractional programming that incorporates the stationary distribution of the policy. We apply the Charnes-Cooper transformation to convert the problem into equivalent linear programming, ROI-LP. Motivated by the DICE RL frameworks, we extend ROI-LP to develop our offline policy optimization algorithm, ROIDICE, by incorporating convex regularization designed to properly regularize the distribution shift. We show that ROIDICE yields a policy with better efficiency than policies from the existing RL-based optimization methods by considering the trade-off between return and accumulated cost. 
limitations: 
 While our work is innovative in optimizing the ROI of a policy, it functions within the constraints of an offline setting where direct interaction with the environment is not feasible. Consequently, the ultimate performance of the policy optimized with ROIDICE is contingent upon the quality of the provided dataset. Since ROIDICE takes into account both returns and accumulated costs, it is sensitive to the design of the reward and cost functions. For instance, the agent may behave similarly to one trained by an unconstrained RL when the cost function approximates a constant. Conversely, the agent may exhibit behaviors aimed at minimizing costs excessively, akin to laziness, when the cost function varies significantly. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper does not directly engage with societal elements or have immediate applications that could be identified as having positive or negative societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Section 5.2, Appendix E, Appendix F Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this section, we introduce how DICE RL algorithms are derived from the regularized policy optimization framework. We derive DICE RL algorithms for both unconstrained and constrained scenario: OptiDICE [12] and COptiDICE [13] .We show how OptiDICE is derived from the regularized policy optimization framework in [12] . The convex optimization of the framework is given as, where convex function f (x) is a convex function generally used in f -divergence: f (x) = 1 2 (x -1) 2 of χ 2 -divergence or f (x) = x log x -x + 1 of KL divergence.EQUATIONwhere ν is the Lagrangian multiplier for the Bellman flow constraint. where e ν (s, a) = r(s, a) + γ s ′ T (s ′ |s, a)ν(s ′ ) -ν(s).OptiDICE [12] simplifies the max-min problem by switching the order of optimization from max w≥0 min ν L(w, ν) to min ν max w≥0 L(w, ν). The optimal solution is not affected due to the strong duality of the problem. The switch enables the closed-form solution of stationary distribution ratio w of the inner maximization problem that satisfies ∂L(w, ν)/∂w(s, a) = 0 ∀s, a.EQUATIONThe closed-form solution w * ν (s, a) can be plugged in L(w, ν) to derive ν loss of OptiDICE given as,EQUATIONAfter minimization on ν is complete, optimal ν * is plugged in w * ν (s, a) to obtain the optimal stationary distribution d * (s, a). Policy π that generates d * (s, a) is trained by using policy extraction techniques from [12] . In this paper, we adopt weighted behavior cloning method, whose loss function is given as,EQUATIONWe show how COptiDICE, the extension of OptiDICE to offline constrained RL, is derived. Constrained RL assumes an additional cost function c(s, a) and limits the discounted sum of cost d(s, a)c(s, a) ≤ C threshold . The constraint is added to the regularized policy optimization framework in [13] . After the similar derivations from OptiDICE, the closed-form solution of stationary distribution ratio w and loss functions for ν are given as,EQUATION)EQUATIONwhere e ν,λ (s, a) := r(s, a) -λc(s, a) + s ′ T (s ′ |s, a)ν(s ′ ) -ν(s). Lagrangian multiplier λ determines how much reward function r(s, a) is penalized by cost function c(s, a) and its loss function is given as,EQUATION)where λ is updated to satisfy the constraint by increasing when discounted sum of cost s,a d(s, a)c(s, a) is larger than the cost upper bound C threshold .We follow similar approaches from Appendix A to derive ROIDICE from the Regularized ROI maximization framework. The main goal of the framework is to obtain the optimal stationary distribution d * (s, a) = d ′ * (s, a)/t * that maximizes ROI of the policy from the optimal solutions of the convex optimization optimization given as, We follow the approaches from OptiDICE [12] and switch the order of the optimization from max w≥0,t≥0 min ν,µ L(ν, µ, w, t) to max t≥0 min ν,µ max w≥0 L(ν, µ, w, t) to obtain the closed-form solution of w. Optimal w that satisfies ∂L(ν,µ,w,t) ∂w(s,a) = 0 in terms of ν, µ and t is given as,max d ′ ≥0,t≥0 s,a d ′ (s, a)r(s, a) -α s,a d D (s, a)f d ′ (s, a) d D (s, a) , t s.t. (B * d ′ )(s) = t(1 -γ)p 0 (s) + γ(T * d ′ )(s) ∀sEQUATIONwhere (f ′ ) -1 (y, t) is an inverse function of ∂ ∂x f (x, t) with respect to x. Loss functions of ν, µ and t are obtained by substituting w * ν,µ,t (s, a) into L(ν, µ, w, t): EQUATIONWe evaluate ROIDICE on random MDP environments and compare with unconstrained and constrained RL algorithms, OptiDICE and COptiDICE. We generate tabular environments with 1000 different seeds. Each environments consists of 50 states and 4 actions. Their transition probability is randomly generated following Dirichlet distribution. The sparse reward is given when an agent reaches on the goal state, and the cost function follows Beta(0.2, 0.2) distribution. We add 1 to all states in order to satisfying the domain condition of linear-fractional programming. r(s, a) = After the MDP wih a cost function is generated, we obtain behavior policy π D (a|s) that achieves 90% of the optimal return to gather fixed datasets with {10, 20, 50, 100, 1000, 2000} trajectories. The dataset is used to estimate the empirical transition probability T (s ′ |s, a) with the given samples.With the estimated transition probability T (s ′ |s, a) and behavior policy π D (a|s), we apply the estimated MDP to a tabular version of ROIDICE. For the tabular version of ROIDICE, we apply the primal problem of the Regularized ROI maximization framework to a convex optimization solver software. For the convex regularization f (x, t), we use f 1 (x, t) = 1 2 (x -t) 2 and α is set to be the inverse of the number of trajectories. We investigate how varying α, which controls the strength of the regularization, affects the performance of ROIDICE by sweeping its values. Figure 5 shows the ROI performance for α ranging from 1 × 10 -6 to 1.0 using expert quality dataset. Recall that the Hopper task datasets generally contain trajectories with low ROI, while most trajectories of Walker2d and Halfcheetah achieve relatively high ROI (see Table 2 ). In the Hopper environment, ROI performance decreases as α increases, indicating an opportunity to increase ROI when the model is less regularized, as the behavior policy collecting the offline dataset mostly does not account for the cost. In the Halfcheetah environment, ROIDICE achieves a high ROI when following the offline dataset due to the dataset's properties, but shows largely varying performance with small α due to its lack of coverage outside the dataset. In the Walker2D environment, ROIDICE shows robustness across various levels of α.We assess ROIDICE across four environments: three locomotion tasks (Hopper, Halfcheetah, Walker2D) and one financial trading task (FinRL). We utilize environment configurations and datasets provided by [6] (CC BY 4.0) and [18] (CC BY 4.0), with modifications to reward and cost functions. Additionally, we incorporate an absorbing state and an absorbing action to precisely calculate the sum of discounted rewards and costs, including an additional dimension to indicate whether the current state and action are absorbed.Locomotion Task In the Hopper environment, the reward is multifaceted, consisting of two primary components: the forward reward and the control cost. The forward reward incentivizes the agent to make progress towards its goal, while the control cost penalizes actions that are overly aggressive or energetically inefficient. Halfcheetah and Walker2D environments introduce an additional aspect to the reward: a healthy reward. This rewards the agent for maintaining stability. The control cost serves as a mechanism to penalize excessive actions, reflecting the idea that in real-world scenarios, significant force often comes with considerable energy expenditure. This cost can be seen as proportional to the energy consumption in physical robots, motivating agents to find smoother and more energy-efficient strategies for accomplishing their tasks. We utilize the forward reward as a reward and the control cost as a cost, normalizing the cost for stable learning. The reward and cost functions are as follows:r(s, a) = f orward_reward_weight * x t -x t+1 dt c(s, a) = w m c |A| • ∥a∥ 2 2 + b m cwhere x t represents the x-coordinate before the action a t , and x t+1 represents the x-coordinate after the action. The parameter f orward_reward_weight corresponds to the forward reward and is specific to each environment. The two hyperparameters, w c > 0 and b c > 0, are weights that control the strength of the cost, ensuring compliance with the linear-fractional program condition. Additionally, |A| denotes the size of the action space dimension.Financial Task FinRL offers a stock trading simulation environment where agents trade stocks to earn compensation, with 30 tradable stocks available (N = 30). The reward is calculated by summing the current balance with the product of the number of shares owned and the closing price of each share. Additionally, a cost function considers trading volume and stock price, imposing a fee on each transaction. The reward and cost functions are as follows:r(s, a, s ′ ) = w f r • (v(s ′ ) -v(s)) v(s) = s[0] + s[1 : 31] * s[31 : 61] c i (s, a) = b f c + w f c s[i + 1] • min(a i , s[i + N + 1] if a i < 0 s[i + 1] • min( s[0] s[i+1] , a i ) * pct if a i > 0where pct = 0.001 denotes a per-share percentage, which is the most commonly used transaction cost rate for each trade, i = 0, 1, . . . , N represents the index of tradable stocks, and s We implement ROIDICE algorithm based on [13] (CC BY 4.0) for the tabular experiment, and [20] (MIT License) for the continuous domain experiment. Our code is available at: https://github. com/ku-dmlab/ROIDICE. The implementation and configuration of OptiDICE and COptiDICE are adopted from the framework presented in [13] , while CDT is utilized within the OSRL framework as described in [14] .The task and dataset name is succinctly stated: Hopper (H), Walker2D (W), Halfcheetah (HC), and Finance (F); for locomotion tasks, medium (m), medium-expert (m-e), and expert (e); for the financial task, medium (M), and high (H). We set the hyperparameter α for OptiDICE and COptiDICE, which controls the strength of the f-divergence regularizer. OptiDICE and COptiDICE have similar regularizer strengths, so we set the same α for both of them. For Hopper, we set α = 0.01, for Walker2D (m) α = 0.1, for Walker2D (m-e and e) α = 1.0, and for Halfcheetah (respectively, m, m-e, and e) we set α = 0.01, α = 0.1, and α = 1.0; for financial task we set α = 1.0. For hyperparameter settings and model parameter configuration of ROIDICE, see Table 3 Table 3 : Hyperparameters and configuration of ROIDICE. TaskWe average the scores and obtain ±2× standard error using 5 seeds across 10 episodes. The task names are concisely abbreviated as follows: Hopper (H), Walker2D (W), HalfCheetah (HC), and Finance (F). Figure 6 illustrates the learning curves of ROIDICE and other offline unconstrained and constrained RL algorithms. In the Hopper environment, the performance of ROIDICE is significantly better. OptiDICE attains excessive returns but relatively lower ROI than ROIDICE due to its disregard for costs in the environment (see Table 5 ). ROIDICE achieves greater returns than OptiDICE when dealing with datasets that contain a mix of expert-level and lower-level data. Our analysis shows that ROIDICE exhibits qualitative behavior by considering both return and accumulated cost. Offline constrained RL algorithms, COptiDICE and CDT, show high ROI with datasets that include high ROI trajectories. In the HalfCheetah and Finance environments with high-quality datasets, CDT performs well but not as well as ROIDICE (see Table 6 ). Justification: Section 1Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.38th Conference on Neural Information Processing Systems (NeurIPS 2024).f (x, t) should be convex in x given t > 0 and convex in t given x > 0. 


Decision: The paper presents a constrained offline policy optimization algorithm that maximizes Return On Investment (ROI) and formulates it as a linear programming problem. The experiments on various offline RL tasks show that it achieves a better trade-off between return and accumulated cost. 

The paper initially received borderline reviews. While the reviewers considered the work novel (Z94y, T1wG, Yyxa) and the proposed framework well-formulated (8zrU, T1wG, Yyxa), they also raised several concerns on the paper: 1) missing computational complexity analysis (Z94y) and theoretical analysis on sample complexity (8zrU) or cost guarantee (Yyxa); 2) lack of clarity on several technical issues, including coping with online setting (Z94y) and large state space (8zrU, T1wG), reward/cost scaling and advantage of the method (Yyxa) ; 3) missing important baselines and experiments (T1wG, Yyxa); 4) limited application domains (Yyxa); and 5) insufficient analysis on the results (T1wG, Yyxa). 

The authors provided a thorough response, including additional results on computational complexity and experimental comparisons, detailed analysis of the theoretical properties and results, and clarifications on the technical issues and application domains. During the rebuttal discussion, most reviewers considered their main concerns well addressed and reached a consensus with positive ratings. After considering the paper, reviews, and rebuttal discussion, the AC does not find sufficient cause to overturn the reviewers' consensus and therefore recommends it for acceptance. The author should take into account the rebuttal and the reviewers' feedback, especially regarding the additional results and analysis, when preparing the final version.

Reviews: review: paper_summary: paper_summary: The paper addresses the problem of maximizing the Return on Investment (ROI) in the context of offline reinforcement learning (RL). The method introduced is ROIDICE, which stands for Return on Investment Decision-making in the Offline Setting. ROIDICE is an offline policy optimization algorithm designed to optimize the ROI of a policy using a fixed dataset of pre-collected experiences.
The method involves formulating the ROI maximization problem as a linear-fractional programming problem, which is then converted into an equivalent linear programming problem using the Charnes-Cooper transformation. This transformation allows the problem to be solved using standard linear programming techniques.
To address the distribution shift inherent in offline learning, ROIDICE incorporates a convex regularization that measures the amount of distribution shift while maintaining the convexity of the problem. This regularization is designed to penalize the trained policy's deviation from the behavior policy used to collect the offline dataset.
The experiments conducted in the paper demonstrate that ROIDICE outperforms other offline RL algorithms, including those that focus solely on maximizing the return of the policy. ROIDICE achieves a superior trade-off between return and accumulated cost, resulting in a more efficient policy.
strength_weakness: strength_weakness: ROIDICE may require significant computational resources, especially when dealing with large-scale datasets and complex environments. Could you provide details on the computational resources and time consumption for different methods?
If the data distribution in the offline dataset significantly differs from that encountered in the online environment, the performance of ROIDICE may be affected. How can this issue be addressed?
questions: questions: See Weakness
limitations: limitations: Not applicable
paper_summary: paper_summary: This paper provides a fraction linear programming framework for solving offline RL problems for return on investment (ROI). The fraction linear programming can be trasformed to a linear programming and a convex regularizer is used to control the distribution mismatch. The authors provide adequate experimental results to demonstrate the advantages of the algorithm.
strength_weakness: strength_weakness: No sample complexity guarantee.  See questions for details.
questions: questions: 1. Can the authors provide theoretical analysis for the sample complexity like the paper  ``Ozdaglar et al Revisitting the linear programming framework for offline reinforcement learning'' which also uses LP framework to solve offline RL?
2. How can we implement this LP using function approximation when the state-action space is large?
3. Missing references: There are papers using LP frameworks to solve offline RL with sample complexity guarantee, such as the two paper mentioned before.
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces a novel approach for solving constrained offline reinforcement learning problems. The authors apply the Charnes-Cooper transformation to convert the linear-fractional programming into an equivalent linear programming problem, and draw inspiration from the DICE framework to maximize ROI under offline setting. Experimental results across finite and continuous domains demonstrate that the proposed ROIDICE achieves a better trade-off between return and accumulated cost.
strength_weakness: strength_weakness: * Important baselines are missing: apart from DICE-based methods, comparisons should include other offline constrained RL methods.
* How is the proposed method implemented in continuous domains? The description lacks clarity; for instance, providing pseudocode would be helpful.
* In the FinRL task, it's intriguing why the unconstrained OptiDICE method performed well while other constrained baselines showed poor performance.
questions: questions: - It is suggested to add the discussion of related works and compare them in experiments, such as [1,2].
- Can you explain why using constraint methods in the FinRL environment actually yields poor results?
- Can you provide experimental results on widely recognized safe RL environments such as SafetyGymnasium[3,4] and BulletSafetyGym[5]?
  [1] Xu H, Zhan X, Zhu X. Constraints penalized q-learning for safe offline reinforcement learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(8): 8753-8760.
  [2] Guan J, Chen G, Ji J, et al. VOCE: Variational optimization with conservative estimation for offline safe reinforcement learning[J]. Advances in Neural Information Processing Systems, 2024, 36.
  [3] Ray A, Achiam J, Amodei D. Benchmarking safe exploration in deep reinforcement learning[J]. arXiv preprint arXiv:1910.01708, 2019, 7(1): 2.
  [4] Ji J, Zhou J, Zhang B, et al. Omnisafe: An infrastructure for accelerating safe reinforcement learning research[J]. arXiv preprint arXiv:2305.09304, 2023.
  [5] Gronauer S. Bullet-safety-gym: A framework for constrained reinforcement learning[J]. 2022.
limitations: limitations: As mentioned in the paper, the ultimate performance of ROIDICE is contingent upon the dataset and exhibits sensitivity, potentially resulting in behaviors such as minimizing costs excessively.
paper_summary: paper_summary: The article proposes a novel offline algorithm for learning policies optimising Return on Investment (ROI): the ratio between the (discounted) cumulated rewards obtained by a policy, and the (discounted) cumulated costs of the actions taken. The paper focuses specifically on offline RL, and in particular builds on formalising the RL problem in a Linear Programming form. 
The proposed approach builds on the recent literature, particularly DICE-RL, but derives a new regularisation term for ROI optimisation. 

The experimental evaluation demonstrates that the proposed approach successfully learns policies with high ROI on various offline RL datasets.
strength_weakness: strength_weakness: - Although optimising ROI could indeed provide a better framework for cost/returns tradeoffs, the article does not provide a substantive argument of why this would be the case. This somewhat reduces the expected impact and relevance of the contribution. 
- Similarly, the experimental evaluation only compares the proposed approach to RL trained to optimise the return solely, with no constraint for cost. In this setting, it seems self-evident that the proposed method would achieve better ROI and lower costs than a policy trained ignoring cost altogether. It is fairly common in RL to approach this tradeoff by integrating the action costs as negative rewards, and it would have been fairer to compare with such an approach as an additional baseline.
- One reason for the broad use of ROI in business, is that the quantities of "return" and "investment" are usually unambiguously in the same quantity making the ratio calculation natural (although different costs can affect the ratio). This is less straightforward in many RL problems where the return and possible costs may be in very different dimensions and distributions. It is unclear from the paper how stable are the policies optimised depending on reward/cost scaling. 
- There is an issue with a number of references in the bibliography that appear to be missing some fields (eg, refs 6,7,9,10,11)Could you provide a better rationale on why ROI would be a good objective for RL in general, beyond its use in the business domain? 
- The difference in return between standard RL and the ROI version is quite large in the experiments. How significant is the decrease in policy performance compared to cost-free policies in practice for 
- Does the proposed approach provide some guarantees on cost, compared to, eg, CoptiDICE?
questions: questions: Could you provide a better rationale on why ROI would be a good objective for RL in general, beyond its use in the business domain? 
- The difference in return between standard RL and the ROI version is quite large in the experiments. How significant is the decrease in policy performance compared to cost-free policies in practice for 
- Does the proposed approach provide some guarantees on cost, compared to, eg, CoptiDICE?
limitations: limitations: - No significant ethical aspects to this work.

Comments: title: Good rebuttal
Good rebuttal that addressed most of my concerns
Rebuttal discussion
Rebuttal discussion
Rebuttal discussion
Rebuttal discussion
Dear AC and PC,
comment: I thank the authors for the response. I will increase the score to 6.  But I still think for offline RL papers, the sample complexity and function approximation setting to deal with large state action space are important.
Dear Reviewer Yyxa,

We're pleased to know that your concerns have been resolved, and we truly appreciate your positive feedback!

However, it seems the system is still displaying the initial review score. Could you kindly verify if your updated score has been correctly applied?

Thank you so much.
Thank you for the detailed and thoughtful response to my comments. After reading the response (and other reviewers' comments), I would like to raise my rating to 6.
I thank the authors for providing detailed responses to my comments. Reading the rebuttal materials solved most of my concerns. I also read the comments from other reviewers and still rate this work positively.

Good luck to the authors for the final decision of this work.

Best wishes,
Dear Reviewer Yyxa,

Could you please take a look at the rebuttal and share your thoughts on the authors' reply? The author has provided a detailed response to the initial comments - does it address your major concerns? 

Thank you very much for your time and effort!

Best,

Your AC
Dear Reviewer T1wG,

Could you please take a look at the rebuttal and share your thoughts on the authors' reply? The author has provided additional comparisons to baselines, explanations, and more experimental results. Does the rebuttal address your initial concerns? 

Thank you very much for your time and effort!

Best,

Your AC
Dear Reviewer 8zrU,

Could you please take a look at the rebuttal and share your thoughts on the authors' reply? The author has provided additional theoretical analysis and clarification in response to Q1. Does the rebuttal sway your opinions?

Thank you very much for your time and effort!

Best,

Your AC
Dear Reviewer Z94y,

Could you please take a look at the rebuttal and share your thoughts on the authors' reply? The author provided additional results on the computational cost and discussion on the online setting. Do they address your concerns? 

Thank you very much for your time and effort!

Best, 

Your AC
As we have now passed the midpoint of the author-reviewer discussion period, we would like to confirm whether our responses have effectively addressed the reviewers' concerns.

Could you kindly encourage the reviewers to begin their discussions? We would greatly appreciate your support in facilitating this process.