Title: ROIDICE: Offline Return on Investment Maximization for Efficient Decision Making
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning

Convex Optimization
Abstract: In this paper, we propose a novel policy optimization framework that maximizes Return on Investment (ROI) of a policy using a fixed dataset within a Markov Decision Process (MDP) equipped with a cost function. ROI, defined as the ratio between the return and the accumulated cost of a policy, serves as a measure of efficiency of the policy. Despite the importance of maximizing ROI in various applications, it remains a challenging problem due to its nature as a ratio of two long-term values: return and accumulated cost. To address this, we formulate the ROI maximizing reinforcement learning problem as a linear fractional programming. We then incorporate the stationary distribution correction (DICE) framework to develop a practical offline ROI maximization algorithm.
Our proposed algorithm, ROIDICE, yields an efficient policy that offers a superior trade-off between return and accumulated cost compared to policies trained using existing frameworks.

ROIDICE: Offline Return on Investment Maximization for Efficient Decision Making

Abstract

In this paper, we propose a novel policy optimization framework that maximizes Return on Investment (ROI) of a policy using a fixed dataset within a Markov Decision Process (MDP) equipped with a cost function. ROI, defined as the ratio between the return and the accumulated cost of a policy, serves as a measure of the efficiency of the policy. Despite the importance of maximizing ROI in various applications, it remains a challenging problem due to its nature as a ratio of two long-term values: return and accumulated cost. To address this, we formulate the ROI maximizing reinforcement learning problem as linear fractional programming. We then incorporate the stationary distribution correction (DICE) framework to develop a practical offline ROI maximization algorithm. Our proposed algorithm, ROIDICE, yields an efficient policy that offers a superior trade-off between return and accumulated cost compared to policies trained using existing frameworks.

1 Introduction

In economics, Return on Investment (ROI) is a financial metric used to evaluate the profitability of an investment relative to its cost. The concept of ROI originates from the work of [5] and is widely regarded as a valuable metric by the majority of marketing managers [4]. ROI is calculated by dividing the profit generated from the investment by the cost of the investment, and it is a key indicator for evaluating the efficiency and effectiveness of various economic decisions and strategies [17]. Maximizing ROI is crucial as a decision with a high ROI yields a higher return at a relatively lower cost.

While ROI is a compelling metric for optimization, ROI maximization for decision making has not been actively explored within the field of machine learning. Previous studies on ROI maximization [3] have focused on multi-armed bandit scenarios, where reward and cost are immediately accessible following a decision. When it comes to sequential decision-making setups like the Markov Decision Process (MDP) that additionally considers a cost function, optimizing a sequence of decisions or a policy to maximize the ratio between return and accumulated cost becomes far more challenging and remains mostly unexplored. This optimization contains two major difficulties: the fractional relationship between the return and accumulated cost of a policy, and the fact that both are longterm quantities collected after a sequence of decisions or actions of the agent. The issues make ROI maximization via existing policy optimization techniques not straightforward, as it includes computing the gradient of the ratio between the two expected values.

For effective optimization of ROI, we refer to the dual formulation of V-LP [16], which represents policy optimization in reinforcement learning (RL) as linear programming in terms of the stationary distribution. The stationary distribution of a policy is the discounted sum of probabilities of the

state-action pairs visited by the agent following the policy. Leveraging the property of stationary distribution, which allows for easy estimation of both return and accumulated cost, we derive ROI-LP, linear programming problem for ROI maximization. In this paper, we focus specifically on the offline setting and aim to derive an offline ROI maximization algorithm. The offline setting, which involves optimizing a policy using a fixed dataset of pre-collected experiences without further environmental interactions (e.g., offline RL), has been actively studied recently due to its compelling use cases [2, 7, 9, 10, 11, 20]. Building on the derived ROI-LP, we incorporate a convex regularization to address the distribution shift inherent in offline learning. This leads to our proposed algorithm, ROIDICE, which achieves offline ROI maximization via stationary distribution correction estimation.

Through a series of diverse experiments, we compare ROIDICE to algorithms from other offline policy optimization frameworks, including offline RL and offline constrained RL. We demonstrate that maximizing the ROI of a policy leads to a different, more efficient behavior compared to frameworks that focus solely on maximizing policy return. In summary, our contributions are threefold:

Formulation of ROI maximization framework in an MDP with a cost function.

Derivation of ROIDICE, an offline ROI maximization algorithm that optimizes the ROI of a policy using a fixed dataset of pre-collected experiences.

Comparison with existing offline policy optimization frameworks, showing that ROIDICE optimizes ROI and improves policy efficiency across various domains.

2 Backgrounds

Markov Decision Process with a cost function We consider an infinite-horizon discounted Markov Decision Process (MDP) with a cost function, represented as a tuple M = ⟨ S, A, T, r, p 0 , γ ⟩ . Here, S is the set of states, A is the set of actions, T ( s ′ | s, a ) denotes the transition probability from stateaction pair ( s, a ) to the next state s ′ , and r ( s, a ) is the reward function for state-action pair ( s, a ) . p 0 ( s ) is the initial state distribution, and γ is the discount factor. Additionally, we introduce a cost function c ( s, a ) > 0 for all state-action pairs ( s, a ) .

An agent begins at an initial state following p 0 ( s ) and repeats the process of collecting reward r ( s, a ) and incurring cost c ( s, a ) by taking action a in state s and transitioning to the next state s ′ . The policy π ( a | s ) represents the distribution of actions to be taken by the agent in state s . The performance of the policy π can be evaluated with the expected return R π = E π [ ∑ ∞ t =0 γ t r ( s t , a t )] and the expected accumulated cost C π = E π [ ∑ ∞ t =0 γ t c ( s t , a t )] , which are the expected cumulative sums of discounted rewards and costs over rollouts of the policy π .

The main goal of reinforcement learning (RL) is to optimize policy π to maximize expected return R π collected by the agent. Constrained RL optimizes the policy to maximize its expected return R π while constraining its expected accumulated cost C π to be lower than a given threshold C threshold .

Stationary Distribution and Linear Programming (LP) formulation of RL The stationary distribution of policy d π represents the discounted sum of probabilities of the state-action pair ( s, a ) visited by an agent following policy π , i.e., d π ( s, a ) := (1 -γ ) ∑ ∞ t =0 γ t Pr ( s t = s, a t = a ) . Using the stationary distribution, R π and C π can be expressed as linear combinations of the stationary distribution and the reward and the cost respectively: R π = 1 1 -γ ∑ s,a d π ( s, a ) r ( s, a ) and C π = 1 1 -γ ∑ s,a d π ( s, a ) c ( s, a ) . This characteristic allows the RL problem to be formulated as a linear programming (LP), known as the dual of V-LP in [16].

Constraint (2), known as Bellman flow constraint, is a requirement that any stationary distribution must satisfy. The condition can be interpreted as ensuring that the probability of state-action pairs leaving state s matches the total probability of state-action pairs entering s , plus the probability of state s initiating a trajectory. Once the optimal stationary distribution d ∗ is obtained, optimal policy π ∗ can be extracted from it by π ∗ ( a | s ) = d ∗ ( s, a ) / ∑ a ′ d ∗ ( s, a ′ ) ∀ s, a .

Offline RL and DICE-RL Framework In this paper, we assume an offline policy optimization setting where interaction between the agent and environment is not allowed. Instead, the agent is provided with a fixed dataset of experiences D = { ( s i , a i , r i , c i , s ′ i ) } N i =1 , consisting of N transition samples, to optimize its policy. A unique challenge in offline RL is balancing between policy optimization and distribution shift, as it is not possible to gather samples from the optimized policy.

In [12], regularized policy optimization is proposed to penalize return maximization (1) with f -divergence between stationary distributions of the trained policy d and the behavior policy d D , defined as D f ( d || d D ) := ∑ s,a d D ( s, a ) f ( d ( s, a ) /d D ( s, a )) . The convexity of f leads to a convex optimization problem that balances return maximization and distribution shift:

where α is a hyper-parameter that controls the trade-off. In the constrained setting studied in [13], a cost constraint ∑ s,a d ( s, a ) c ( s, a ) ≤ C threshold is added to the problem.

Previous studies have leveraged the characteristics of the above convex optimization problem to develop practical offline RL algorithms. They follow a similar approach by applying a Lagrangian multiplier ν ( s ) to the Bellman flow constraint (2), reformulating the Lagrangian dual of the problem into the main loss function of the algorithms. We refer to this family of algorithms as the DICE-RL framework. Detailed explanations of the loss functions and the full derivations of related DICE-RL algorithms (OptiDICE [12] and COptiDICE [13]) are provided in Appendix A.

Linear-fractional Programming Linear-fractional programming is defined as maximizing a ratio of two affine functions over a polyhedron. Its standard form is given by:

The domain of f is restricted to ensure the denominator remains positive, i.e., { x | d T x + β > 0 } , as a zero denominator would result in an infeasible solution.

Assuming the region satisfying the constraints is non-empty and bounded, any linear-fractional programming can be transformed into equivalent linear programming using the Charnes-Cooper transformation [1]. The main idea of the transformation is to replace variable x with two new variables t ≥ 0 and y , such that t ( d T x + β ) = 1 and y = t x . This change of variables eliminates the denominator of f ( x ) , resulting in the following linear programming problem:

3 ROI Maximization in Linear Programming form (ROI-LP)

In this section, we start with a return on investment (ROI) maximization problem by substituting the objective of the dual of V-LP from return R π to ROI. This results in a linear-fractional programming problem, as the objective is the ratio of two linear functions representing return and accumulated cost. We then apply the Charnes-Cooper transformation to derive an equivalent linear programming (LP), ROI-LP.

ROI Maximization Problem We begin by defining the ROI of policy π in a Markov Decision Process (MDP). The ROI of policy π is the ratio between the return R π and its accumulated cost R π , which can be expressed in terms of its stationary distribution d π : ROI ( π ) = R π C π = ∑ s,a d π ( s,a ) r ( s,a ) ∑ s,a d π ( s,a ) c ( s,a ) . Substituting the objective of the dual of V-LP (1) with ROI, we get our ROI maximization problem, which adheres to the standard form of linear-fractional programming in (3).

ROI-LP We apply the Charnes-Cooper transformation to reformulate the ROI maximization problem (4) into an equivalent linear programming problem, ROI-LP. This involves introducing a non-negative variable t ≥ 0 and replacing d ( s, a ) with d ′ ( s, a ) and t , which are defined as:

and thus d ( s, a ) = d ′ ( s, a ) /t . Substituting d , ROI-LP, the linear programming formulations for maximizing ROI, is given by:

where the equality constraint (6) is newly introduced to ensure that (5) is satisfied. Once the optimization on ROI-LP is complete, optimal stationary distribution d ∗ ( s, a ) is obtained by dividing d ′∗ ( s, a ) with t ∗ .

Remark. When c ( s, a ) is a fixed constant C for all states and actions, the optimal solution of ROI-LP becomes equivalent to that of the dual of V-LP. In this case, the optimal t becomes 1 C as per (5), since any stationary distribution satisfies ∑ s,a d ( s, a ) = 1 . Consequently, the optimal d ′∗ becomes d ∗ from the dual of V-LP scaled by t . This special case illustrates that ROI maximization simplifies return maximization when the accumulated cost is constant, regardless of the policy.

4 Offline ROI maximization

In this section, we derive an offline ROI maximization algorithm that extends ROI-LP to also include the trade-off for distribution shifts. We formulate a convex optimization problem for offline ROI maximization and use it to derive ROIDICE, the first offline algorithm to optimize policy efficiency in terms of ROI given a fixed dataset.

4.1 Regularized ROI Maximization Framework

To formulate an offline ROI maximization problem, we add a convex regularization to ROI-LP representing a distribution shift. The approach is similar to the DICE-RL framework [13], which incorporates an f -divergence between the stationary distribution of trained policy d and behavior policy d D into the dual of V-LP. Our regularized ROI maximization framework is given by:

where the design of f ( x, t ) will be provided later in this subsection. We denote D f,t ( d ′ , d D ) = ∑ s,a d D ( s, a ) f ( d ′ ( s,a ) d D ( s,a ) , t ) .

The primary difference between our proposed problem and DICE-RL framework lies in the convex regularization. The f -divergence D f ( d || d D ) from DICE-RL framework is not directly applicable to d ′ ( s, a ) , as d ′ ( s, a ) is not a valid probability distribution; it is derived by scaling the stationary distribution d ( s, a ) by t . However, naively incorporating D f ( d ′ t || d D ) results in the loss of the convexity, as f ( x t ) of f -divergence is generally not convex with respect to t . To this end, we design a new convex regularization that measures the amount of distribution shift while not breaking the convexity of the problem. We begin by outlining three conditions that the regularization should satisfy.

1. f ( x, t ) should be convex in x given t > 0 and convex in t given x > 0 .

f ( x, t ) should be zero when x = t .

f ( x, t ) should be well-defined on x > 0 and t > 0 .

The first condition is that the problem must maintain convexity with respect to both x and t . The second condition is that it must effectively regularize the distribution shift. Specifically, D f,t ( d ′ , d D ) should be minimized to zero when the stationary distribution d ( s, a ) = d ′ ( s, a ) /t matches d D ( s, a ) . The last condition is to avoid infeasible optimization scenarios, e.g., negative values inside a log function. We introduce two convex functions as examples.

From f ( x ) = 1 2 ( x -1) 2 of χ 2 -divergence, we can obtain the first function f 1 ( x, t ) by shifting it by t -1 . We introduce a softened variant of f 1 , denoted as f 2 , where a scaled version of the function f ( x ) = x log x -x +1 (arising from the KL divergence) is seamlessly combined with the left segment of f 1 . Although f 2 ( x, t ) is not entirely convex with respect to t , it serves as a valid regularization term, as f 2 ( x, t ) = 0 holds exclusively when x = t .

Remark. A unique characteristic of the proposed regularization D f,t ( d ′ , d D ) is that its magnitude is affected not only by the degree of distribution shift but also by the trained policy's accumulated cost. We demonstrate this by comparing the magnitude of D f,t ( d ′ , d D ) and f -divergence D f ( d ′ t ∥ d D ) that corresponds to typical regularizer of DICE-RL framework. In the case of χ 2 -divergence, D f,t ( d ′ , d D ) = t 2 D f ( d ′ t || d D ) as D f ( d ′ t || d D ) = ∑ s,a d D ( s, a ) 1 2 t 2 ( d ′ ( s,a ) d D ( s,a ) -t ) 2 . Since t represents the inverse of the expected accumulated cost (5), the strength of the regularization with D f,t ( d ′ , d D ) becomes stronger than the actual f -divergence when the expected accumulated cost of the trained policy is less than 1.

4.2 ROIDICE

We are now prepared to derive ROIDICE, an offline algorithm for ROI maximization based on the Regularized ROI maximization framework. To enforce the constraints (7) and (8), we introduce Lagrangian multiplier ν ∈ R | S | and µ ∈ R :

where w ( s, a ) = d ′ ( s, a ) /d D ( s, a ) . We follow a similar derivation process as outlined by [12] and obtain a closed-form solution of w ∗ ν,µ,t ( s, a ) :

where e ν ( s, a ) = r ( s, a ) + γ ∑ s ′ T ( s ′ | s, a ) ν ( s ′ ) -ν ( s ) and ( f ′ ) -1 ( y, t ) is the inverse function of ∂ ∂x f ( x, t ) with respect to x . Loss functions of ν, µ and t are obtained by substituting w ∗ ν,µ,t ( s, a ) into L ( ν, µ, w, t ) :

After the minimization on ν, µ and t is complete, we apply policy extraction techniques from [12] to obtain the policy that generates optimal stationary distribution d ∗ ( s, a ) = 1 t w ∗ ν,µ,t ( s, a ) d D ( s, a ) . In this paper, we adopt weighted behavior cloning method, whose loss function is given as,

The more detailed derivation and algorithm of ROIDICE can be found in Appendix B.

Offline RL

ROIDICE

Behavior

10

20

50

100

1000

2000

Number of Trajectories

1.5

2.0

2.5

3.0

3.5

ROI

10

20

50

100

1000

2000

Number of Trajectories

4.0

4.5

5.0

5.5

RETURN

10

20

50

100

1000

2000

Number of Trajectories

1.5

2.0

2.5

3.0

COST

(a) ROIDICE vs. offline RL

0.7

0.9

1.0

1.1

1.3

cost multiplier

2.0

2.5

3.0

3.5

ROI

True MDP

0.7

0.9

1.0

1.1

1.3

cost multiplier

3

4

5

ROI

MLE MDP

Offline RL (N=100)

ROIDICE (N=50)

ROIDICE (N=100)

ROIDICE (N=2000)

(b) ROIDICE vs. offline constrained RL.

5 Experiments

In this section, we demonstrate that the offline policy obtained from ROIDICE achieves a superior trade-off between return and accumulated cost compared to other RL approaches such as offline RL and constrained offline RL, in both finite and continuous domains.

5.1 Random Finite MDP Experiments

We test ROIDICE in a Random MDP setting similar to [12]. A tabular MDP with | S | = 50 and | A | = 4 is randomly generated and a behavior policy that achieves 90 % of the optimal return is used to gather fixed datasets with { 10 , 20 , 50 , 100 , 1000 , 2000 } trajectories. We estimate the empirical transition probability ̂ T ( s ′ | s, a ) with the given samples and use it to run a tabular version of ROIDICE. Details of the experiments are provided in the Appendix C. We compare ROIDICE with DICE-RL on other offline policy optimization schemes, OptiDICE [12] for offline RL and COptiDICE [13] for offline constrained RL.

vs. Offline RL In Figure 1a, we compare the performance of ROIDICE with OptiDICE. While OptiDICE achieves the highest return, it incurs a higher accumulated cost than ROIDICE. This is expected, as RL does not account for the accumulated cost it incurs. In contrast, ROIDICE optimizes the policy's ROI, resulting in a highly efficient policy that delivers a substantial return at a relatively lower accumulated cost.

vs. Offline Constrained RL In Figure 1b, we compare the performance of ROIDICE with COptiDICE. Offline constrained RL aims to maximize return while keeping the accumulated cost below a specified threshold C π ≤ C threshold. As demonstrated in Appendix A, COptiDICE optimizes its policy using a penalized reward function r ( s, a ) -λ c c ( s, a ) , where λ c is the Lagrange multiplier for the cost constraint. To ensure satisfaction of the cost constraint, λ c is updated to increase when the constraint is violated. To set the cost thresholds for our experiment, we obtain the accumulated cost of the ROIDICE policy on the Maximum Likelihood Estimate (MLE) MDP and multiply it with [0 . 7 , 0 . 9 , 1 . 0 , 1 . 1 , 1 . 3] . The left side of the figure shows the true ROI performance of the agents, while the right side shows the ROI performance on the MLE MDP, representing the performance agents expect. The cost thresholds from the MLE MDP are based on the offline assumption which excludes the interaction with the true MDP.

With a sufficient number of trajectories in the offline dataset, COptiDICE with a cost multiplier of 1.0, which corresponds to the same cost budget as the optimized ROIDICE policy, achieves the best ROI among the tested multipliers. Both increasing or decreasing the cost threshold result in reduced

policy efficiency, as constrained RL agents typically use the entire available budget to maximize return. When the number of trajectories is insufficient, ROIDICE is rarely outrun by COptiDICE as the large discrepancy between True MDP and MLE MDP causes ROIDICE to be optimized on the erroneous MLE MDPs.

While it is theoretically possible to identify a policy with optimal ROI using a constrained RL algorithm, doing so requires multiple runs with different cost thresholds to determine the best one. This process can become computationally intractable and less accurate as the complexity of the problem increases. In contrast, a single run of ROIDICE is sufficient to find a policy with high ROI without the need to explicitly search for the right cost threshold.

5.2 ROI Maximization in Continuous Domains

We evaluate our algorithm's performance across diverse domains using datasets from D4RL [6] (CC BY 4.0) and NeoRL [18] (CC BY 4.0). From the D4RL benchmark, we select three locomotion tasks (Hopper, Halfcheetah, Walker2D), and from NeoRL, we include a stock trading task (FinRL). To assess ROIDICE's capability to maximize Return on Investment (ROI), we conduct comparative analyses with various unconstrained or constrained offline RL algorithms. We utilize offline datasets from D4RL and NeoRL, which are collected using data collection policies that do not consider cost. Similar to the random MDP experiments, we include OptiDICE and COptiDICE for comparison. Additionally, we incorporate the Constrained Decision Transformer (CDT, [15]) to represent a recent approach that leverages the power of generative models on RL.

5.2.1 Environment and Offline Dataset

In the three locomotion environments, observations include the positional values and velocities of various body parts, with actions corresponding to the torques exerted between links. The common reward function used in recent benchmarks consists of three components: incentive for forward running speed, incentive for maintaining a healthy state, and penalty for applying torques on joints. In this experiment, we use the control penalty as the cost function, aiming to train the agent to strive for efficient energy consumption from an ROI perspective. We used only forward running speed as our reward function.

FinRL, the time-driven stock trading simulator, enables us to replicate live stock markets using real market data. This allows us to assess our algorithm's proficiency in both resource management and decision-making amidst uncertainty. Observations include balances, daily stock amounts, closing

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

4

6

8

10

MEDIUM

Hopper

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

0

5

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

0

5

Halfcheetah

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

5

10

MEDIUM_EXPERT

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

0

5

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

0

5

10

0.0

0.2

0.4

0.6

0.8

1.0

Env step

×10 5

5

10

EXPERT

0.0

0.2

0.4

0.6

0.8

1.0

Env step

×10 5

0

5

0.0

0.2

0.4

0.6

0.8

1.0

Env step

×10 5

0

5

10

ROIDICE

BC20%

BC50%

BC80%

BC100%

prices, and various technical indicators for each of the 30 stocks in the trading pool. Agents use their balances to trade these stocks. We define a cost function based on trading volume and stock price, imposing a fee on each transaction. This requires the agent to strategically optimize its transactions, considering both stock price fluctuations and associated fees. For more details on each environment and experimental setting, see Appendix E.

5.2.2 Results

Table 1 presents our results for all four tasks. Each algorithm is trained for 100k steps and evaluated for a maximum of 1000 steps with 5 seeds across 10 episodes. The data quality levels are denoted as follows: for locomotion tasks, medium (m), medium-expert (m-e), and expert (e); for the financial task, medium (M), and high (H). Complete experimental results are provided in Appendix H.

ROIDICE exhibits superior ROI performance compared to other algorithms across most tasks, particularly excelling in the Hopper environment where there is significant potential for ROI improvement. OptiDICE also demonstrates high ROI in environments other than Hopper due to the high ROI of the offline datasets themselves for Walker2D and Halfcheetah (see Figure 2 for dataset ROIs). However, compared to OptiDICE, ROIDICE shows a greater return with lower accumulated cost in a significant number of tasks (see Appendix H).

The constrained offline RL algorithms are evaluated under two cost thresholds, specifically using the 80 th and 50 th percentiles of the accumulated costs from the offline dataset to set the cost conditions for COptiDICE (discounted with γ = 0 . 99 ) and CDT (undiscounted). This constraint setting ensures that the constrained offline RL algorithms make in-distribution decisions regarding cost constraints. In some domains, particularly where costs can be lowered with a slight loss in return, COptiDICE demonstrates a higher ROI compared to OptiDICE, but without fine-grained tuning of cost constraints, it does not reach the performance of ROIDICE. Conversely, CDT tends to overfit under the current experiment settings when the dataset lacks diversity. In most cases, CDT either fails to optimize a policy (resulting in negative ROI), or optimizes a policy with a very high return but also a very high accumulated cost (resulting in a small ROI).

Figure 2 illustrates how ROIDICE effectively learns from diverse behaviors in the dataset. We compare the ROI of ROIDICE with that of Behavior Cloning (BC) that clones behaviors only from high ROI trajectories. We compute the ROI of trajectories in the dataset and select the top 20%, 50%, and 80% ROI trajectories to train BC n % agents. We can observe that in a significant number of experiment settings, ROIDICE outperforms BC n % agents. This demonstrates ROIDICE's capability

OptiDICE

ROI:4.962Discountedreturn:149.829

Accumulatedcost:30.247

ROIDICE

ROI:8.145 Discountedreturn:147.821 Accumulated cost:18.148

to stitch together various segments from different trajectories to optimize a policy that behaves more efficiently than any other trajectory in the dataset.

Compared to locomotion tasks, the financial task adds another challenge of high stochasticity. Despite this, ROIDICE demonstrates strong performance on the financial task with both medium and highquality datasets, as shown in Table 1. OptiDICE, although achieving high returns, shows relatively low ROI performance due to high transaction fees (cost). The high ROI observed from CDT is actually coincidental, as CDT fails to meet the given cost constraint due to overfitting; its return maximization policy accidentally results in an efficient policy with high ROI.

Qualitative Behavior Examples Figure 3 illustrates the different behaviors between ROIDICE and OptiDICE in the Hopper task at the same time steps, selected to clearly represent these differences. Recall that in the locomotion task, the cost corresponds to the amount of torque applied, i.e., the energy consumed forcing it to move forward. OptiDICE focuses solely on maximizing return, resulting in the agent making large, far-jumping movements, even applying torque to the joint after the jump, which is inefficient for obtaining additional rewards. In contrast, ROIDICE finds the optimal behavior to increase return while reducing energy consumption. Consequently, the agent attempts to hop far but avoids inefficient actions that decrease ROI.

6 Conclusion

In this paper, we propose a novel policy optimization framework designed to maximize the Return on Investment (ROI) of the policy, the ratio between the return and the accumulated cost. Maximizing ROI is particularly relevant in practical scenarios: the fuel efficiency (km/L) of autonomous vehicles, managing the trade-off between decision quality and planning time in meta-controllers for real-time decision-making, or balancing generation quality and inference time in large language models (LLMs). The ROI maximization is approached through linear-fractional programming that incorporates the stationary distribution of the policy. We apply the Charnes-Cooper transformation to convert the problem into equivalent linear programming, ROI-LP. Motivated by the DICE RL frameworks, we extend ROI-LP to develop our offline policy optimization algorithm, ROIDICE, by incorporating convex regularization designed to properly regularize the distribution shift. We show that ROIDICE yields a policy with better efficiency than policies from the existing RL-based optimization methods by considering the trade-off between return and accumulated cost.

7 Limitations

While our work is innovative in optimizing the ROI of a policy, it functions within the constraints of an offline setting where direct interaction with the environment is not feasible. Consequently, the ultimate performance of the policy optimized with ROIDICE is contingent upon the quality of the provided dataset. Since ROIDICE takes into account both returns and accumulated costs, it is sensitive to the design of the reward and cost functions. For instance, the agent may behave similarly

to one trained by an unconstrained RL when the cost function approximates a constant. Conversely, the agent may exhibit behaviors aimed at minimizing costs excessively, akin to laziness, when the cost function varies significantly.

References

Abraham Charnes and William W Cooper. Programming with linear fractional functionals. Naval Research logistics quarterly , 9(3-4):181-186, 1962.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

Yuan Deng, Negin Golrezaei, Patrick Jaillet, Jason Cheuk Nam Liang, and Vahab Mirrokni. Multi-channel autobidding with budget and roi constraints. In International Conference on Machine Learning , pages 7617-7644. PMLR, 2023.

Paul W Farris. Marketing Metrics: The Definitive Guide to Measuring Marketing Performance . Pearson Education, Inc, 2010.

Dale L Flesher and Gary John Previts. Donaldson brown (1885-1965): the power of an individual and his ideas over time. Accounting Historians Journal , 40(1):79-101, 2013.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02328 , 2023.

Jiayi Guan, Guang Chen, Jiaming Ji, Long Yang, Zhijun Li, et al. Voce: Variational optimization with conservative estimation for offline safe reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems , 34:12731286, 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:11791191, 2020.

Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning , pages 6120-6130. PMLR, 2021.

Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations , 2022.

Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303 , 2023.

Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning , pages 21611-21630. PMLR, 2023.

Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality. arXiv preprint arXiv:2001.01866 , 2020.

Patricia Pulliam Phillips. Return on investment (ROI) basics . Association for Talent Development, 2023.

Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offline reinforcement learning. Advances in Neural Information Processing Systems , 35:24753-24765, 2022.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7(1):2, 2019.

Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations , 2023.

Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8753-8760, 2022.

A Full derivation of DICE RL algorithms

In this section, we introduce how DICE RL algorithms are derived from the regularized policy optimization framework. We derive DICE RL algorithms for both unconstrained and constrained scenario: OptiDICE [12] and COptiDICE [13].

A.1 Derivation of OptiDICE

We show how OptiDICE is derived from the regularized policy optimization framework in [12]. The convex optimization of the framework is given as,

where convex function f ( x ) is a convex function generally used in f -divergence: f ( x ) = 1 2 ( x -1) 2 of χ 2 -divergence or f ( x ) = x log x -x +1 of KL divergence.

Assuming d ( s, a ) = w ( s, a ) d D ( s, a ) is the stationary distribution of the dataset d D ( s, a ) weighted by w , the Lagrangian dual of the convex optimization problem is given as,

where ν is the Lagrangian multiplier for the Bellman flow constraint.

We use ∑ s ν ( s ) ∑ ¯ s, ¯ a T ( s | ¯ s, ¯ a ) w (¯ s, ¯ a ) d D (¯ s, ¯ a ) = ∑ s,a w ( s, a ) d D ( s, a ) ∑ s ′ T ( s ′ | s, a ) ν ( s ′ ) to switch the order of summation and transform the Lagrangian.

OptiDICE [12] simplifies the max-min problem by switching the order of optimization from max w ≥ 0 min ν L ( w,ν ) to min ν max w ≥ 0 L ( w,ν ) . The optimal solution is not affected due to the strong duality of the problem. The switch enables the closed-form solution of stationary distribution ratio w of the inner maximization problem that satisfies ∂L ( w,ν ) /∂w ( s, a ) = 0 ∀ s, a .

The closed-form solution w ∗ ν ( s, a ) can be plugged in L ( w,ν ) to derive ν loss of OptiDICE given as,

After minimization on ν is complete, optimal ν ∗ is plugged in w ∗ ν ( s, a ) to obtain the optimal stationary distribution d ∗ ( s, a ) . Policy π that generates d ∗ ( s, a ) is trained by using policy extraction techniques from [12]. In this paper, we adopt weighted behavior cloning method, whose loss function is given as,

A.2 Derivation of COptiDICE

We show how COptiDICE, the extension of OptiDICE to offline constrained RL, is derived. Constrained RL assumes an additional cost function c ( s, a ) and limits the discounted sum of cost

∑ d ( s, a ) c ( s, a ) ≤ C threshold. The constraint is added to the regularized policy optimization framework in [13].

Lagrangian multiplier λ is applied to the cost constraint and the Lagrangian of the constrained problem is given as,

After the similar derivations from OptiDICE, the closed-form solution of stationary distribution ratio w and loss functions for ν are given as,

where e ν,λ ( s, a ) := r ( s, a ) -λc ( s, a ) + ∑ s ′ T ( s ′ | s, a ) ν ( s ′ ) -ν ( s ) . Lagrangian multiplier λ determines how much reward function r ( s, a ) is penalized by cost function c ( s, a ) and its loss function is given as,

where λ is updated to satisfy the constraint by increasing when discounted sum of cost ∑ s,a d ( s, a ) c ( s, a ) is larger than the cost upper bound C threshold .

B Derivation of ROIDICE

We follow similar approaches from Appendix A to derive ROIDICE from the Regularized ROI maximization framework. The main goal of the framework is to obtain the optimal stationary distribution d ∗ ( s, a ) = d ′∗ ( s, a ) /t ∗ that maximizes ROI of the policy from the optimal solutions of the convex optimization optimization given as,

We derive the Lagrangian of the convex problem by introducing Lagrangian multipliers ν and µ for the equality constraints. We change the optimization variable d ′ ( s, a ) to w ( s, a ) = d ′ ( s, a ) /d D ( s, a ) , but the ratio is not a valid stationary distribution ratio in contrast to the previous DICE RL algorithms.

We use ∑ s ν ( s ) ( γ ( T ∗ d ′ )( s ) -( B ∗ d ′ )( s )) = ∑ s,a d ′ ( s, a ) ( γ ∑ s ′ T ( s ′ | s, a ) ν ( s ′ ) -ν ( s )) to obtain an another expression of L ( ν, µ, w, t ) .

We follow the approaches from OptiDICE [12] and switch the order of the optimization from max w ≥ 0 ,t ≥ 0 min ν,µ L ( ν, µ, w, t ) to max t ≥ 0 min ν,µ max w ≥ 0 L ( ν, µ, w, t ) to obtain the closed-form solution of w . Optimal w that satisfies ∂L ( ν,µ,w,t ) ∂w ( s,a ) = 0 in terms of ν, µ and t is given as,

where ( f ′ ) -1 ( y, t ) is an inverse function of ∂ ∂x f ( x, t ) with respect to x . Loss functions of ν, µ and t are obtained by substituting w ∗ ν,µ,t ( s, a ) into L ( ν, µ, w, t ) :

Algorithm 1 ROIDICE

Input : The offline dataset D , the initial state offline dataset p 0 , parameterized Lagrangian multipliers ν ϕ , µ, t , and policy π θ .

Output : Optimal policy π ∗ θ .

Initialize all parameters.

while convergence do

Offline RL (N=100)

ROIDICE (N=50)

ROIDICE (N=100)

ROIDICE (N=2000)

0.7

0.9

1.0

1.1

1.3

Cost Multiplier

2.0

2.5

3.0

3.5

ROI

0.7

0.9

1.0

1.1

1.3

Cost Multiplier

3.5

4.0

4.5

5.0

5.5

RETURN

0.7

0.9

1.0

1.1

1.3

Cost Multiplier

1.5

2.0

2.5

3.0

COST

(a) True MDP

0.7

0.9

1.0

1.1

1.3

Cost Multiplier

3

4

5

ROI

0.7

0.9

1.0

1.1

1.3

Cost Multiplier

3

4

5

6

RETURN

0.7

0.9

1.0

1.1

1.3

Cost Multiplier

1.0

1.5

2.0

2.5

COST

(b) MLE MDP

C Details of Random MDP experiment

We evaluate ROIDICE on random MDP environments and compare with unconstrained and constrained RL algorithms, OptiDICE and COptiDICE. We generate tabular environments with 1000 different seeds. Each environments consists of 50 states and 4 actions. Their transition probability is randomly generated following Dirichlet distribution. The sparse reward is given when an agent reaches on the goal state, and the cost function follows Beta (0 . 2 , 0 . 2) distribution. We add 1 to all states in order to satisfying the domain condition of linear-fractional programming.

where the discount factor γ = 0 . 95 and β ∼ Beta (0 . 2 , 0 . 2) .

After the MDP wih a cost function is generated, we obtain behavior policy π D ( a | s ) that achieves 90 % of the optimal return to gather fixed datasets with { 10 , 20 , 50 , 100 , 1000 , 2000 } trajectories. The dataset is used to estimate the empirical transition probability ̂ T ( s ′ | s, a ) with the given samples.

With the estimated transition probability ̂ T ( s ′ | s, a ) and behavior policy π D ( a | s ) , we apply the estimated MDP to a tabular version of ROIDICE. For the tabular version of ROIDICE, we apply the primal problem of the Regularized ROI maximization framework to a convex optimization solver software. For the convex regularization f ( x, t ) , we use f 1 ( x, t ) = 1 2 ( x -t ) 2 and α is set to be the inverse of the number of trajectories.

5.0

7.5

Hopper

1e-06 1e-05 0.0001 0.001

0.01

0.1

1.0

0

2

4

6

1e-06 1e-05 0.0001 0.001

0.01

0.1

1.0

0

1

2

5

10

Halfcheetah

1e-06 1e-05 0.0001 0.001

0.01

0.1

1.0

0

2

BC50%

BC100%

D Impact of Various α

We investigate how varying α , which controls the strength of the regularization, affects the performance of ROIDICE by sweeping its values. Figure 5 shows the ROI performance for α ranging from 1 × 10 -6 to 1 . 0 using expert quality dataset. Recall that the Hopper task datasets generally contain trajectories with low ROI, while most trajectories of Walker2d and Halfcheetah achieve relatively high ROI (see Table 2). In the Hopper environment, ROI performance decreases as α increases, indicating an opportunity to increase ROI when the model is less regularized, as the behavior policy collecting the offline dataset mostly does not account for the cost. In the Halfcheetah environment, ROIDICE achieves a high ROI when following the offline dataset due to the dataset's properties, but shows largely varying performance with small α due to its lack of coverage outside the dataset. In the Walker2D environment, ROIDICE shows robustness across various levels of α .

E Data and Environment Details

We assess ROIDICE across four environments: three locomotion tasks (Hopper, Halfcheetah, Walker2D) and one financial trading task (FinRL). We utilize environment configurations and datasets provided by [6] (CC BY 4.0) and [18] (CC BY 4.0), with modifications to reward and cost functions. Additionally, we incorporate an absorbing state and an absorbing action to precisely calculate the sum of discounted rewards and costs, including an additional dimension to indicate whether the current state and action are absorbed.

Locomotion Task In the Hopper environment, the reward is multifaceted, consisting of two primary components: the forward reward and the control cost. The forward reward incentivizes the agent to make progress towards its goal, while the control cost penalizes actions that are overly aggressive or energetically inefficient. Halfcheetah and Walker2D environments introduce an additional aspect to the reward: a healthy reward. This rewards the agent for maintaining stability. The control cost serves as a mechanism to penalize excessive actions, reflecting the idea that in real-world scenarios, significant force often comes with considerable energy expenditure. This cost can be seen as proportional to the energy consumption in physical robots, motivating agents to find smoother and more energy-efficient strategies for accomplishing their tasks. We utilize the forward reward as a reward and the control cost as a cost, normalizing the cost for stable learning. The reward and cost functions are as follows:

where x t represents the x-coordinate before the action a t , and x t +1 represents the x-coordinate after the action. The parameter forward _ reward _ weight corresponds to the forward reward and is specific to each environment. The two hyperparameters, w c > 0 and b c > 0 , are weights that control the strength of the cost, ensuring compliance with the linear-fractional program condition. Additionally, | A | denotes the size of the action space dimension.

Financial Task FinRL offers a stock trading simulation environment where agents trade stocks to earn compensation, with 30 tradable stocks available ( N = 30 ). The reward is calculated by

summing the current balance with the product of the number of shares owned and the closing price of each share. Additionally, a cost function considers trading volume and stock price, imposing a fee on each transaction. The reward and cost functions are as follows:

where pct = 0 . 001 denotes a per-share percentage, which is the most commonly used transaction cost rate for each trade, i = 0 , 1 , . . . , N represents the index of tradable stocks, and s [ j ] , j ∈ { 0 , . . . , 181 } , represents j -th dimension of state s . Each state dimension contains different information: s [0] represents the balance, s [1 : 31] denotes today's closing value for each of the 30 stocks, and s [31 : 61] indicates today's trading volume for each of the 30 stocks.

F Implementation Details

We implement ROIDICE algorithm based on [13] (CC BY 4.0) for the tabular experiment, and [20] (MIT License) for the continuous domain experiment. Our code is available at: https://github. com/ku-dmlab/ROIDICE . The implementation and configuration of OptiDICE and COptiDICE are adopted from the framework presented in [13], while CDT is utilized within the OSRL framework as described in [14].

The task and dataset name is succinctly stated: Hopper (H), Walker2D (W), Halfcheetah (HC), and Finance (F); for locomotion tasks, medium (m), medium-expert (m-e), and expert (e); for the financial task, medium (M), and high (H). We set the hyperparameter α for OptiDICE and COptiDICE, which controls the strength of the f-divergence regularizer. OptiDICE and COptiDICE have similar regularizer strengths, so we set the same α for both of them. For Hopper, we set α = 0 . 01 , for Walker2D (m) α = 0 . 1 , for Walker2D (m-e and e) α = 1 . 0 , and for Halfcheetah (respectively, m, m-e, and e) we set α = 0 . 01 , α = 0 . 1 , and α = 1 . 0 ; for financial task we set α = 1 . 0 . For hyperparameter settings and model parameter configuration of ROIDICE, see Table 3

G Experiments Compute Resources

ROIDICE was conducted on a single CPU (AMD Ryzen Threadripper PRO 5995WX) with 256GB of RAM and a single GPU (NVIDIA RTX 4090). Each training session took about 10 to 20 minutes, utilizing approximately 20% of the RAM and from 40% to 60% of the GPU memory.

H Experiments Results

We average the scores and obtain ± 2 × standard error using 5 seeds across 10 episodes. The task names are concisely abbreviated as follows: Hopper (H), Walker2D (W), HalfCheetah (HC), and Finance (F). Figure 6 illustrates the learning curves of ROIDICE and other offline unconstrained and constrained RL algorithms. In the Hopper environment, the performance of ROIDICE is significantly better. OptiDICE attains excessive returns but relatively lower ROI than ROIDICE due to its disregard for costs in the environment (see Table 5). ROIDICE achieves greater returns than OptiDICE when dealing with datasets that contain a mix of expert-level and lower-level data. Our analysis shows that ROIDICE exhibits qualitative behavior by considering both return and accumulated cost. Offline constrained RL algorithms, COptiDICE and CDT, show high ROI with datasets that include high ROI trajectories. In the HalfCheetah and Finance environments with high-quality datasets, CDT performs well but not as well as ROIDICE (see Table 6).

EXPERT

cdt (50%)

cdt (20%)

coptidice (50%)

coptidice (20%)

optidice

roidice

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

2

4

6

8

10

MEDIUM

Hopper

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

4

2

0

2

4

6

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

0

2

4

6

8

10

Halfcheetah

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

2

4

6

8

10

MEDIUM-EXPERT

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

4

2

0

2

4

6

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

0.0

2.5

5.0

7.5

10.0

0.0

0.2

0.4

0.6

0.8

1.0

Env step

×10 5

0.0

2.5

5.0

7.5

10.0

0.0

0.2

0.4

0.6

0.8

1.0

Env step

×10 5

4

2

0

2

4

6

0.0

0.2

0.4

0.6

0.8

1.0

Env step

×10 5

0.0

2.5

5.0

7.5

10.0

12.5

(a) ROI of locomotion tasks

0.0

0.2

0.4

0.6

0.8

1.0

Env step

×10 5

0

20

40

60

Finance Medium

0.0

0.2

0.4

0.6

0.8

1.0

Env step

×10 5

0

20

40

60

Finance High

(b) ROI of financial tasks

I Comparison of Offline Constrained RL Algorithms

Our experiment in continuous domains from Section 5.2 includes non-DICE offline constrained algorithm, CDT. We additionally provide experiments on other offline constrained RL frameworks, VOCE [8] and CPQ [21]. Table 7 presents the ROI performance of these constrained offline RL algorithms in the Hopper environment [6]. These algorithms use two cost budgets, corresponding to the 50th and 80th percentiles of the accumulated costs from the offline dataset.

Over various experiments, constrained offline RL algorithms commonly have shown low ROIs. We conjecture that these results are mainly due to the overestimation of costs, making constrained offline RL algorithms to be overly conservative on choosing costly state-actions. On the other hand, ROIDICE seems to be less affected by such overestimation, as overestimated costs are partially mitigated with overestimated rewards when estimating ROIs.

J Experiment Results in Safety RL Environments

We evaluate our approach on OpenAI SafetyGym [19] tasks including CarGoal and PointPush. Table 8 provides results averaged over 5 seeds across 10 episodes. We observed that ROIDICE outperforms COptiDICE in terms of ROI. We use the rewards and costs from the environment, adding a constant value of ϵ = 0 . 1 to each cost to maintain our assumption that c ( s, a ) > 0 ∀ s, a . The offline dataset is collected by PPO Lagrangian. ROIDICE uses α = 0 . 001 for CarGoal and α = 0 . 01 for PointPush. COptiDICE uses α = 0 . 01 for CarGoal and α = 1 . 0 for PointPush.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Section 1

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 4.2, Appendix C, Appendix F, Appendix E

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Appendix C, Appendix F, Appendix E

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 5, Appendix C, Appendix F, Appendix E, Appendix H

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Section 5, Appendix H

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix G

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code fo Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper does not directly engage with societal elements or have immediate applications that could be identified as having positive or negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Section 5.2, Appendix E, Appendix F

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article proposes a novel offline algorithm for learning policies optimising Return on Investment (ROI): the ratio between the (discounted) cumulated rewards obtained by a policy, and the (discounted) cumulated costs of the actions taken. The paper focuses specifically on offline RL, and in particular builds on formalising the RL problem in a Linear Programming form. \nThe proposed approach builds on the recent literature, particularly DICE-RL, but derives a new regularisation term for ROI optimisation. \n\nThe experimental evaluation demonstrates that the proposed approach successfully learns policies with high ROI on various offline RL datasets.\n\nSTRENGTHS:\n- The idea of optimising ROI is, to my knowledge, novel, and interesting as it proposes a new way to balance the trade-off between action costs and policy rewards, which is relevant in a wide range of applications of RL. \n- Although the RL literature has proposed ways to handle this trade-offs since its early days (for example, by modelling costs as negative rewards), there is an argument that an ROI objective could have some advantages. \n- The derivation of the algorithm is well described in the article and is a substantive contribution with respect to the state-of-the-art\n\nWEAKNESSES:\n- Although optimising ROI could indeed provide a better framework for cost/returns tradeoffs, the article does not provide a substantive argument of why this would be the case. This somewhat reduces the expected impact and relevance of the contribution. \n- Similarly, the experimental evaluation only compares the proposed approach to RL trained to optimise the return solely, with no constraint for cost. In this setting, it seems self-evident that the proposed method would achieve better ROI and lower costs than a policy trained ignoring cost altogether. It is fairly common in RL to approach this tradeoff by integrating the action costs as negative rewards, and it would have been fairer to compare with such an approach as an additional baseline.\n- One reason for the broad use of ROI in business, is that the quantities of \"return\" and \"investment\" are usually unambiguously in the same quantity making the ratio calculation natural (although different costs can affect the ratio). This is less straightforward in many RL problems where the return and possible costs may be in very different dimensions and distributions. It is unclear from the paper how stable are the policies optimised depending on reward/cost scaling. \n- There is an issue with a number of references in the bibliography that appear to be missing some fields (eg, refs 6,7,9,10,11)Could you provide a better rationale on why ROI would be a good objective for RL in general, beyond its use in the business domain? \n- The difference in return between standard RL and the ROI version is quite large in the experiments. How significant is the decrease in policy performance compared to cost-free policies in practice for \n- Does the proposed approach provide some guarantees on cost, compared to, eg, CoptiDICE?\n\nQUESTIONS:\nCould you provide a better rationale on why ROI would be a good objective for RL in general, beyond its use in the business domain? \n- The difference in return between standard RL and the ROI version is quite large in the experiments. How significant is the decrease in policy performance compared to cost-free policies in practice for \n- Does the proposed approach provide some guarantees on cost, compared to, eg, CoptiDICE?\n\nLIMITATIONS:\n- No significant ethical aspects to this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach for solving constrained offline reinforcement learning problems. The authors apply the Charnes-Cooper transformation to convert the linear-fractional programming into an equivalent linear programming problem, and draw inspiration from the DICE framework to maximize ROI under offline setting. Experimental results across finite and continuous domains demonstrate that the proposed ROIDICE achieves a better trade-off between return and accumulated cost.\n\nSTRENGTHS:\n* The paper's approach is mathematically sound, and the motivation is well elucidated.\n* The experimental results conducted in various environments demonstrate its superiority over the baseline.\n\nWEAKNESSES:\n* Important baselines are missing: apart from DICE-based methods, comparisons should include other offline constrained RL methods.\n* How is the proposed method implemented in continuous domains? The description lacks clarity; for instance, providing pseudocode would be helpful.\n* In the FinRL task, it's intriguing why the unconstrained OptiDICE method performed well while other constrained baselines showed poor performance.\n\nQUESTIONS:\n- It is suggested to add the discussion of related works and compare them in experiments, such as [1,2].\n- Can you explain why using constraint methods in the FinRL environment actually yields poor results?\n- Can you provide experimental results on widely recognized safe RL environments such as SafetyGymnasium[3,4] and BulletSafetyGym[5]?\n  [1] Xu H, Zhan X, Zhu X. Constraints penalized q-learning for safe offline reinforcement learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(8): 8753-8760.\n  [2] Guan J, Chen G, Ji J, et al. VOCE: Variational optimization with conservative estimation for offline safe reinforcement learning[J]. Advances in Neural Information Processing Systems, 2024, 36.\n  [3] Ray A, Achiam J, Amodei D. Benchmarking safe exploration in deep reinforcement learning[J]. arXiv preprint arXiv:1910.01708, 2019, 7(1): 2.\n  [4] Ji J, Zhou J, Zhang B, et al. Omnisafe: An infrastructure for accelerating safe reinforcement learning research[J]. arXiv preprint arXiv:2305.09304, 2023.\n  [5] Gronauer S. Bullet-safety-gym: A framework for constrained reinforcement learning[J]. 2022.\n\nLIMITATIONS:\nAs mentioned in the paper, the ultimate performance of ROIDICE is contingent upon the dataset and exhibits sensitivity, potentially resulting in behaviors such as minimizing costs excessively.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper provides a fraction linear programming framework for solving offline RL problems for return on investment (ROI). The fraction linear programming can be trasformed to a linear programming and a convex regularizer is used to control the distribution mismatch. The authors provide adequate experimental results to demonstrate the advantages of the algorithm.

STRENGTHS:
The framework is clean which uses linear programming to model the offline ROI maximization problems and the authors provide sufficient experimental results with good performance.

WEAKNESSES:
No sample complexity guarantee.  See questions for details.

QUESTIONS:
1. Can the authors provide theoretical analysis for the sample complexity like the paper  ``Ozdaglar et al Revisitting the linear programming framework for offline reinforcement learning'' which also uses LP framework to solve offline RL?
2. How can we implement this LP using function approximation when the state-action space is large?
3. Missing references: There are papers using LP frameworks to solve offline RL with sample complexity guarantee, such as the two paper mentioned before.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of maximizing the Return on Investment (ROI) in the context of offline reinforcement learning (RL). The method introduced is ROIDICE, which stands for Return on Investment Decision-making in the Offline Setting. ROIDICE is an offline policy optimization algorithm designed to optimize the ROI of a policy using a fixed dataset of pre-collected experiences.\nThe method involves formulating the ROI maximization problem as a linear-fractional programming problem, which is then converted into an equivalent linear programming problem using the Charnes-Cooper transformation. This transformation allows the problem to be solved using standard linear programming techniques.\nTo address the distribution shift inherent in offline learning, ROIDICE incorporates a convex regularization that measures the amount of distribution shift while maintaining the convexity of the problem. This regularization is designed to penalize the trained policy's deviation from the behavior policy used to collect the offline dataset.\nThe experiments conducted in the paper demonstrate that ROIDICE outperforms other offline RL algorithms, including those that focus solely on maximizing the return of the policy. ROIDICE achieves a superior trade-off between return and accumulated cost, resulting in a more efficient policy.\n\nSTRENGTHS:\n1. The paper introduces a novel policy optimization framework that maximizes the Return on Investment (ROI) of a policy. This is a significant departure from traditional approaches that focus solely on maximizing return, without considering the cost associated with the actions.\n2. The proposed method operates within an offline setting, where direct interaction with the environment is not possible. This is particularly useful in scenarios where online interaction is costly or risky.\n3. The authors derive an offline algorithm, ROIDICE, which optimizes the ROI of a policy using a fixed dataset of pre-collected experiences. \n4. The paper demonstrates that ROIDICE yields policies with better efficiency than policies from existing RL-based optimization methods. This is achieved by considering the trade-off between return and accumulated cost. The authors conduct extensive experiments across various domains, including locomotion tasks and a financial stock trading task. T\n\nWEAKNESSES:\nROIDICE may require significant computational resources, especially when dealing with large-scale datasets and complex environments. Could you provide details on the computational resources and time consumption for different methods?\nIf the data distribution in the offline dataset significantly differs from that encountered in the online environment, the performance of ROIDICE may be affected. How can this issue be addressed?\n\nQUESTIONS:\nSee Weakness\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}