ROIDICE: Offline Return on Investment Maximization for Efficient Decision Making
Authors: Woosung Kim
Hayeong Lee
Jongmin Lee
Byung-Jun Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Convex Optimization
Primary area: reinforcement_learning
TL;DR: Policy optimization that maximizes Return on Investment (ROI) of a policy

Abstract: In this paper, we propose a novel policy optimization framework that maximizes Return on Investment (ROI) of a policy using a fixed dataset within a Markov Decision Process (MDP) equipped with a cost function. ROI, defined as the ratio between the return and the accumulated cost of a policy, serves as a measure of efficiency of the policy. Despite the importance of maximizing ROI in various applications, it remains a challenging problem due to its nature as a ratio of two long-term values: return and accumulated cost. To address this, we formulate the ROI maximizing reinforcement learning problem as a linear fractional programming. We then incorporate the stationary distribution correction (DICE) framework to develop a practical offline ROI maximization algorithm.
Our proposed algorithm, ROIDICE, yields an efficient policy that offers a superior trade-off between return and accumulated cost compared to policies trained using existing frameworks.

ROIDICE: Offline Return on Investment Maximization for Efficient Decision Making
Woosung Kim
Hayeong Lee
Jongmin Lee2t
Byung-Jun Lee
Korea University UC Berkeley
Gauss Labs Inc.
{usk208, wwwehnye lee, (al.guuuueeeeeorea.aaeeae kr Satgminn.ee@berkeee edu
byungjunlee)
'Equal contribution !Corresponding authors
Abstract
In this paper, propose novel policy optimization framework that maximizes Return on Investment (ROI) of a policy using a fixed dataset within a Markov Decision Process (MDP) equipped with a cost function. ROI, defined as the ratio between the return and the accumulated cost of a policy, serves as a measure of the efficiency of the policy. Despite the importance of maximizing ROI in various applications, it remains a challenging problem due to its nature as a ratio of two long-term values: return and accumulated cost. To address this, we formulate the ROI maximizing reinforcement learning problem as linear fractional programming. We then incorporate the stationary distribution correction (DICE) framework develop practical offline ROI maximization algorithm. Our proposed algorithm, ROIDICE, yields an efficient policy that offers a superior trade-off between return and accumulated cost compared to policies trained using existing frameworks.
1 Introduction
In economics, Return on Investment (ROI) is a financial metric used to evaluate the profitability
of an investment relative its cost. The concept of ROI originates from the work of (5) and is
widely regarded as valuable metric by the majority of marketing managers [4] ROI is calculated by dividing the profit generated from the investment by the cost of the investment, and it key
indicator for evaluating the efficiency and effectiveness of various economic decisions and strategies
lower cost. [17] Maximizing ROI is crucial decision with a high ROI yields higher return at a relatively
While ROI is a compelling metric for optimization, ROI maximization for decision making has not been actively explored within the field of machine learning. Previous studies on ROI maximization S have focused on multi-armed bandit scenarios, where reward and cost are immediately accessible following a decision. When it comes to sequential decision-making setups like the Markov Decision Process (MDP) that additionally considers a function, optimizing sequence of decisions policy to maximize the ratio between return and accumulated cost becomes far more challenging and remains mostly unexplored. This optimization contains two major difficulties: the fractional relationship between the return and accumulated cost of policy, and the fact that both are long- term quantities collected after a sequence of decisions or actions of the agent. The issues make ROI maximization via existing policy optimization techniques not straightforward, as it includes computing the gradient of the ratio between the two expected values.
For effective optimization ROI, we refer to the dual formulation of V-LP [6], which represents policy optimization in reinforcement learning (RL) as linear programming in terms of the stationary distribution. The stationary distribution of policy is the discounted sum of probabilities of the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

state-action pairs visited by the agent following the policy. Leveraging the property of stationary distribution, which allows for easy estimation of both return and accumulated cost, we derive ROI-LP, linear programming problem for ROI maximization. In this paper, we focus specifically on the offline setting and aim to derive an offline ROI maximization algorithm. The offline setting, which involves optimizing a policy using fixed dataset of pre-collected experiences without further environmental interactions (e.g., offline RL), has been actively studied recently due to its compelling use cases 2 2 To D1 20). Building on the derived ROI-LP, we incorporate convex regularization to address the distribution shift inherent in offline learning. This leads to our proposed algorithm, ROIDICE, which achieves offline ROI maximization via stationary distribution correction estimation.
Through series of diverse experiments, we compare ROIDICE algorithms from other offline policy optimization frameworks, including offline RL and offline constrained RL. We demonstrate that maximizing the ROI of policy leads to a different, more efficient behavior compared to frameworks that focus solely on maximizing policy return. In summary, our contributions are threefold:
Formulation of ROI maximization framework in an MDP with a cost function.
Derivation of ROIDICE, offline ROI maximization algorithm that optimizes the ROI of policy using a fixed dataset of pre-collected experiences.
Comparison with existing offline policy optimization frameworks, showing that ROIDICE optimizes ROI and improves policy efficiency across various domains.
2 Backgrounds
Markov Decision Process with a cost function We consider an infinite-horizon discounted Markov Decision Process (MDP) with a cost function, represented as tuple M = (S, A, T, Po. 7)- Here, S the set of states, A is the set of actions. T(s' s, a) denotes the transition probability from state- action pair (s, a) to the next state s', and r(s, a) is the reward function for state-action pair (s, a). Po(s) the initial state distribution, and 7 is the discount factor. Additionally, we introduce cost function c(s, a) > 0 for all state-action pairs (s,a).
An agent begins at an initial state following (s(s) and repeats the process of collecting reward r(s. a) and incurring cost c(s, a) by taking action a in state s and transitioning to the next state s'. The policy T (alss) represents the distribution of actions be taken by the agent in state s. The performance of the policy π can be evaluated with the expected return Rr = Ez 1 (rr(, dz)] and the expected accumulated cost C = Ex IE y'c(s,, a.)] which are the expected cumulative sums of discounted rewards and costs over rollouts of the policy n.
The main goal of reinforcement learning (RL) is to optimize policy to maximize expected return R2 collected by the agent. Constrained RL optimizes the policy to maximize its expected return R, while constraining its expected accumulated cost C₂ to be lower than a given threshold Cihreshold-
Stationary Distribution and Linear Programming (LP) formulation of RL The stationary distribution of policy d, represents the discounted sum of probabilities of the state-action pair (s. a)
visited by an agent following policy æ, i.e., (s, a) (1 - 7) Ei= " >'Pr(s: = 8, a = a). Using the stationary distribution, Rz and Cz can expressed as linear combinations of the stationary distribution and the reward and the cost respectively: R₂ I- Σn, da (8, a)r(s, a) and C,
1 Es, dx a)c(s, a). This characteristic allows the RL problem to be formulated as a lincar programming (LP),, known as the dual of V-LP in [16..
Σ M(s,, [()s,a)r(ssa)
max d20
5.a
s.t. (B.d)(s) (1 ()))) + (s) Vs = E., T(s|s, a))(s.a)
where (B.d)s)= = d(s and
Constraint (2). known as Bellman flow constraint, is requirement that any stationary distribution must satisfy. The condition can be interpreted as ensuring that the probability of state-action pairs leaving state matches the total probability of state-action pairs entering plus the probability of state s initiating a trajectory. Once the optimal stationary distribution d' is obtained, optimal policy nt can be extracted from it by x (als) = d' d'(s,a)) Ea d" (s, a') Vs,a.

---

Offline RL and DICE-RL Framework In this paper, we assume an offline policy optimization setting where interaction between the agent and environment not allowed. Instead, the agent is provided with a fixed dataset of experiences D = {(s,, di, ri, Ca, &)) R= consisting of N transition samples, to optimize its policy. A unique challenge in offline RL is balancing between policy optimization and distribution shift, as it is not possible to gather samples from the optimized policy.
In [2]. regularized policy optimization is proposed to penalize return maximization ( with f- divergence between stationary distributions of the trained policy d and the behavior policy dp. defined as (y(didd == E., do(s, a) (d(s, a)/dp(s, a)). The convexity of leads to a convex optimization problem that balances return maximization and distribution shift:
d(s,
Σ dd(s,a)r(ss a)r(a)))) - Edd(s.o) a)f dd(s,a)
max d20
2,0
6,0
s.t. (B) = (1-)) (T.d))ss)) Vs
where a is hypper-parameter that controls the trade-off. In the constrained setting studied in 13 cost constraint Σe, d(s,, a)c(s, a) S Cihreshold is added to the problem.
Previous studies have leveraged the characteristics of the above convex optimization problem to develop practical offline RI. algorithms. They follow a similar approach by applying a Lagrangian multiplier v(s) the Bellman flow constraint (2). reformulating the Lagrangian dual of the problem into the main loss function of the algorithms. We refer to this family of algorithms as the DICE-RL framework. Detailed explanations of the loss functions and the full derivations of related DICE-RL algorithms (OptiDICE 12 and COptiDICE 13 are provided in Appendix A
Linear-fractional Programming Lincar-fractional programming is of two affine functions over polyhedron. Its standard form is given by:
defined as maximizing a ratio
c_ x +
max f(x)
s.t. Gx hh -x
(3)
d=
The domain of f is restricted ensure the denominator remains positive, i.e., (xld"x x 0). as zero denominator would result in infeasible solution.
Assuming the region satisfying the constraints is non-empty and bounded, any linear-fractional programming can be transformed into equivalent linear programming using the Charnes-Coope
transformation []]. The main idea of the transformation is to replace variable x with two new variables t 2 0 and y. such that t(d''x x + 3) and = tx. This change of variables eliminates the denominator of (x), resulting in the following linear programming problem:
max TTy + at s.t. Gy 3 th, Ay =bb y,200
y
3 ROI Maximization in Linear Programming form (ROI-LP)
In this section, we start with a return on investment (ROI) maximization problem by substituting the objective of the dual of V-LP from return R₂ to ROI. This results in linear-fractional programming problem, as the objective is the ratio of two linear functions representing return and accumulated cost. We then apply the Charnes-Cooper transformation to derive an equivalent linear programming (LP), ROI-LP.
ROI Maximization Problem We begin by defining the ROI of policy in a Markov Decision Process (MDP). The ROI of policy is the ratio between the return R, and its accumulated cost R which can be expressed in terms of its stationary distribution d,: ROI(T) R_ E., ((. E.. (s..))(()))
Substituting the objective of the dual of V-LP ① with ROI, we get our ROI maximization problem, which adheres to the standard form of linear-fractional programming in (3).
E.a d(s, a(ssa)r(s,a) E.g ((s
maX d20
s.t. (B.d)(s) 7))((s) + (T.d)(ss Vs.
(4)
3

---

We apply the Charnes-Cooper transformation to reformulate the ROI maximization problem 4 into an equivalent linear programming problem, ROI-LP. This involves introducing
ROI-LP
non-negative variable 2 O and replacing d(s,a) with d' (8, a) and I, which are defined as:
d' ds(s, a) E. a(s, d(s,a);
t T= E.,a d(s, a)({s,)
and thus d(s.a) d' (s, a)tt Substituting d, ROI-LP, the linear programming formulations for maximizing ROI, is given by:
Σ d((s,a)r(s,a)
12,0.20
s.t. (B.d)(s) -(1==((11000 t(-- ()po(s) ((T.))))) Vs
Σ d'(s.a)((s,a)= =1
(6)
where the equality constraint (6) is newly introduced to ensure that (5) is satisfied. Once the optimization on ROI-LP complete. optimal stationary distribution d' (s. a) is obtained by dividing (s. with L*.
Remark. When c(s, a) is fixed constant C' for all states and actions, the optimal solution of ROI-LP becomes equivalent to that of the dual V-LP. this case, the optimal becomes き as per ( since any stationary distribution satisfies Σ,, d(s,, a) 1. Consequently, the optimal d becomes d' from the dual of V-LP scaled by t. This special case illustrates that ROI maximization simplifies return maximization when the accumulated cost is constant, regardless of the policy.
4 Offline ROI maximization
In this section, we derive an offline ROI maximization algorithm that extends ROI-LP to also include the trade-off for distribution shifts. We formulate convex optimization problem for offline ROI maximization and use derive ROIDICE, the first offline algorithm to optimize policy efficiency in terms of ROI given a fixed dataset.
4.1 Regularized ROT Maximization Framework
To formulate offline ROI maximization problem, add convex regularization ROI-LP representing a distribution shift. The approach is similar to the DICE-RI. framework (13, which incorporates f-divergence between the stationary distribution of trained policy d and behavior policy dp into the dual of V-LP. Our regularized ROI maximization framework is given by:
d'(s,a)
Σ [d((s,a)r(s,a)) a Σ dn(s, a)
max 22,120
(d(ssa))))
s.t.
1(1 - ()) + (T= {(s =1
(7)
(8)
where the design of f(x t) will be provided later in this subsection. We denote Dyæ(d', dp)
E., dp(s, f d'(sa)
[n(s,a] t).
The primary difference between our proposed problem and DICE-RL framework lies in the convex regularization. The f-divergence (y(d)dd)) from DICE-RI. framework is not directly applicable to d' as, as d'(s, a) is not valid probability distribution; it is derived by scaling the stationary distribution d(s, a) by I. However, naively incorporating Dy ( (ldd) results in the loss of the convexity, as [({) of f-divergencc is generally not convex with respect to L. To this end, we design new convex regularization that measures the amount of distribution shift while not breaking the convexity of the problem. We begin by outlining three conditions that the regularization should satisfy.
1. f(x, should be convex in æ given t > 0 and convex in given 0.

---

2. f(r, t) should be zero when æ = t.
3. f(r,t) should be well-defined on x: > and >
The first condition is that the problem must maintain convexity with respect to both and t. The second condition is that it must effectively regularize the distribution shift. Specifically, Dy dp) should be minimized to zero when the stationary distribution d(s, a) = d'(s, a)/t matches dp a). The last condition is avoid infeasible optimization scenarios, e.g., negative values inside log function, We introduce two convex functions as examples.
1 (x= 1)), 12(1, fi () 2
2 (z t)² log( -
x x2t 0< <t
From f(x) (z - 1)2 diveeeege,,e, we can obtain the first function fi (z, t) by shifting
it by 1- We introduce softened variant of fi. denoted as f2, where a scaled version of the function f(x) = log x x + 1 (arising from the KL divergence) is seamlessly combined with the left segment of f1- Although fa(z, t) is not entirely convex with respect to t, it serves valid regularization term, as f2(2, t) = 0 holds exclusively when = t. Remark. A unique characteristic of the proposed regularization D :..(ddddp) is that its magnitude is affected not only by the degree of distribution shift but also by the trained policy's accumulated cost. We demonstrate this by comparing the magnitude of Dj.a(d' ,d) and f-divergence Dy lldp) that corresponds to typical regularizer of DICE-RL framework. In the case of X divdiverg, d'is.a) ( dp) = DD ( [[dp) as 1,( = Es,a (8, a) 2 (p(a,a) - Since represents the inverse of the expected accumulated cost (5), the strength of the regularization with a dp) becomes stronger than the actual f f-divergence when the expected accumulated cost of the trained policy is less than 1.
4.2 ROIDICE
We are now prepared to derive ROIDICE, an offline algorithm for ROI maximization based on the Regularized ROI maximization framework. To enforce the constraints (7) and (8) we introduce Lagrangian multiplier v E RIS and μ E I::
min T(V, (4,,,,,,,, w. (,())=)) Σ wr(s (((s,))d Σ d)((s))s( w20,120 V.N
(9)
+ Σ (11- 7)0()) + (B.d'( + (1 (1- E w(s (o(s:))d((s,a)((sa))
where w(s a) = d's, a)/dp(s, a). We follow a similar derivation process as outlined by 12 and obtain a closed-form solution of w (s)
w (e.,e(s,0 = max 0. (F(()))1 (ecss, a) uels, aelo,..,)))
(1 .)) Vs, a.
(10)
a
where (u(s, a) = r(s, a) +yE T(s'ss, a))(s) - v(s) and (f') 2.
(y. t) is the inverse function of dz fxx () with respect to x. Loss functions of v, j and I are obtained by substituting (8, a) into
[(,, u. w. t):
min Esppo - >) + E ( (s) () - pc((s af (s [) min (saa)ddd [w (8, (s, nc(ssa)) as (w a), n,
max Exppo [1(1 - ))sss) R(aaa)ddo (s, 120
After the minimization on and t is complete, we apply policy extraction techniques from 12 to obtain the policy that generates optimal stationary distribution d' (s, a) 1 w " (s, a)d a). In this paper, we adopt weighted behavior cloning method, whose loss function is given as,
max E (s.a)dd
w i a) log (a)
(11)
The more detailed derivation and algorithm of ROIDICE can be found in Appendix
B
5

---

Offiine
3.5
5.5
3.0
2.0
2,0
1.5
4.0
1000
2000
1000
2000
50
1000
2000
Number Trajectooies
Trajectories
Number Tajectories
(a) ROIDICE vs. offline RL
True MDP
MLE MDP
Offline RL (N=50)
(N-100)
Offline RL (N-2000)
3.0
ROIDICE (N=50)
ROIDICE (N=100)
ROIDICE IN-20001
Constralned (N-50)
Constrained RL (N- 100)
0.7
0.9
1.0
1.3
0.7
0.9
1.0
1.1
1.3
Constrained (N=2000)
cost multiplier
cost multiplier
ROIDICE vs. offline constrained RL.
Figure I: Comparison of ROIDICE with other offline algorithms, We average the scores and obtain +2x standard error using 1000 seeds. N denotes the number of trajectories within the dataset.
Experiments
In this section, demonstrate that the offline policy obtained from ROIDICE achieves superior trade-off between return and accumulated cost compared to other RL approaches such as offline RL and constrained offline RL, in both finite and continuous domains.
5.1 Random Finite MDP Experiments
We test ROIDICE Random MDP setting similar to 112 A tabular MDP with [S] 50 and |A| = 4 is randomly generated and behavior policy that achieves 90% of the optimal return is used to gather fixed datasets with {10, 20, 50, 100, 1000, 2000} trajectories. We estimate the empirical transition probability (ss)) 8 a) with the given samples and use it to run tabular version ROIDICE. Details the experiments are provided in the Appendix C We compare ROIDICE with DICE-RL on other offline policy optimization schemes, OptiDICE [12] for offline RL and COptiDICE [13 for offline constrained RL.
vs. Offline RL In Figure la we compare the performance of ROIDICE with OptiDICE. While OptiDICE achieves the highest return, incurs higher accumulated cost than ROIDICE. This is expected, as RL does not account for the accumulated cost it incurs. In contrast, ROIDICE optimizes the policy's i ROI, resulting a highly efficient policy that delivers a substantial return a relatively lower accumulated cost.
vs. Offline Constrained RL In Figure [b we compare the performance ROIDICE with COPiDICE.. Offflin constrained RL aims to maximize return while keeping the accumulated cost below a specified threshold C₂ < Citreshola- As demonstrated Appendix A COptiDICE optimizes its policy using penalized reward function r(s, a) Xee(s, a), where X the Lagrange multiplier for the cost constraint. To ensure satisfaction of the cost constraint, A updated to increase when the constraint is violated. To set the cost thresholds for experiment, cost of the ROIDICE policy on the Maximum Likelihood Estimate (MLE) MDP and multiply with [0.7, 0.9, 1.0, 1.1, 1.3). The left side of the figure shows the true ROI performance of the agents, while the right side shows the ROI performance on the MLE MDP, representing the performance agents expect. The cost thresholds from the MLE MDP based the offline assumption which excludes the interaction with the true MDP.
obtain the accumulated
With sufficient number of trajectories in the offline dataset, COptiDICE with cost multiplier of 1.0, which corresponds to the same cost budget as the optimized ROIDICE policy, achieves the best ROI among the tested multipliers. Both increasing or decreasing the cost threshold result in reduced

---

Table 1: ROI of ROIDICE compared with offline RL and offline constrained RL algorithms. We average each score and get =2x standard error with 5 seeds across 10 episodes. The task name is succinctly stated: Hopper (H), Walker2D (W), Halfcheetah (HC), and Finance (F).
Task
ROIDICE
OptiDICE
COptiDICE
CDT
50th
80th
50th
80th
H-m
9.21
3.58++0.03
7.37=#0.19
7.21
3.04+10.2
3.0
H-m-e
8.29::0.18
4.93
7.6:10.44
7.88+0.21
3.71
3.73:10.16
H-e
8.51:10.25
5.23+0.04
7.940.14
7.9940.18
-0.1+0.17
-0.17++0.23
W-m
3.06:10.14
2.34
2.66+0.63
2.92+:0.48
0.6=00.4
0.82+0.31
W-m-e
4.21:10.78
4.03
3.45
3.3::0:49
0.91-10.49
0.77:L0.35
W-e
5.01-1:.42
4.59
4.25++012
4.2++0.12
-0.01+0.2
-0.01++0.2
HC-m
8.48-10.19
6.18++0005
8.04-10.4
7.98+-0.32
-0.03++0.01
-0.01+00.01
HC-m-e
10.17::0.4
9.04+11.0
8.31
8.06++0.47
6.12+0.78
6.02++0.08
HC-e
12.69-1..09
12.7410.13
7.42
0.91
12.6710.07
12.66::0.06
F-M
43.2.13.113
40.3+10.29
24.41:..5
16.345.84
20.5+
20.5+
F-H
47.2:18.67
41.8:1..3
27.316.41
26.4144.1
46.5
46.5
policy efficiency, as constrained RL agents typically use the entire available budget to maximize return. When the number of trajectories insufficient, ROIDICE is rarely outrun by COptiDICE as the large discrepancy between True MDP and MLE MDP causes ROIDICE to be optimized on the erroneous MIE MDPs.
While theoretically possible to identify a policy with optimal ROI using a constrained RL. algorithm, doing requires multiple runs with different cost thresholds determine the best one. This process can become computationally intractable and less accurate as the complexity of the problem increases. In contrast, single run of ROIDICE is sufficient to find policy with high ROI without the need to explicitly search for the right cost threshold.
5.2 ROI Maximization in Continuous Domains
We evaluate our algorithm's 's performance across diverse domains using datasets from D4RL 6 (CC BY 4.0) and NeoRL [8 (CC BY 4.0). From the D4RL benchmark, we select three locomotion tasks (Hopper, Halfcheetah, Walker2D), and from NeoRL, include stock trading task (FinRL). To assess ROIDICE's capability to maximize Return on Investment (ROI), we conduct comparative analyses with various unconstrained constrained offline RL algorithms. We utilize offline datasets from D4RL and NeoRL,, which are collected using data collection policies that do not consider cost. Similar the random MDP experiments, Additionally, we incorporate the Constrained Decision Transformer (CDT, [5] to represent a recent approach that leverages the power of generative models RL.
include OptiDICE and COptiDICE for comparison.
5.2.1 Environment and Offline Dataset
In the three locomotion environments, observations include the positional values and velocities of various body parts, with actions corresponding the torques exerted between links. The common reward function used in recent benchmarks consists of three components: incentive for forward running speed, incentive for maintaining healthy state, and penalty for applying torques joints. In this experiment, we use the control penalty as the cost function, aiming to train the agent to strive for efficient energy consumption from an ROI perspective. We used only forward running speed as our reward function.
FinRL,, the time-driven stock trading simulator, enables us to replicate live stock markets using real market data. This allows to assess our algorithm's proficiency in both resource management and decision-male amidst uncertainty. Observations include balances, daily stock amounts, closing

---

Hopper
Wa ker2D
Halfcheetah
10-
MEDIUM
0
0.0
0.2
0.6
0.8
0.0
0.4
0.6
1.0
022
0.4
0.6
0.8
1.0
MEDIUM EXPERT
10
5
1.0
10
10-
EXPERT
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
step
step
x10²
step
x10
ROIDICE
BC20%
0C50%
BC80%
BC100%
Figure 2: ROI Comparison of ROIDICE and Dataset with varying dataset qualities. We average the each scores and get =2x standard error with 5 seeds across 10 episodes. BCn% refers to behavior cloning utilizing the top 1 of the offline dataset, ranked by ROI.
prices, and various technical indicators for each of the 30 stocks in the trading pool. Agents use their balances to trade these stocks. We define cost function based on trading volume and stock price, imposing a fee on each transaction. This requires the agent to strategically optimize its transactions, considering both stock price fluctuations and associated fees. For more details on each environment and experimental setting, see Appendix E
5.2.2 Results
Table presents our results for all four tasks. Each algorithm is trained for 100k steps and evaluated for a maximum of 1000 steps with 5 seeds across 10 episodes. The data quality levels are denoted as follows: for locomotion tasks, medium (m), medium-expert (m-e), and expert (e); for the financial task, medium (M), and high (H). Complete experimental results are provided Appendix H
ROIDICE exhibits superior ROI performance compared to other algorithms across most tasks, partic- ularly excelling in the Hopper environment where there is significant potential for ROI improvement. OptiDICE also demonstrates high ROI in environments other than Hopper due to the high ROI of the offline datasets themselves for Walker2D and Halfcheetah (see Figure 2 for dataset ROIs). However, compared to OptiDICE, ROIDICE shows greater return with lower accumulated cost in significant number tasks (see Appendix H).
The constrained offline RL algorithms are evaluated under two cost thresholds, specifically using the 80th and 50th percentiles of the accumulated costs from the offline dataset to set the cost conditions for COptiDICE (discounted with 7 0.99) and CDT (undiscounted). This constraint setting ensures that the constrained offline RL algorithms make in-distribution decisions regarding cost constraints. In some domains, particularly where costs can be lowered with slight loss in return, COPtiDICE demonstrates higher ROI compared to OPtiDICE. but without fine mee-grained tuning cost constraints, it does not reach the performance of ROIDICE. Conversely, CDT tends to overfit under the current experiment settings when the dataset lacks diversity. In most cases, CDT either fails to optimize a policy (resulting in negative ROI), optimizes policy with very high return but also very high accumulated cost (resulting in a small ROI).
Figure 2 illustrates how ROIDICE effectively learns from diverse behaviors in the dataset. We compare the ROI of ROIDICE with that Behavior Cloning (BC) that clones behaviors only from high ROI trajectories. We compute the ROI of trajectories in the dataset and select the top 20%. 50%, and 80% ROI trajectories train BCn% agents. We can observe that in significant number of experiment settings, ROIDICE outperforms BCn% agents. This demonstrates ROIDICE's capability

---

OpdiDICE
ROE: 4.962 Discounted return: 149.829
Accumulated cost: 30.247
ROIDICE
ROE 8.145
Discounted 147.821
Accumuluted 18.148
Figure 3: Visualization of ROIDICE and OptiDICE in Hopper environment.
stitch together various segments from different trajectories optimize a policy that behaves more efficiently than any other trajectory in the dataset.
Compared to locomotion tasks, the financial task adds another challenge of high stochasticity. Despite this, ROIDICE demonstrates strong performance on the financial task with both medium and high- quality datasets, shown in Table I OptiDICE, although achieving high returns, shows relatively low ROI performance due to high transaction fees (cost). The high ROI observed from CDT is actually coincidental, as CDT fails to meet the given cost constraint due to overfitting; its return maximization policy accidentally results an efficient policy with high ROI.
Qualitative Behavior Examples Figure SigureBillusee Billustrates the different behaviors between ROIDICE and OptDICE in the Hopper task at the same time steps, selected to clearly represent these differences. Recall that in the locomotion task, the cost corresponds to the amount of torque applied, i.e., the energy consumed forcing it to move forward. OptiDICE focuses solely on maximizing return, resulting in the agent making large, far-jumping movements, even applying torque to the joint after the jump, which is inefficient for obtaining additional rewards. In contrast, ROIDICE finds the optimal behavior to increase return while reducing energy consumption. Consequently, the agent attempts to hop far but avoids inefficient actions that decrease ROI.
Conclusion
In this paper, propose novel policy optimization framework designed to maximize the Return Investment (ROI) of the policy, the ratio between the return and the accumulated cost. Maximizing ROI is particularly relevant in practical scenarios: the fuel efficiency (km/L) of autonomous vehicles, managing the trade-off between decision quality and planning time in meta-controllers for real-time decision-making or balancing generation quality and inference time in large language models (LLMs). The ROI maximization is approached through linear-fractional programming that incorporates the stationary distribution of the policy. We apply the Charnes-Cone Cooper transformation convert the problem into equivalent linear programming, ROI-LP. Motivated by the DICE RL frameworks, we extend ROI-LP develop our offline policy optimization algorithm, ROIDICE, by incorporating convex regularization designed to properly regularize the distribution shift. We show that ROIDICE yields policy with better efficiency than policies from the existing RL-based optimization methods by considering the trade-off between return and accumulated cost.
7 Limitations
While our work is innovative in optimizing the ROI of policy, it functions within the constraints of an offline setting where direct interaction with the environment not feasible. Consequently, the ultimate performance of the policy optimized with ROIDICE is contingent upon the quality of the provided dataset. Since ROIDICE takes into account both returns and accumulated costs, sensitive to the design of the reward and cost functions. For instance, the agent may behave similarly

---

to one trained by unconstrained RL when the cost function approximates constant. Conversely, the agent may exhibit behaviors aimed at minimizing costs excessively, akin laziness, when the cost function varies significantly.
Acknowledgements
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IIT)) grant funded by the Korea govermment (MSIT) (No. RS-2022-1122033 Develop- ment of Goal-Oriented Reinforcement Learning Techniques for Contact-Rich Robotic Manipulation of Everyday Objects, No. RS-2024-0045788 AI Research Hub Project, and No. RS-2019-II1
Artificial Intelligence Graduate School Moogram(Koee) University)). the [ITPtlnt of Information & Coummunications Technology Planning & grant funded by the Korea (Mvisty of Science and ICT), the NRF (RS-2024-0045) 1662 funded by the Ministry of Science and ICT, Korea, BK21 Four project of the National Research Foundation Korea, and NSF A14OPT AI Centre.
Technology Research
10

---

References
[1] Abraham Charns and William W Cooper. Programming with linear fractional functionals. Naval Research logistics quarterly, 93--4)188-188, 1962.
[2] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 14:15084-15097,
2021.
[3] Yuan Deng, Negin Golrezaei, Patrick Jaillet, Jason Cheuk Nam Liang, and Vahab Mirrokni. Multi-channel autobidding with budget and roi constraints. In International Conference Machine Learning, pages 7617-7644. PMLR, 2023.
[4] Paul W Farris. Marketing Metrics: The Definitive Guide to Measuring Marketing Performance. Pearson Education, Inc. 2010.
[5] Dale L Flesher and Gary John Previts. Donaldson brown (1885-1965); the power of an individual and his ideas over time. Accounting Historians Journal, 40(1):79-101, 2013,
[6] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[7] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02328, 2023.
[8] Jiyy Guan, Guang Chen, Jiamiin Ji. Long Yang, Zhijum Li, et al. Voce: Variational optimization with conservative estimation for offiin safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[9] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021.
[10] Iya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint 10.0666,, 2021.
[11] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 1191, 2020.
[12] Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120-6130. PMLR, 2021.
[13] Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup. Kee-Eung Kim, and Arthur Guez. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations, 2022.
[14] Zuxin Liu, Zijian Guo. Haohong Lin, Yihang Yao. Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning, arXiv preprint arXiv:2306.0930) 2023.
[15] Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning, pages 2161 16611-21630. PMLR, 2023.
[16] Ofir Nachum and Bo Dai. Reinforcement learning via fonchel-rockafellaar duality, arXiv preprint vwXiv:2001.08888 2020.
[17] Patricia Pulliam Phillips. Return on investment (ROI) basics. Association for Talent Develop- ment, 2023.

---

[18] Rong-Jun Qin, Xingyuan Zhang. Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: near real-world benchmark for offline reinforcement learning. Advances in Neural Information Processing Systems, 15:24753-24766 2022.
[19] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning, arXiv preprint arXiv:1910.01708, 7(1):2, 2019.
[20] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations, 2023.
[21] Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 776-0 2022.
12

---

A Full derivation of DICE RL algorithms
In this section, introduce how DICE RL algorithms are derived from the regularized policy optimization framework. We derive DICE RL algorithms for both unconstrained and constrained scenario: OptiDICE [2 and COptiDICE [3]
A.1 Derivation of OptiDICE
We show how OptiDICE is derived from the regularized policy optimization framework in convex optimization of the framework given as,
12 The
max Σ Ed(s, a))(8)) E dss d(s,a) d20
d
S.t. Σ d(s ( -1--7)p(ss) + Σ IT(sss
where convex function f(x) is convex function generally used in f-divergence: f(x) = })((zz 1)² of x2 diverrgee fz) = log x + of KL divergence.
Assuming d(ss, a) w(s, a (s, a) is the stationary distribution of the dataset (p(ss weighted by w. the Lagrangian dual of the convex optimization problem is given as,
max min L(w, v) = Σ w(s (() - a Σ d(((sa))((((saa)) 120
(12)
8.a
,,a
E(s) v(s) (1 (1- (s) +7 Σ
TT( Σ w(s ((dd(sss
8,a
where v is the Lagrangian multiplier for the Bellman flow constraint.
We use E,v(s) E., T(s|s, a)w(s, (), E., (ss,a)d a) E. ((s(s, to switch the order summation and transform the Lagrangian.
min L(w, v) (1 - Le- [0)] + (s, /w ()ess))))(sss))) aafa:s,,))a((((.)))))) w20
where (u(s, a) = r(s, +.
- v(s).
OptiDICE 12 simplifies the max-min problem by switching the order of optimization from max020 min, I(w, v) to min, max Iww, v). The optimal solution not affected due to the strong duality of the problem. The switch enables the closed-form solution of stationary distribution ratio w of the inner maximization problem that satisfies @L(w, v)/Ow(( a) = 0 VV,a. a.
(, a) = max (0.C0)- (cv(s.a) (()))((((())))
(13)
The closed-form solution w (s, a) can be plugged in L(w, v/) to derive v loss of OptiDICE given as, min L(w v) (1 Eu [s]s + (sa)) (s, a)ev(s, (s
(14)
After minimization on is complete, optimal n² plugged in w ( a) to obtain the optimal stationary distribution d' (s, a). Policy that generates d' (s, a) is trained by using policy extraction techniques from [2]. In this paper, we adopt weighted behavior cloning method, whose loss function given as,
max BaaxE Ed (we- (s, a) log Ta (a]s)] #
(15)
A.2 Derivation of COptiDICE
We show how COptiDICE, the extension OptiDICE to offline constrained RL, is derived. Con- strained RL assumes additional cost function c(s, a) and limits the discounted sum of cost
13

---

Edss, a))(s,)) a) v threshold- The constraint is added to the regularized policy optimization frame- work in [3
Σ d(s.a)r(s,a) Ed d(s.a) dp(s,a)
max d20
Σ d(s, a) =(1-7)poss) T((
s.t.
Σ d(s, a))(ss(s) a) I Citreshold 9,0
Lagrangian multiplier À is applied to the cost constraint and the Lagrangian of the constrained problem is given as,
max min () I(w, X X(Citreshold #20 U.
Σ [=((s_)ddb(saa)((s.)))
(16)
After the similar derivations from OptiDICE, the closed-form solution of stationary distribution ratio and loss functions for are given as,
(8,1(,, (8, a) max (0.cr)-1(0)))) (cualo..))
a
(17)
min (1 - 71) [1()] (B()))+B((a))a ((aa) (W (s, (s, - a (wi
(18)
where (r, a) := r(s, tc(s,a) Eg T(s'ls, v(s). Lagrangian multiplier determines how much reward function r(s, a) is penalized by cost function c(s, a) and its loss function is given as,
Σ 5,a
max A
Chhreshold
(19)
where À is updated to satisfy the constraint by increasing when discounted sum of cost Ex,a d(,, a))(s,)) a) is larger than the cost upper bound Cihreshold
B Derivation of ROIDICE
We follow similar approaches from Appendix A to derive ROIDICE from the Regularized ROI maximization framework. The main goal of the framework is to obtain the optimal stationary distribution d' (s, ds a)/t" that maximizes ROI of the policy from the optimal solutions of the convex optimization optimization given as,
Ed [d((s,a)r(s,d)) a ddl(s,a) ( 2220.20
[d'(s,a),
(dp(s,a).))
s.t. (B)d' = (010)))0(0 7))p((s) + =1 =1
(T=d')
Σ d(((sa)c((s,a)
We derive the Lagrangian of the convex problem by introducing Lagrangian multipliers and μ for the equality constraints. We change the optimization variable d' (s, a) to wos a) ds a) ///ppss, a), but the ratio not valid stationary distribution ratio contrast the previous DICE RL algorithms.
min L(v, H4 w. w.t): := Σ ( ()) w20,20 d.
Σ ddd(s,))(w(ss.)))
+ (6) ((1--))0())) (TTd (B))') + p 1- Σ
wo u(s,a)dd(ss.))cs.o)
8,0
14

---

We use (s
- Ex, (rE T(s(s,a))()) - -v()) to ob-
tain another expression of L(V, H4, t).
max min LL,, w, - Esorpo [11 - -7)p(so) + w20,120
-cc(s,)))
aa((w(s, +
where ()) r() +. (( a)u(s)) - v(s).
We follow the approaches from OptiDICE 12 and switch the order of the optimization from max miny,u LV,, μ4, t) to max120 mint,, max L(v, μe, w, f) to obtain the closed-form solution of w. Optimal ur that satisfies 8L(2.p.a..)
terms of v, f and f given as,
Bu(a,a)
/ p a) max (0.(0)-1 ( (
pc((s (
(20)
a
where (y. f) is an inverse function of az fxx f) with respect to æ. Loss functions of v, p and t are obtained by substituting (,,n a) into Lvv, μ, w):):
min Lu. == Esupo [t(1 ())(s}]
(21)
+ vo (s
pc(s, a)) as af a). pd(s, -cl(s,)))) as (s (), t)] + p.
min Lp = E (s,a)dd (8,
1
max L. = Ey-po [t(1 - [))[s)] B(oa))do 120
Algorithm ROIDICE
Input:
The offline dataset D, the initial state offine dataset Po. parameterized Lagrangian multipli- ers Vg-/4 ns, t, and policy ma.
Output:
Optimal policy Tt. Initialize parameters.
while convergence do Update Vo: μ., with Lugs Lur - Lr
Update with Lo - B(aa)) It (isaal(,, a) log To (als)] end while
15

---

Ofite R-W050
Coneraaa 14 (.
Offine R. 6.00-004
50 4018C 200
Conersiied 14 (ew22a
3.0
3.5
3,0
RETURN
5.0
2.5
2.5
4.5
2.0
4.0
2.0
1.5
3.5
0.7
0.7
Cost Multiplier
Cost Multiplier
Cost Multiplier
(a) True MDP
RETURN
80
0.9
1.0
1.1
0.7
1.1
13
1.0
1.1
Muttipller
Muttipller
Multipller
(b) MLE MDP
Figure ROIDICE vs. offline constrained RL. We average the scores and obtain =2x standard error using 1000 seeds. N denotes the number of trajectories within the dataset.
C Details of Random MDP experiment
We evaluate ROIDICE on random MDP environments and compare with unconstrained and con- strained RL algorithms, OptiDICE and COptiDICE. We generate tabular environments with 1000 different seeds. Each environments consists of 50 states and actions. Their transition probability randomly generated following Dirichlet distribution. The sparse reward is given when agent reaches on the goal state, and the cost function follows Bela(0.2, 0.2) distribution. We add to all states in order satisfying the domain condition of linear-fractional programming.
if is the goal state otherwise
rs,
I-7
0
0 110 ++1 otherwise
& the goal state
c(s,a)
where the discount factor y = 0.95 and B Beta(0.2,0.2)
After the MDP wih cost function is generated, we obtain behavior policy TD(als) that achieves 90% of the optimal return gather fixed datasets with {10, 20, 50, 100, 1000, 2000) trajectories. The dataset is used to estimate the empirical transition probability T(s'|s, a) with the given samples.
With the estimated transition probability T(s'|s, a) and behavior policy TD(als). we apply the estimated MDP tabular version of ROIDICE. For the tabular version of ROIDICE, we apply the primal problem of the Regularized ROI maximization framework to a convex optimization solver software. For the convex regularization fx, t), we use (z, 1 (x t)² and is set to be the inverse of the number of trajectories.
d'(s,a)
max d' 8'20.20 Σ dd'(s,))(ssa))- ddb(s,a) do((s)))))
- s.t. =t) + (F
(22)
Σ d((s,a)((,,a)=1 8.0
(23)
where (s.d)ss= =x.a ((s
16

---

Hopper
KerED
Halfcheetth
755
20
OUL
Lo03
1o03
BC50%
BC100%
Figure 5: Comparison of different levels of the hyperparameter (a) of ROIDICE in locomotion environments using expert data quality. We average the scores and obtain #2x standard error using 5 seeds across 10 episodes.
D Impact of Various a
We investigate how varying a, which controls the strength of the regularization, affects the perfor- mance of ROIDICE by sweeping its values. Figure Figure shows the ROI performance for a ranging from 1 x 10 to 1.0 using expert quality dataset. Recall that the Hopper task datasets generally contain trajectories with low ROI, while most trajectories high ROI (see Table 2) In the Hopper environment, ROI performance decreases indicating an opportunity to increase ROI when the model is less regularized, as the behavior policy collecting the offline dataset mostly does not account for the cost. In the Halfcheetah environment, ROIDICE achieves high ROI when following the offline dataset due to the dataset's properties, but shows largely varying performance with small due its lack of coverage outside the dataset. In the Walker21) environment, ROIDICE shows robustness across various levels of a.
Walker2d and Halfcheetah achieve relatively increases,
E Data and Environment Details
We assess ROIDICE across four environments: three locomotion tasks (Hopper, Halfcheetah, Walker2D) and one financial trading task (FinRL). We utilize environment configurations and datasets provided by [6] (CC BY 4.0) and [18] (CC BY 4.0), with modifications to reward and cost functions. Additionally, incorporate absorbing state and absorbing action precisely calculate the sum of discounted rewards and costs, including an additional dimension to indicate whether the current state and action are absorbed.
Locomotion Task In the Hopper environment, the reward is multifaceted, consisting of two primary components: the forward reward and the control cost. The forward reward incentivizes the agent to make progress towards its goal, while the control cost penalizes actions that are overly aggressive or energetically inefficient. Halfcheetah and Walker2D environments introduce an additional aspect to the reward: healthy reward. This rewards the agent for maintaining stability. The control cost serves as a mechanism to penalize excessive actions, reflecting the idea that in real-world scenarios, significant force often comes with considerable energy expenditure. This cost can be as proportional to the energy consumption in physical robots, motivating agents to find smoother and more energy- energy-efficient strategies for accomplishing their tasks. We utilize the forward reward reward and the control cost as a cost, normalizing the cost for stable learning. The reward and cost functions are follows:
r(s, a) forward_reward_uve
Et I2+1
dl.
w
cls, a)
||a|| 6mh
[A
where I, represents the x-coordinate before the action at, and J+++1 represents the x-coordinate after the action. The parameter forward rewdddeewrrdddddrr weiight corresponds to the forward reward and is specific to each environment. The two hyperparameters, we 0 and b. > 0, are weights that control the strength of the cost, ensuring compliance with the linear-fractionll program condition. Additionally, [A| denotes the size of the action space dimension.
Financial Task FinRI. offers a stock trading simulation environment where agents trade stocks to carn compensation, with 30 tradable stocks available (N 30). The reward is calculated by
17

---

summing the current balance with the product of the number of shares owned and the closing price of each share. Additionally, cost function considers trading volume and stock price, imposing a fee on each transaction. The reward and cost functions are as follows:
r(s, a. 8') = w. (v(s') v(s))
(8)) = =(0] + Σ 811 31 s : 61]
s/i + min (ai sfi N +1]
if 0z < 0
c1 a) by + w
s/i +1 min(
3|=+1] a;) pct if a4
+00
where pct = 0.001 denotes per-share percentage, which is the most commonly used transaction cost
rate for each trade, i 0, 1, ......... N represents the index of tradable stocks, and $ E (O..... 181].
represents j-th dimension of state s. Each state dimension contains different information: s(0]
represents the balance, s[1 31] denotes today's s closing value for each of the 30 stocks, and s(31 61
indicates today's s trading volume for each of the 30 stocks.
Table 2: Reward and cost function hyperparameters
Hyperparameter
Value
w
1.0
6m
0.1
we
0.001
w.!
0.0001
by
0.1
Implementation Details
We implement ROIDICE algorithm based 13 (CC BY 4.0) for the tabular experiment, and (20) (MIT License) for the continuous domain experiment. Our code is available at: hhttps://githii
com/ku-dml awtaaa//Baaa /DDIIII The implementation and configuration of OptiDICE and COptDICE adopted from the framework presented [3]. while CDT utilized within the OSRL framework as described in [1].
The task and dataset name succinetly stated: Hopper (H), Walker2D (W), Halfcheetah (HC), and Finance (F); for locomotion tasks, medium (m), medium-expert (m-e), and expert (c): for the financial task, medium (M), and high (H). We set the hyperparameter a for OptiDICE and COptiDICE, which controls the strength of the f-divergence regularizer. OptiDICE and COpúDICE have similar regularizer strengths, so we set the same a for both of them. For Hopper, we set a 0.01, for Walker2D (m) 0.1, for Walker2D (m-e and e) and e) we set a = 0.01, a 0.1, and a = 1.0: for financial task we set a 1.0. For hyperparameter settings and model parameter configuration of ROIDICE, see Table
1.0, and for Halfcheetah (respectively, m, m-e,
18

---

Table 3: Hyperparameters and configuration of ROIDICE.
Task
Value
a
lc-06 (H-m, H-e), lc-05 (W-c, HC-m), 1e-04 (H-m-e, W-m), 1e-03 (W-m, HC-m-e), 1.0 (HC-e)
Locomotion
hidden dim of
MLP (256, 256)
hidden dim of æ
MLP (256, 256)
learning rate
3e-04
7
0.99
optimizer
Adam
1.0 (F-M), 0.001 (F-H)
hidden dim of
MLP (256, 256, 256)
Finance
hidden dim of π
MLP (256, 256, 256)
learning rate
3e-04
0.99
optimizer
Adam
G Experiments Compute Resources
ROIDICE was conducted on single CPU (AMD Ryzen Threadripper PRO 5995WX) with 256GB of RAM and a single GPU (NVIDIA RTX 4090). Each training session took about 10 to 20 minutes, utilizing approximately 20% RAM and from 40% to 60% of the GPU memory.
Table 4: Comparison of the runtime and number of parameters between algorithms. All algorithms, including baseline methods, were trained for 100K iterations on a single NVIDIA RTX 4090 GPU.
ROIDICE (Ours)
OptiDICE
COptiDICE
CDT
Run time (wall-clock) locomotion
10 min.
min.
35 min.
150 min.
Run time (wall-clock) finance
20 min.
16 min.
120 min.
250 min.
The number of parameters
140K
140K
357K
730K
H Experiments Results
We average the scores and obtain +2x standard eπror using 5 seeds across 10 episodes. The task names are concisely abbreviated follows: Hopper (H), Walker2D (W), HalfCheetah (HC), and Finance (F). Figure 6 illustrates the learning curves of ROIDICE and other offline unconstrained and constrained RI. algorithms. In the Hopper environment, the performance ROIDICE is significantly better. OptiDICE attains excessive returns but relatively lower ROI than ROIDICE due to its disregard for costs the environment (see Table ROIDICE achieves greater returns than OptiDICE when dealing with datasets that contain mix of expert-level and lower-level data. Our analysis shows that ROIDICE exhibits qualitative behavior by considering both return and accumulated cost. Offline constrained RL algorithms, COptiDICE and CDT, show high ROI with datasets that include high ROI trajectories. the HalfCheetah and Finance environments with high- high-quality datasets, CDT performs well but not well ROIDICE (see Table
19

---

(50%)
(20%)
coptidice (50%)
(20%)
optidice
Hopper
Walker2D
Halfcheetah
10
MEDIUM
2
0-
-2
2
4-
0.4
1.0
0.4
1.0
0.8
10.0
MEDIUM EXPERT
7.5
2.5
0.0
0.4
0.6
0.4
1.0
10.0
12.5
7.5
10.0
EXPERT
2
7.5
2.5
0-
5.0
-2
2.5
0.0
0.0
0.0
0.2
0.4
0.6
0.3
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Erv step
x10
Eni step
x10
Enw step
(a) ROI of locomotion tasks
Finance Medium
Finance High
60
60
40
20
0.4
0.6
1.0
0.0
0.2
0.6
0.8
(b) ROI of financial tasks
Figure 6: For each tasks, we report average ROI, return, and return with =2x standard error with 5 seeds across 10 episodes.
20

---

Table 5: Results of ROIDICE compared with offline RL algorithms.
Task
ROIDICE
OptiDICE
R
102.414=10.19
134.61:0.64
H-m
C
11.124
37.61
R
1.9.36=1.96
149.84+t1033
H-m-e
C
18.26=0.27
30.43:+0226
Rr
11..11
171.95
H-e
C
13.52+1.28
37.61++0.24
R
69.9410.63
111.88:16.3
W-m
C
22.92+ 1.07
47.87++0.62
R
148.71+27.95
192.13++2.46
W-m-e
C
35.35=1.122
47.68110.7
Rr
14.51
211.95:0.51
W-e
C₁
24.49:10.95
46.2:10.18
R
361.39+14.67
435.59+3.522
HC-m
C
42.57:11005
70.4:10.13
Rr
277.59-+2..47
574.94:54.68
HC-m-e
Cn
56.78+0.27
64.39+1..
R,,
762.98
15.55
HC-e
C
60.11+0.117
59.61+0022
R₂
1032.66++365622
981.2+3312.66
F-M
Cn
23.56:2.002
13.91
R₂
1093.86:4288.69
1076.16:1557.11
F-H
C
23.05++2.64
25.03++2.3
21

---

Table 6: Results of offline constrained RL algorithms.
Task
COptiDICE
CDT
50th
80th
50th
20th
R
130.02+4..0
131.45+2.35
81.7=#.09
84.03#11.63
H-m
Cx
17.64:10.65
18.25:40.65
27.08
27.43:12.02
R
93.2+22299
103.23+11..77
126.3=7.62
127.86:15.54
H-m-e
Cx
12.14#2.42
13.1
34.23=1.69
33,99:2.06
R
12..43++
114.05+:5.54
-0.6+t1.44
-1.16-11.84
H-e
Cx
15.24++0.97
14.28++0.71
7.93+-0.36
7.79+-0.41
R,
75.38
79.6=13.44
30.0+:0.799
40.04
W-m
Ox
29. 19.95
29.08:0..56
48.87
49.7413.14
R
100.7+16.86
94.33++14225
47.52++25.17
41.76++18.24
W-m-e
Cx
29.18:10.89
29.6+10.45
47.91
48.67++4.3
R₂
120.7:99.7
117.9111.45
-0.03
0.29115.45
W-e
Ox
28.38:41.64
28.03
24.96-10.39
24.98
R
122.85
297.36+:14002
-0.2610.12
-0.11+0..13
HC-m
Ox
37.07
37.21
10.0=0.0
10.0+0.0
Rr
323.48
1.0.2-416.211
432.69
423.53
HC-m-e
Cx
38.3+
38.76-+0.94
70.37+0.35
70.71+0.63
R₂
235.13:28..8
255.62++66.18
768.6744.38
768.6414.41
HC-e
Ox
31.04
31.67:20.84
60.66-0.65
60.71
R,
570.02++246.88
351.89++14499
528.08-#321.16
528.541:311.15
F-M
C=
22.75+22.03
22.13 ,,44
24.84
2..84
Rs
665.92-1 146.82
179.27-117.22
1012.48-212.15
1012.51+212.17
F-H
Cx
24.5+10.95
24.21+:0.45
21.68
21.68+0.91
22

---

I Comparison of Offline Constrained RL Algorithms
Table ROI of ROIDICE compared with offline constrained RL algorithms. We average each score and get x:2x standard error with 5 seeds across 10 episodes.
Data quality
medium
medium-expert
expert
ROIDICE
9.21=0.49
8.29+0.18 18
8.51
VOCE 50th
1.807=0.63
1.33+:128
1.34+0067
VOCE 80th
1.799:10.62
1.34
1.47:0.76
CPQ 50th
1.13=0.58
-0.06110.41
-0.11=00.71
CPQ 80th
-0.10:1..4
-0.15:10:48
-0.70:10.19
Our experiment in continuous domains from Section 5.2 includes non-DICE offline constrained algorithm, CDT. We additionally provide experiments on other offline constrained RL frameworks, VOCE g and CPQ [21]. Table 7 presents the ROI performance of these constrained offline RL algorithms in the Hopper environment (6]. These algorithms use two cost budgets, corresponding to the 50th and 80th percentiles of the accumulated costs from the offline dataset.
Over various experiments, constrained offline RL algorithms commonly have shown low ROIs. We conjecture that these results mainly due to the overestimation of costs, making constrained offline RL algorithms to be overly conservative on choosing costly state-actions. On the other hand, ROIDICE seems to be less affected by such overestimation, as overestimated costs partially mitigated with overestimated rewards when estimating ROIs.
J Experiment Results in Safety RL Environments
Table ROI of ROIDICE compared with offline constrained RL algorithms. We average each score and get + standard error with 5 seeds across 10 episodes.
Task ROIDICE
CarGoal
PointPush
0.160::..02
0.033=0.01
COptiDICE 50th
0.11940.00
0.026-10.01
COptiDICE 80th
0.115:40.01
0.022+0.01
We evaluate approach OpenAl SafetyGym 19 tasks including CarGoal and PointPush. Table provides results averaged over seeds across 10 episodes. We observed that ROIDICE outperforms COptiDICE in terms of ROI. We use the rewards and costs from the environment, adding a constant value of ε = 0.1 to each cost to maintain our assumption that c(s, a) > OAs, The offline dataset is collected by PPO Lagrangian. ROIDICE uses a 0.001 for CarGoal and 0.01 for PointPush. COptiDIEE uses a 0.01 for CarGoal and = 1.0 for PointPush.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 4.2 Appendix
C
Appendix
Appendix
E
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Appendix C
Appendix
Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
ooiiioinoliyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwtc//ggue////eeeeeeoocc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section Guidelines:
Appendix
Appendix
Appendix
Appendix
H
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section Appendix
H
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code fo Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper does not directly engage with societal elements or have immediate applications that could identified as having positive or negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Section 5.2 Appendix Appendix
F
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "aa04edeVmz", "reviewer": "Reviewer_Z94y", "summary": "The paper addresses the problem of maximizing the Return on Investment (ROI) in the context of offline reinforcement learning (RL). The method introduced is ROIDICE, which stands for Return on Investment Decision-making in the Offline Setting. ROIDICE is an offline policy optimization algorithm designed to optimize the ROI of a policy using a fixed dataset of pre-collected experiences.\nThe method involves formulating the ROI maximization problem as a linear-fractional programming problem, which is then converted into an equivalent linear programming problem using the Charnes-Cooper transformation. This transformation allows the problem to be solved using standard linear programming techniques.\nTo address the distribution shift inherent in offline learning, ROIDICE incorporates a convex regularization that measures the amount of distribution shift while maintaining the convexity of the problem. This regularization is designed to penalize the trained policy's deviation from the behavior policy used to collect the offline dataset.\nThe experiments conducted in the paper demonstrate that ROIDICE outperforms other offline RL algorithms, including those that focus solely on maximizing the return of the policy. ROIDICE achieves a superior trade-off between return and accumulated cost, resulting in a more efficient policy.", "questions": "See Weakness", "limitations": "Not applicable", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel policy optimization framework that maximizes the Return on Investment (ROI) of a policy. This is a significant departure from traditional approaches that focus solely on maximizing return, without considering the cost associated with the actions.\n2. The proposed method operates within an offline setting, where direct interaction with the environment is not possible. This is particularly useful in scenarios where online interaction is costly or risky.\n3. The authors derive an offline algorithm, ROIDICE, which optimizes the ROI of a policy using a fixed dataset of pre-collected experiences. \n4. The paper demonstrates that ROIDICE yields policies with better efficiency than policies from existing RL-based optimization methods. This is achieved by considering the trade-off between return and accumulated cost. The authors conduct extensive experiments across various domains, including locomotion tasks and a financial stock trading task. T", "weaknesses": "ROIDICE may require significant computational resources, especially when dealing with large-scale datasets and complex environments. Could you provide details on the computational resources and time consumption for different methods?\nIf the data distribution in the offline dataset significantly differs from that encountered in the online environment, the performance of ROIDICE may be affected. How can this issue be addressed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Z94y,\n\nWe thank the reviewer for the thorough and constructive comments. We hope we can address your concerns below.\n\n**1. Could you provide details on the computational resources and time consumption for different methods**\n\nWe provide details on the resources and runtime to demonstrate that ROIDICE does not require a significant amount of resources for large-scale domains, such as locomotion and finance. All algorithms, including baseline methods, were trained for 100K iterations on a single NVIDIA RTX 4090 GPU.\n\n||ROIDICE (Ours)|OptiDICE[1]|COptiDICE[2]|CDT[3]|\n|---|---|---|---|---|\n|Run time (wall-clock) locomotion|10 min.|8 min.|35 min.|150 min.|\n|Run time (wall_clock) finance|20 min.|16 min.|120 min.|250 min.|\n|# of parameters|140K|140K|357K|730K|\n    \n**2. What if the data distribution in the offline dataset significantly differs from that encountered in the online environment?**\n\nIn the context of offline reinforcement learning (RL), a major challenge arises from the degradation in performance due to increasing discrepancies between the offline data distribution and the online environment. To mitigate this issue, DICE-based RL frameworks—such as ROIDICE, OptiDICE, and COptiDICE—employ f-divergence to regulate the extent of conservativeness, which is modulated by the hyperparameter $\\alpha$. As illustrated in Figure 5 of Appendix D, which corresponds to Figure 1 in the supplementary materials, selecting an appropriate $\\alpha$ can identify an offline RL policy that is less affected by distributional shift.\n\n[1] J. Lee, et al. Optidice: Offline policy optimization via stationary distribution correction estimation. ICML, 2021.\n\n[2] J. Lee, et al. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. ICLR, 2022. \n\n[3] Z. Liu, et al. Constrained decision transformer for offline safe reinforcement learning. ICML, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lq9Zqx0sX3", "reviewer": "Reviewer_8zrU", "summary": "This paper provides a fraction linear programming framework for solving offline RL problems for return on investment (ROI). The fraction linear programming can be trasformed to a linear programming and a convex regularizer is used to control the distribution mismatch. The authors provide adequate experimental results to demonstrate the advantages of the algorithm.", "questions": "1. Can the authors provide theoretical analysis for the sample complexity like the paper  ``Ozdaglar et al Revisitting the linear programming framework for offline reinforcement learning'' which also uses LP framework to solve offline RL?\n2. How can we implement this LP using function approximation when the state-action space is large?\n3. Missing references: There are papers using LP frameworks to solve offline RL with sample complexity guarantee, such as the two paper mentioned before.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The framework is clean which uses linear programming to model the offline ROI maximization problems and the authors provide sufficient experimental results with good performance.", "weaknesses": "No sample complexity guarantee.  See questions for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 8zrU,\n\nWe appreciate your comprehensive comments. Please find the response to your questions below.\n\n**1. Theoretical analysis for the sample complexity in ROI-LP.**\n\nSample complexity analyses in the LP formulation of RL traditionally estimate a policy's return using a linear combination of its stationary distribution and reward. On the other hand, ROI is defined as a ratio between the linear combinations of the stationary distribution with reward and cost, making existing approaches not directly applicable.\n\nTo illustrate this, we demonstrate that the method proposed in [1] cannot be directly applied to determine the sample complexity of the ROI-LP. Specifically, Lemma 3 in [1] cannot be straightforwardly derived from the ROI-LP. The lemma assumes an approximated stationary distribution ${\\theta}\\in \\mathbb{R}^{|S||A|}$ and policy $\\pi_{\\theta}$ extracted from $\\theta$. $d_{\\pi_{\\theta}}\\in \\mathbb{R}^{|S||A|}$ is the true stationary distribution generated by policy $\\pi_{\\theta}$. Lemma 3 provides an error bound that relates the Bellman flow constraint violation of $\\theta$ and the absolute difference between $\\sum_{s,a}\\theta(s,a)r(s,a)$ and $\\sum_{s,a}d_{\\pi_\\theta}(s,a)r(s,a)$.\n \n **Lemma 3** For any $\\theta\\geq0$ and $r(s,a)\\in[0,1]$, we have $|\\sum_{s,a}r(s,a)(\\theta(s,a)-d_{\\pi_{\\theta}}(s,a))|\\leq{||M\\theta -(1-\\gamma)p_{0}||_{1} \\over 1-\\gamma}$,\n\nwhere $M=B_{\\*}- \\gamma T_{\\*}$.\n \nWe show that a straightforward extension of the lemma is not feasible in the ROI-LP context. The pair $(\\tilde{\\theta},\\tilde{t})$ serves as an approximation of $(d^{'},t)$ in ROI-LP, and the policy $\\pi_{\\tilde{\\theta}}$ is extracted from $\\tilde{\\theta}/\\tilde{t}$. The actual stationary distribution produced by $\\pi_{\\tilde{\\theta}}$ is $d_{\\pi_{\\tilde{\\theta}}}^{'} / t_{\\pi_{\\tilde{\\theta}}}$. We begin by assessing the extent to which $(\\tilde{\\theta},\\tilde{t})$ violates the scaled Bellman flow constraint in ROI-LP.\n \n$\\Vert M\\tilde{\\theta}-(1-\\gamma)p_{0}\\tilde{t}\\Vert_{1} = \\Vert M\\tilde{\\theta}-M\\tilde{t}d_{\\pi_{\\tilde{\\theta}}}^{'} / t_{\\pi_{\\tilde{\\theta}}}\\Vert_{1}\\nonumber\\geq(1-\\gamma)\\Vert B_{\\*}\\tilde{\\theta}-B_{\\*}d_{\\pi_{\\tilde{\\theta}}}^{'} \\tilde{t}/t_{\\pi_{\\tilde{\\theta}}}\\Vert_{1}$\n\nwhere a valid stationary distribution $d_{\\pi_{\\tilde{\\theta}}}^{'} / t_{\\pi_{\\tilde{\\theta}}}$ satisfies $(1-\\gamma)p_0 = Md_{\\pi_{\\tilde{\\theta}}}^{'} / t_{\\pi_{\\tilde{\\theta}}}$.\n        \nThe ROI difference between $\\sum_{s,a}\\tilde{\\theta}(s,a)r(s,a)$ and $\\sum_{s,a}d_{\\pi_{\\tilde{\\theta}}}^{'}(s,a)r(s,a)$ can be bounded by,\n    \n$|\\sum_{s,a}r(s,a)(\\tilde{\\theta}(s,a) - d_{\\pi_{\\tilde{\\theta}}}^{'}(s,a))|\\leq\\sum_{s}|\\sum_a \\tilde{\\theta}(s,a) - d_{\\pi_{\\tilde{\\theta}}}^{'}(s,a)|=\\Vert B_{\\*}\\tilde{\\theta}-B_{\\*}d_{\\pi_{\\tilde{\\theta}}}^{'}\\Vert_{1}$\n\nIt can be noted that, due to the additional factor $\\tilde{t} / t_{\\pi_{\\tilde{\\theta}}}$ that cannot be easily bounded under finite samples, ROI-LP cannot use the same proof technique to [1] for sample complexity analysis.\n\nWhile the sample complexity analysis is indeed an important research topic, we believe that conducting a new type of sample complexity analysis appropriate for the Linear-Fractional Programming framework is beyond of the scope of this paper.\n\n[1] Ozdaglar, Asuman E., et al. \"Revisiting the linear-programming framework for offline rl with general function approximation.\" International Conference on Machine Learning. PMLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dhNPRKpEJg", "reviewer": "Reviewer_T1wG", "summary": "This paper introduces a novel approach for solving constrained offline reinforcement learning problems. The authors apply the Charnes-Cooper transformation to convert the linear-fractional programming into an equivalent linear programming problem, and draw inspiration from the DICE framework to maximize ROI under offline setting. Experimental results across finite and continuous domains demonstrate that the proposed ROIDICE achieves a better trade-off between return and accumulated cost.", "questions": "- It is suggested to add the discussion of related works and compare them in experiments, such as [1,2].\n- Can you explain why using constraint methods in the FinRL environment actually yields poor results?\n- Can you provide experimental results on widely recognized safe RL environments such as SafetyGymnasium[3,4] and BulletSafetyGym[5]?\n  [1] Xu H, Zhan X, Zhu X. Constraints penalized q-learning for safe offline reinforcement learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(8): 8753-8760.\n  [2] Guan J, Chen G, Ji J, et al. VOCE: Variational optimization with conservative estimation for offline safe reinforcement learning[J]. Advances in Neural Information Processing Systems, 2024, 36.\n  [3] Ray A, Achiam J, Amodei D. Benchmarking safe exploration in deep reinforcement learning[J]. arXiv preprint arXiv:1910.01708, 2019, 7(1): 2.\n  [4] Ji J, Zhou J, Zhang B, et al. Omnisafe: An infrastructure for accelerating safe reinforcement learning research[J]. arXiv preprint arXiv:2305.09304, 2023.\n  [5] Gronauer S. Bullet-safety-gym: A framework for constrained reinforcement learning[J]. 2022.", "limitations": "As mentioned in the paper, the ultimate performance of ROIDICE is contingent upon the dataset and exhibits sensitivity, potentially resulting in behaviors such as minimizing costs excessively.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper's approach is mathematically sound, and the motivation is well elucidated.\n* The experimental results conducted in various environments demonstrate its superiority over the baseline.", "weaknesses": "* Important baselines are missing: apart from DICE-based methods, comparisons should include other offline constrained RL methods.\n* How is the proposed method implemented in continuous domains? The description lacks clarity; for instance, providing pseudocode would be helpful.\n* In the FinRL task, it's intriguing why the unconstrained OptiDICE method performed well while other constrained baselines showed poor performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer T1wG,\n\nWe thank the reviewer for the detailed feedback. We address your remarks below.\n\n**1. Important baselines are missing: apart from DICE-based methods, comparisons should include other offline constrained RL methods.**\n\nOur experiment in continuous domains from Section 5.2 includes non-DICE offline constrained algorithm, CDT[1]. We additionally provide experiments on other offline constrained RL frameworks, VOCE[2] and CPQ[3]. Table 1 in the rebuttal supplementary presents the ROI performance of these constrained offline RL algorithms in the Hopper environment[4]. These algorithms use two cost budgets, corresponding to the 50th and 80th percentiles of the accumulated costs from the offline dataset.\n\nOver various experiments, constrained offline RL algorithms commonly have shown low ROIs. We conjecture that these results are mainly due to the overestimation of costs, making constrained offline RL algorithms to be overly conservative on choosing costly state-actions. On the other hand, ROIDICE seems to be less affected by such overestimation, as overestimated costs are partially mitigated with overestimated rewards when estimating ROIs.\n\n**2. How is the proposed method implemented in continuous domains?**\n\nWe provide the pseudocode of ROIDICE in the rebuttal supplementary. Lagrangian multipliers are parameterized and updated to optimize ROIDICE objectives presented in our paper. Optimal policy $\\pi^*_\\theta$ is extracted from the optimal ratio $w^{*}_{\\nu,\\mu,t}$ obtained from the Lagrangian multipliers.\n\n**3. In the FinRL task, why the unconstrained RL performed better than constrained RL and what makes the constrained RL methods perform poorly?**\n\nAs reported in the appendix, in FinRL task, constrained RL algorithms are trying to achieve lower cost compared to unconstrained RL algorithms by sacrificing large amount of its return. While both constrained and unconstrained RL agents are resulting in a policy with smaller cost compared to the cost budget, we assume that constrained RL algorithms are largely overestimating the cost, resulting in a overly conservative policies.\n\n**4. Can you provide experimental results on widely recognized safe RL environments?**\n\nWe evaluate our approach on OpenAI SafetyGym[5] tasks including CarGoal and PointPush. Table 2 in the rebuttal supplementary materials provides results averaged over 5 seeds across 10 episodes. Similar to the reported results in the paper, we observed that ROIDICE outperforms COptiDICE in terms of ROI. We use the rewards and costs from the environment, adding a constant value of $\\epsilon=0.1$ to each cost to maintain our assumption that $c(s,a) > 0 \\ \\forall s,a$. The offline dataset is collected by PPO Lagrangian. ROIDICE uses $\\alpha=0.001$ for CarGoal and $\\alpha=0.01$ for PointPush. COptiDICE[6] uses $\\alpha=0.01$ for CarGoal and $\\alpha=1.0$ for PointPush. We will include the results in the final version of the paper if accepted.\n\n[1] Z. Liu, et al. Constrained decision transformer for offline safe reinforcement learning. ICML, 2023.\n\n[2] J. Guan et al. VOCE: Variational optimization with conservative estimation for offline safe reinforcement learning. NeurIPS, 2024.\n\n[3] H. Xu, et al. Constraints penalized q-learning for safe offline reinforcement learning. AAAI, 2022.\n\n[4] J. Fu, et al. D4rl: Datasets for deep data-driven reinforcement learning, 2020.\n\n[5] A. Ray, et al. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 2019.\n\n[6] J. Lee, et al. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. ICLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qb2ARCbDNI", "reviewer": "Reviewer_Yyxa", "summary": "The article proposes a novel offline algorithm for learning policies optimising Return on Investment (ROI): the ratio between the (discounted) cumulated rewards obtained by a policy, and the (discounted) cumulated costs of the actions taken. The paper focuses specifically on offline RL, and in particular builds on formalising the RL problem in a Linear Programming form. \nThe proposed approach builds on the recent literature, particularly DICE-RL, but derives a new regularisation term for ROI optimisation. \n\nThe experimental evaluation demonstrates that the proposed approach successfully learns policies with high ROI on various offline RL datasets.", "questions": "Could you provide a better rationale on why ROI would be a good objective for RL in general, beyond its use in the business domain? \n- The difference in return between standard RL and the ROI version is quite large in the experiments. How significant is the decrease in policy performance compared to cost-free policies in practice for \n- Does the proposed approach provide some guarantees on cost, compared to, eg, CoptiDICE?", "limitations": "- No significant ethical aspects to this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of optimising ROI is, to my knowledge, novel, and interesting as it proposes a new way to balance the trade-off between action costs and policy rewards, which is relevant in a wide range of applications of RL. \n- Although the RL literature has proposed ways to handle this trade-offs since its early days (for example, by modelling costs as negative rewards), there is an argument that an ROI objective could have some advantages. \n- The derivation of the algorithm is well described in the article and is a substantive contribution with respect to the state-of-the-art", "weaknesses": "- Although optimising ROI could indeed provide a better framework for cost/returns tradeoffs, the article does not provide a substantive argument of why this would be the case. This somewhat reduces the expected impact and relevance of the contribution. \n- Similarly, the experimental evaluation only compares the proposed approach to RL trained to optimise the return solely, with no constraint for cost. In this setting, it seems self-evident that the proposed method would achieve better ROI and lower costs than a policy trained ignoring cost altogether. It is fairly common in RL to approach this tradeoff by integrating the action costs as negative rewards, and it would have been fairer to compare with such an approach as an additional baseline.\n- One reason for the broad use of ROI in business, is that the quantities of \"return\" and \"investment\" are usually unambiguously in the same quantity making the ratio calculation natural (although different costs can affect the ratio). This is less straightforward in many RL problems where the return and possible costs may be in very different dimensions and distributions. It is unclear from the paper how stable are the policies optimised depending on reward/cost scaling. \n- There is an issue with a number of references in the bibliography that appear to be missing some fields (eg, refs 6,7,9,10,11)Could you provide a better rationale on why ROI would be a good objective for RL in general, beyond its use in the business domain? \n- The difference in return between standard RL and the ROI version is quite large in the experiments. How significant is the decrease in policy performance compared to cost-free policies in practice for \n- Does the proposed approach provide some guarantees on cost, compared to, eg, CoptiDICE?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Yyxa,\n\nWe appreciate your feedback and have provided responses to your questions below.\n\n**1. Could you provide a substantive argument of why optimizing ROI could offer a better framework for cost/returns trade-offs?**\n\nConstrained reinforcement learning is a framework that aims to maximize the return within a given cost budget, rather than optimizing the trade-off between returns and costs. In other words, constrained RL agent does not try to minimize the accumulated cost if it is below the budget. However, ROI maximization focuses on maximizing the return/cost ratio, meaning the accumulated cost can be minimized further to achieve a higher ROI.\n\nIf our main goal is to maximize ROI, ROIDICE is much more efficient compared to utilizing constrained RL algorithms. As described in Section 5.1, for ROI maximization, constrained RL requires multiple runs with different levels of cost budgets, and the policy with the best ROI performance can be selected afterward. However, ROIDICE requires only a single run to obtain the policy with the best ROI performance, eliminating the need to search among cost budget candidates.\n\n**2. It is fairly common in RL to approach cost/return trade-off by integrating the action costs as negative rewards.**\n\nIn our paper, ROIDICE has been compared with offline RL and offline constrained RL. As the reviewer has pointed out, the return/cost trade-off can be optimized with a regularized RL framework with its reward $r(s,a)-\\lambda c(s,a)$. However, we believe that constrained RL is a better baseline for ROIDICE when considering practical usages.\n    \nTheoretically, there always exists a regularized RL with a certain $\\lambda$ that provides the same solution as constrained RL for a given cost budget. In some constrained RL algorithms using Lagrangian, $\\lambda$ is implicitly optimized to satisfy the cost constraint[1].\nHowever, finding an appropriate $\\lambda$ to compare regularized RL methods with ROIDICE can be difficult, particularly when the scale of reward and cost varies significantly. In contrast, constrained RL is more practical, as a suitable cost budget can be inferred from the dataset, alleviating the need for an explicit search for $\\lambda$.\n\n**3. How stable are the policies optimized in problems where the returns and possible costs may have very different dimensions and distributions?**\n\nAs mentioned in the Limitations section, ROIDICE can be influenced by the design of reward and cost functions. The variation of reward and cost across state-action pairs significantly impacts ROIDICE as it maximizes the ratio of return to accumulated cost. For instance, as noted in Section 3, when the cost is constant across all states and actions, the ROIDICE policy becomes equivalent to the standard offline RL policy. Conversely, as the distributions of cost become more diverse, the ROIDICE policy deviates from the offline RL policy.\n\nOn the other hand, analogous to typical RL algorithms not being affected by the scales of reward function, ROIDICE is basically invariant to different scales of rewards and costs apart from the choice of appropriate $\\alpha$. We also empirically observed that ROIDICE resulted in a same level of performance when $\\alpha$ is appropriately scaled to align with the cost scales.\n\n**4. Could you provide a better rationale on why ROI would be a good objective for RL in general, beyond its use in the business domain?**\n\nWe can come up with a number of practical scenarios where ROI would be a good objective, such as optimizing the mileage (km/L) of an autonomous vehicle, the (decision optimality/planning time) trade-off when adopting a meta controller for real-time decision making, or the speed-accuracy tradeoff (generation quality/inference time) of a large language model (LLM). We illustrate a comparison between two approaches for optimizing the mileage problem.\n\n- Constrained RL: An agent trained with constrained RL uses a limited budget to maximize the reward. Setting an appropriate budget that is feasible for the problem may require domain knowledge. Therefore, to compare the ROI performance of its policy, it is necessary to conduct multiple runs of constrained RL with different levels of constraints (e.g., fuel consumption $\\leq$ [5, 10, 15, 20]). The policy with the best mileage is then selected from these multiple runs of constrained RL.\n- ROI maximization: A single run of ROIDICE is sufficient to find a policy that maximizes the vehicle's mileage.\n\nIdeally, the mileage (ROI) performance of the ROIDICE policy would be equal to or greater than that of the constrained RL policy. Equality occurs when the upper bound of the constraint matches the fuel consumption of the ROIDICE policy.\n\n**5. How significant is the decrease in policy performance compared to cost-free policies in practice for?**\n\nThe remark in Section 3 notes that the optimal policies of standard RL and ROI maximization are equivalent when the cost is constant across all states and actions. This gap increases when the cost varies among states and actions or when there is an action with high cost that leads to high return. The difference between standard RL and ROIDICE can be managed by adjusting the cost distribution.\n\n**6. Does the proposed approach provide some guarantees on cost?**\n\nCurrent form of ROIDICE does not guarantee the upper bound on cost. However, ROIDICE with the cost guarantee can be easily derived by adding the cost constraint $\\sum_{s,a}d(s,a)c(s,a)\\leq C_{\\text{threshold}}$ to the LP formulation of our algorithm, ROI-LP.\n\n\nThanks for pointing out the issue with references. We will fix this in the next revision.\n\n[1] J. Lee, et al. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. ICLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their time and effort in providing constructive feedback and insightful reviews of our paper. We are grateful that the reviewers recognized our paper for presenting a novel offline policy optimization framework that effectively optimizes the trade-off between return and accumulated cost, is mathematically well-derived, and yields policies with better efficiency across various domains.

In response to the valuable feedback from the reviewers, we address each concern individually. Below, we summarize the major points of the feedback:
- How to address the discrepancy between the offline data distribution and the online environment.
- Whether a theoretical analysis of sample complexity is available for ROI-LP.
- Comparative analysis of ROI performance across policy optimization approaches
- Clarifying the advantages of ROIDICE for return/cost trade-off optimization
- Exploring potential applications of ROIDICE beyond business domains.

Meta-review: The paper presents a constrained offline policy optimization algorithm that maximizes Return On Investment (ROI) and formulates it as a linear programming problem. The experiments on various offline RL tasks show that it achieves a better trade-off between return and accumulated cost. 

The paper initially received borderline reviews. While the reviewers considered the work novel (Z94y, T1wG, Yyxa) and the proposed framework well-formulated (8zrU, T1wG, Yyxa), they also raised several concerns on the paper: 1) missing computational complexity analysis (Z94y) and theoretical analysis on sample complexity (8zrU) or cost guarantee (Yyxa); 2) lack of clarity on several technical issues, including coping with online setting (Z94y) and large state space (8zrU, T1wG), reward/cost scaling and advantage of the method (Yyxa) ; 3) missing important baselines and experiments (T1wG, Yyxa); 4) limited application domains (Yyxa); and 5) insufficient analysis on the results (T1wG, Yyxa). 

The authors provided a thorough response, including additional results on computational complexity and experimental comparisons, detailed analysis of the theoretical properties and results, and clarifications on the technical issues and application domains. During the rebuttal discussion, most reviewers considered their main concerns well addressed and reached a consensus with positive ratings. After considering the paper, reviews, and rebuttal discussion, the AC does not find sufficient cause to overturn the reviewers' consensus and therefore recommends it for acceptance. The author should take into account the rebuttal and the reviewers' feedback, especially regarding the additional results and analysis, when preparing the final version.
Decision: Accept (poster)