Robust Reinforcement Learning in Finance:  Modeling Market Impact with Elliptic Uncertainty Sets
Authors: Shaocong Ma
Heng Huang
Venue: NeurIPS 2025 poster ()
Keywords: finance
robust reinforcement learning
algorithmic trading
reinforcement learning
Primary area: reinforcement_learning
TL;DR: A robust RL framework for quantitative trading with introducing novel elliptic uncertainty sets to model the market impact.

Abstract: In financial applications, reinforcement learning (RL) agents are commonly trained on historical data, where their actions do not influence prices. However, during deployment, these agents trade in live markets where their own transactions can shift asset prices, a phenomenon known as market impact. This mismatch between training and deployment environments can significantly degrade performance.  Traditional robust RL approaches address this model misspecification by optimizing the worst-case performance over a set of uncertainties, but typically rely on symmetric structures that fail to capture the directional nature of market impact. To address this issue, we develop a novel class of elliptic uncertainty sets. We establish both implicit and explicit closed-form solutions for the worst-case uncertainty under these sets, enabling efficient and tractable robust policy evaluation. Experiments on single-asset and multi-asset trading tasks demonstrate that our method achieves superior Sharpe ratio and remains robust under increasing trade volumes, offering a more faithful and scalable approach to RL in financial markets.

Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets
Shaocong Ma Department of Computer Science University of Maryland College Park, MD 20742, USA scma0908@umd, edu
Heng Huang*
Department of Computer Science
University of Maryland College Park, MD 20742, USA heng@umd... edu
Abstract
In financial applications, reinforcement learning (RL) agents are commonly trained on historical data, where their actions do not influence prices. However, during deployment, these agents trade in live markets where their own transactions can shift asset prices, phenomenon known market impact. This mismatch between training and deployment environments can significantly degrade performance. Tra- ditional robust RL approaches address this model misspecification by optimizing the worst-case performance over set of uncertainties, but typically rely on symmet- ric structures that fail to capture the directional nature market impact. To address this issue, we develop novel class of elliptic uncertainty sets. We establish both implicit and explicit closed-form solutions for the worst-case uncertainty under these sets, enabling efficient and tractable robust policy evaluation. Experiments on single-asset and multi-asset trading tasks demonstrate that our method achieves superior Sharpe ratio and remains robust under increasing trade volumes, offering a more faithful and scalable approach to RL in financial markets.
Introduction
Reinforcement learning (RL) has emerged promising decision- decision-making framework for quantitative trading strategies [1, 2]. including portfolio optimization [3-13], automatic trading [3, 14-19), market making [20-22], and option hedging [23, 24]. RL's appeal in finance lies in its ability to learn adaptive strategies directly from data, without strong market assumptions [1]. This makes well- suited for capturing complex market dynamics and aligning with the sequential nature of financial decision-making
One of the primary challenges in training a robust and consistently profitable RL agent lies in handling the market impact [25-27]; that is, the influence of the agent's own trades on asset prices in the deployed environment. For example, when trader buys or sells a large volume of an asset, it can temporarily drive the price up down, respectively (as illustrated Figure Typically, RL agents are trained on historical market data where market impact is absent. However, during deployment, the environment shifts from a passive historical setting to the real market, where the agent's actions actively affect prices. This discrepancy between training and deployment environments undermines the optimality and robustness of the learned policy, and leads us the central question in this paper:
Q: Can we train RL agents on historical data while robustly accounting for market impact during deployment?
"This work was partially supported by NSF IIS 2347592, 2348169, DBI 2405416, CCF 2348306, CNS 2347617, RISE 2536663.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Market impact Large Buy Trades
Market Impact of Large Sell Taadss
Baseliee
1000 2000
2000
4000
5000
1000
2300
3200
4022
5000
Deuaivv (mr))
Relabive
Figure 1: Market impact illustration using AMZN stock on June 21, 2012, based on S-level LOBSTER data [47]. The left panel shows the price response executing buy order of 100 shares at the time = 0 ms. which consumes ask-side liquidity within 1000 ms and induces an immediate upward shift in price. The right panel shows the analogous impact of sell order. This plot indicates that the transition dynamics induced by trading are not symmetrically distributed around the nominal kernel.
To address this central question, we adopt the framework of robust RL [28 (8844], which is designed to handle model misspecification; that is, the mismatch between the training and deployment envi- ronments. The robust RI. framework explicitly acknowledges that the real market environment may differ from the simulated training environment, and it aims to learn policies that are resilient under range of perturbations.
However, existing robust RL approaches face a key limitations in financial applications: that is, traditional uncertainty sets are typically symmetrie, which fail capture the directional nature of market impact. Addressing this challenge motivates our threefold contributions:
(1) We propose a novel class of uncertainty sets, elliptic incertainty sets (Definition 3.2), which generalize traditional f--norm balls the ellipse-like like structure. These sets better capture the empirically observed directional nature of market impact as illustrated in Figure 1.
(2) On the theoretical side, we derive closed-form solutions for solving the worst-case transition kernel under the proposed uncertainty sets (Theorem 3.4). Furthermore, under certain conditions, we present the explicit solutions (Theorem 3.5). This development significantly broadens the scope of tractable robust RL problems beyond symmetric (ball-shaped) uncertainty sets, enabling more faithful representation of market impact.
(3) We empirically evaluate our approach on real-world financial data using trade-level market impact simulations in Section 4. Experimental results demonstrate that our method consistently outperforms the standard single-asset intra-day trading strategy and existing RL baselines in terms of Sharpe ratio. Moreover, we validate the robustness of our method under increasing strategy volume, confirming effectiveness in high-volume regimes.
1.1 Related Work
Existing Approaches to Handle Market Impact The most common approach for handling market impact simulate the electronic market more accurately. By applying high-fidelity market simulators, often built upon limit order book (LOB) dynamics [25, 26, 48-50], trade-level data [51, 52], large-scale agent-based simulators [53-56], it captures more detailed market microstructure and reduces the gap between the simulated and the real trading environments. Prominent approaches incorporating market impact into backtesting include agent- based simulation frameworks (53-56), data-driven LOB reconstruction models [57, 58, 50], and hybrid systems that integrate historical data replay with synthetic order flow generation [59, 60]. However, access to high-quality market data often limited, and simulating market environments with agent-based systems remains prohibitively expensive. Therefore, practical alternative to train the agent directly on historical data without market impact while still encouraging it to account for potential worst-case scenarios.

---

Robust RL in Finance Robust RL, with its intrinsic ability to handle model misspecification, pro- vides natural framework for incorporating market impact considerations during training. Jaimungal et al. [3] propose a robust reinforcement learning framework based on rank-dependent utility to address uncertainty in financial decision-making. demonstrating the effectiveness of robust RI. in portfolio allocation, benchmark strategy optimization, and statistical arbitrage. Shi et al. [5] formulate portfolio optimization as a robust RL problem to enhance resilience. We et al. [24] extend robust risk-aware RL to manage the risks associated with path-dependent financial derivatives, showcasing its effectiveness in complex hedging scenarios. However, existing work primarily focuses on fully symmetric uncertainty sets, which fail to capture the directional characteristics of financial markets. Addressing this limitation is the central focus of our paper.
Modeling the Uncertainty Set in Robust RL The uncertainty set captures the discrepancy between training and deployment environments. However, robust RL becomes computationally intractable when the uncertainty set is highly irregular [28, 44, 39, 40]. To mitigate this issue, it is common to impose structural assumptions that enable tractable solutions. We highlight several representative structures, with further details deferred to Appendix A.I. The R-contamination model [31] defines an uncertainty set as a sphere of radius R centered around the nominal transition kernel, admitting analytical solutions for robust policy evaluation. The e--norm uncertainty sets are also widely studied due to their closed-form solutions [39, 40]. The integral probability metric (IPM) and double-sampling uncertainty sets have been shown allow efficient computation [43]. Other works include uncertainty sets based on the Wasserstein metric and f-divergence, which have also received considerable attention [61-63].
2 Preliminaries: Robust Reinforcement Learning
We focus the discounted aafnite-hooizzn Markov Decision Processes (MDPs) [64], formally defined as five-tuple (S, A, P, r, r,,)), where S and A denote the state and action spaces", respectively. The transition kernel (ss 8, a) specifies the probability of transitioning to state s' from state after taking action a. The reward function r: S - [0. 1] assigns a bounded reward to each state, and the discount factor y E (0. 1) models the agent's preference for immediate rewards over future ones.
Instead of assuming a fixed transition kernel P. we account for the effects of model misspecification. Let Po denote the nominal transition kernel, and define the (s, a) uncertainty set Us R S] the point (s, a) S x A as
((s, a) a)-Uncertainty Set) Uz.a {ua,a a
R
Ueea Cz.a: u (s) 0),
where Ca.a c |S a convex set, [| is an all-one vector with the dimension |S] (for convenience,
we will usually omit the subscript (S). The convex set is commonly chosen as a ball-shaped set (e.g.
Cs.a Bp(3,.e) := {ua,a E R Si |ss.aalp: & B3,a) for the nn-nr uncertainty set), where 9s.a > a radius parameter that quantifies the allowable deviation. The zero-sum constraint o
ensures the perturbed transition P(- s, a) + usaa remains valid probability distribution. The uncertainty set l and the robust transition model are then defined as
(Uncertainty Set)
u:=
X Us (s.a)ESxA
(1)
(Robust Transition Model)
Pu:=
P.. == Po -uuuuu u
(2)
respectively. Throughout this paper, we assume the parameter of uncertainty set chosen appro- priately such that all elements in the robust transition model is well-defined [39, 40). Given these notations place, we define the value function of the policy æ with the uncertainty u as the value
"For theoretical analysis, we restrict attention MDPs with finite state and action spaces. This assumption avoids the technical complications arising from continuous or hybrid spaces, which, although explored some prior work [65-68], remain analytically open without imposing additional assumptions, especially for robust RL problems. Nevertheless. our empirical results extend to continuous settings. demonstrating the practical applicability of our approach beyond the theoretical scope.
3

---

Market Impact
Upward Shift
Downward Shift
Ug [0,0,0.0,0]
U1 [,, 2,, 0.0]
U2 -0,0,-2a,,,40
Figure 2: Illustration of the transition kernel in simplified robust RL setting using an &y -orm uncertainty set. Only one of the upward (buy) or downward (sell) shift is plausible in given scenario; however, the ball-shaped uncertainty set must include both shifts due to its symmetric structure, which motivates to propose uncertainty set with non-symmetric structures.
function with the transition probability P. P:
(s) == E Σ (r)()),))) so= = s, T.
The robust value function is the worst-case value function over all uncertainties; that is (T(s) := min (s). Similarly, we can also define the robust Q-function and the robust advantage function
as (*( a) min,, E D 2 7r((st, (s),,, respectively.
80
8, n0 a, Pn, π. and A A"(s, a) Q" (8.aa
The goal of robust RL is learn a parameterized policy we that maximizes the worst-case value function *T (so). where 8o denotes the initial state. A standard approach applies the robust policy gradient formula [28]:
1 (s) Ee-d" [Q" (s. V log [e(a|s)]
where T is the stationary distribution induced by we, and QTe denotes the robust Q-function. This formulation reduces robust RL to a gradient- optimization problem, shifting the main challenge of robust RL to accurately evaluate the robust value function, which is our focus in Section 3.3.
3 Modeling Market Impact with Elliptic Uncertainty Sets
3.1 Limitations of Symmetric Uncertainty Sets
Robust MDPs offer an ideal framework handle model misspecification by allowing the transition kernel to deviate within prescribed uncertainty set. However, most existing formulations adopt symmetric structures, typically the ball defined by a specific norm, that treat all directions of perturbation equally. While mathematically convenient, these symmetric structures often fail to reflect the directional nature of real-world uncertainties.
Symmetry in this context typically refers to invariance under the signed permutation group (see Appendix A.2 for details). A canonical example is the fy-norm ball:
B3) = {u IRd
||w||p < $3,
which satisfies the property that for any u (3), all signed permutations of u are also contained in the set. enforces implicit assumption of isotropic uncertainty, equally plausible in all directions, which often includes unrealistic perturbations. We demonstrate in the following example:
Example 3.1 (Symmetric Sets Fail to Capture Directional Uncertainty). In financial markets, large buy sell orders induce directional shifts in asset prices due to liquidity consumption (see Figure 1). In Figure 2, we consider the following classical -orm (s, a) uncceratayy set:
Us {ue gd |u|co & 2s, u² 0).

---

note that it is unavoidable that the element Us may be This set includes. for example, the perturbation vectors
U1 = [,, - 28, 00, and U2 - [0,0, -0,,-2,,,,0 E,,e,
Both i1 and li satisfy the norm and mean constraints, and since they signed permutations of each other, they must either both belong to Us or be excluded together. However, this symmetry fails reflect market realities: under buy action, Wi represents a plausible upward shift due to liquidity-driven market impact, while U2, corresponding to a downward shift, is implausible. Thus, the symmetric structure forces inclusion of perturbations that contradict the directional market impact, potentially leading to overly conservative or unrealistic robust policies.
This observation underscores the importance of developing a robust RL framework that can capture the directional nature of environment shifts observed in financial markets. To this end, we introduce a novel class of elliptic uncertainty sets, which generalize traditional norm-bounded sets by allowing non-symmetric perturbations, while retaining closed-form tractability under certain conditions.
On the Conservativeness of Robust Policies Although our proposed elliptic uncertainty sets better capture the directional nature of market impact, this alone does not immediately clarify why they improve robustness. To illustrate the underlying intuition, present simple example:
The robust value function V := minnet V models the worst-case discounted future return. If the uncertainty set U is larger, it is conservative, it yields smaller value.
In the ideal case, the uncertainty set consists of single element uo that exactly characterizes the MDP induced by the true market impact. Training an RL agent on the robust MDP with = uo is then equivalent to training directly on the real LOB data.
In the less ideal case where U/ includes additional elements, e.g. 24 {wo, 3. The robust value function may still achieve its minimum at uh (uo arg min, min Vg). In this scenario, the robust formulation reduces to the ideal case. Otherwise, if the minimum is attained at some u * uo, the robust value becomes strictly smaller than V making the policy more conservative.
Unfortunately, neither our approach nor standard robust RI. methods can theoretically rule out this latter "overly conservative" scenario. Our guiding intuition, however, is that smaller uncertainty sets are less likely to contain such overly conservative elements. By trimming down the traditional uncertainty sets, our method reduces the risk of unnecessary conservatism, though it does not eliminate it entirely.
3.2 The Elliptic Uncertainty Sets
The elliptic uncertainly sets generalize the classical f--norm uncertainty by incorporating direc- tional non-symmetry. Formally, we have the following definition:
Definition 3.2 (Elliptic (s, a) -)-uncertannty Set). For each state-action pair (s, a) E SSx x A, the elliptic (s, a) o-uunerrtayyty set is defined as
N Σ n=1
1 x Be,a u u111s) ||s| = =
Us.a = u HERRI
1
(3)
where [us.a).N Jn=1 E IR are called the foci of the ellipse, 32.0 > 0 is the uncertainty size, and ||. IRd - an arbitrary norm. Particularly, when || || is taken as the fy-norm (p [o,+ Us is called the ey-ellipse uncertainty set.
Here the constraint a uT 1|st ensures the perturbed transition still defines valid probability distribution. For convenience, we will omit the superscript in u² and the subscript in 8x,a when the context clear. Importantly, valid probability distribution when some entries of u + Po[-|s, a) are negative, To avoid this scenario, we include the following regular condition, which is also presented in the f--norm uncertainty set literature [39, 40]:
7) for any set of foci {usaa] (u=a))=11 Jnzp there exists constant such that for all Bxa < 3, Eq. (3) induces valid probubility transitions; that is, all entries of + Po(-]s (-1ss, a) are non-negative
Assumption 3.3. For the given MDP
5

---

Connection to the Classical Ellipse Our definition draws directly from the geometric characteriza- tion of an ellipse: the set of points for which the sum of distances to multiple foci is bounded by constant 8s,a- We show that it recovers classical structures as special cases with the following two concrete examples:
(1) When = and #1 = O, the set Eq. (3) reduces to the f--norm ball
(3( == {u ||ll>> < 8),
which aligns with the standard uncertainty set used robust RL (c.g., [39, 40, 28]).
(2) When N 2 and 2, Eq. (3) becomes ||u ||2 + ||u 22/la B. Defining the midpoint
We #22 there exists a matrix such that this constraint is equivalent to a classical quadratic 2
form of an ellipse:
(u a) A(u T) ū) =
where the detailed derivation is given in Lemma B.10.
These examples demonstrate that our formulation significantly generalizes classical ellipses by allowing arbitrary norms and accommodating multiple foci, thereby enabling more flexible modeling of non-symmetric perturbations. However, such generality often comes the cost of increased complexity. To address this concern, in the next section, we show that, under certain parameter choices, our proposed uncertainty remains as trackable traditional e--norm uncertainty sets.
Limitations & Potential Directions As our approach is mainly adopted from the f--norm un- certainty set, it does not guarantee that the resulting distributions absolutely continuous with respect to the nominal transition distribution even with Assumption 3.3. One potential solution is incorporating f-divergence or distributionally robust RL techniques [69-74].
3.3 Solving the Worst-Case Uncertainty
Solving the worst-case uncertainty u* := arg min V (sa) plays crucial role in efficient robust policy evaluation. The f--norm uncertainty set [39, 40] and the I--contamination model [31] are popular the solution u* is closed-form. When the uncertainty set complicated, many existing robust RL methods require an external optimization loop to determine the worst-case transition probability, which can be impractical in real-world scenarios.
To address this issue, we derive an implicit solution (Theorem 3.4) for the worst-case uncertainty that avoids additional interaction with the environment. Under certain conditions, further provide an explicit closed-form solution (Theorem 3.5), enabling direct computation without the need for external solvers.
We start with recapping some backgrounds in robust TD lcarning. Let V denote all functions mapping from the state space S to the Euclidean space I. Given that t/ is arbitrary uncertainty set, the robust Bellman operator associated with the policy n, Tíi V - V, is defined as
v()
Σ a(a|s) aEA
r(s.a) + min uT u + 7 Σ Pol6 uell,,a
As shown by [75. 28], when 7 E (0, 1), the robust Bellman operator is contraction operator, which admits the unique fixed point as the robust value function V E V. When If is given by the e--eliipse uncertainty set (Eq. (3)), then
v() = Σ T(als)
r(s.a) +y
min |
TT uu+ Σ P((('(s,,))((s)
aEA
aES
turns that if can solve the optimization problem
min
TT U,
Σ. 2
1-0
the robust Bellman operator just the standard Bellman operator (over the nominal transition probability) with a solved shift. As the result, we can simply apply the standard TD-learning with adding this correction term solve the desired robust value function. In the following theorem, we present general recipe of solving this optimization problem.

---

Theorem 3.4 (Implicit). d= [S] be the cardinal of state space. Suppose that {un (unin-) C R for each (8, a) E x A. the uncertainty size B že O, and u E TR.. Then there exists X and μ²x such that the optimization problem defined by Eq. (4) is solved by
N Σ uuu unllp]
= arg min (v #
1) u
Remark. We say a solution of the optimization problem is "implicit" if can be represented as equation the form g(u, v, B, (un)) = O. In this theorem, as the right-hand side
G(u, u, B, (un)) (o + a 1)
Σ
lu unllp
n=
proper, convex, and coercive; we can surely re- e-write as the sub- uub-gradient form by letting
g(u, v, B, {un}} E O.,G(u, 3, (un)).
Then obtain the implicit representation g( u. B, (2, }) 0. Moreover, derive the formula of X and j beyond the existence; the full result is presented in Theorem B.12 with more details.
The implicit solution has already shown significant advances compared to some of existing robust RL methods which typically require to solve the worst-case transition probability using additional state-action sample generated from the agent-envvrrommet iteration. However, it is still (slightly) impractical to solve additional convex optimization problems in every iteration. Fortunately, under certain conditions, the solution can be "explicit"; that is, we can write the optimal solution in the form of u f(v, B, {un)).
Theorem 3.5 (Explicit). Letd [S| be the cardinal of state space. Suppose that {un Rd for each (s, a) S A, the uncertainty size 3 3 O, and u E Rd. The optimization problem defined by Eq. (4) is explicitly solved in the following cases:
(a) Let N = 1 and
1 + 1 = 1. Then the minimizer of Eq. (4) is given by p
9
a 21 sign(r 1 1) O n 1 ||v +
Here sign and 1 are coordinate wise sign function and absolute value, respectively; + the coordinate-wise product; and μ** := arg min mER
IV p1
(b) Let p and N 2. Suppose that Iu2 un ||, Then the minimizer of Eq. (4) is given by
11 + U2 B |uu u2
u
[sign(v + 0110)[(1===1122)))
2
Here sign, 1. and I are coordinate-wise sign function, absolute value, and indicator function, respectively; O is the coordinate-win product;
n max; Uj + min, U.
and x* x= max Uj mimy Uj
4
(c) Let p and N 2. Suppose that B > 112 w1/12 2/2
Then the minimizer of Eq. (4) is given by
u* U1 + U2
v 32
U22
u1
1
52 10 + 11)
2
2
Here Q :0==1
ge (ug - U1 (uz - uz)T;
21 1TQ-11
1 (u 1x TΩ 1 1=1), and
Remark. This result presents a clean form of the worst-case uncertainty u under certain conditions. Unlike the implicit case where it takes an additional root-finding algorithm to solve an approximated u*, in the explicit case, if the current value function V is given and the parameters of the uncertainty set (u, and (3) have been determined, the uncertainty u can explicitly solved. The full proof is presented in Theorem 131
7

---

3.4 Robust TD Learning Algorithm
Given Theorem 3.4 and Theorem 3.5 in place, we immediately obtain the robust TD learning algorithm for robust policy evaluation. Given the current value function we can calculate u" using the implicit and the explicit formula; assume the current state-action pair is given as (s, a, s'), then the updated value function is given by
('(s) v(8) + (((s + (Po(s'(ss + (²(())) [e())) + ru(s)] v(8))
(5)
We can further simplify this update rule by using an unbiased estimator of that: (a) v(s) s7y0" Tyyo"" u²* 1y (r(s, a) + +vv(s') - v(8)),
(6)
where o' Fo(-|s, a). The formulation leads us to Algorithm
Algorithm/ 1: Robust Policy Evaluation
Input: The target policy æ, the foci (p:)/= < Rd, and (3a,a) the uncertainty size
Sample the initial state 80 from the initial distribution; Initialize the value function oo:
for t = 0, 2......... T -- -1
Sample the action a₃ T(([s;); Transition from st to sc+1 Po(-Is;,
Calculate u using [p:) N (Ba,a)- ). and Us;
Robust TD-leaming: (s) v(s) myr" u (r((,,)) 70(8') u(s)):
end
Output: The final value function UT
The convergence of this algorithm, as well as its corresponding Acttr-Critic-style policy gradient algorithm, follows directly by applying the standard proof routine from the robust RL literature (e.g. [43]). For completeness, we include the convergence result and full proof in the supplementary material.
Remark.
4 Experiments
To validate our theoretical findings and demonstrate the practical effectiveness of robust RI. framework in the environment with the market impact, we conduct experiments on two different tasks that are closely tied to market impact: (1) minute-level single-asset strategy, and (2) large-volume portfolio rebalancing. In minute-level trading, even small trade sizes can noticeably move prices, leading to slippage. Similarly, large-scale portfolio rebalancing, often performed by large financial institutions, can significantly affect asset prices due to the large order volumes involved.
4.1 Performance Comparison on Single-Asset Intra-Day Trading
We start with the single-asset minute-level trading. The non-RL baseline is chosen as the momentum strategy [76], which designed based on the empirical observation where assets that have performed well in the recent past are more likely to continue performing well in the near future.
Training and Evaluation of RL Agents We implement a Gym-like fyme RL environment [77, 78] constructed on historical data, with full environment details provided in Appendix C.3. All RL agents are trained on one year of earlier historical data (from May 9th, 2021 to May 9th, 2022 as the nominal environment) without accounting for market impact. Their performances are then evaluated over the period from June 9 to December 9, 2022, with the market impact included. To simulate market impact, reconstruct LOB dynamics using short period of real trading orders and determine the execution price via the volume- weightee average price (VWAP). A simple example illustrating this estimation process is shown in Table 2, Appendix C.
Results As shown Table 1, our proposed method consistently outperforms the momentum strategy, the non-robust RL, and the symmetric robust RL baselines (based on &, no-mr balls) in terms of the

---

Table I: Performance comparison of different RL agents on selected assets under the simulated market impact from June 9 to December 9, 2022. Robust RL with E e--elippee uncertainty set consistently achieves the highest Sharpe ratio.
Asset
Method
Final Value ($)
Annualized Return (%)
Sharpe Ratio
Max Drawdown (%)
Momentum
96 334
-7.3%
-0.95
-4.2%
Non-Robust
120347
44.8%
1.74
11.3%
META
Robust RI. (fg-Ball)
97
-5.7%
-0.28
-3.4%
Robust RL (fg (ly-Blipsee
138011
90.8%
2.48
9..%%
Momentum
105 163
10.6%
1.10
5.3%
Non-Robuse Robust RL
87440
233.5%
-1.75
16.9%
MSFT
Robust RL (fg-Ball)
92 159
-15.1%
-0.82
-11.%%
Robust RI. (f,-FEliipe
111 485
24.4%
1.20
-0.1%
Momentum
107 333
15.2%
1.69
33.2%
Non-Robust
91 947
-15.5%
-1.64
11.9%
SPY
Robust RI. (fg-Ball)
100560
1.1%
0.17
-6.4%
Robust RL (,. (l.BBlipse)
179-722
19.4%
1.60
-5.8%
140000
140000
Non-Fobuu ft.
Mo
Ritved F2. (f,4xx)
Finel Portfoli Value
130000
120000
Rubost (f-E
Portfat Valle
120000
100000
110000
80000
50000
Momentum
Non Rabust
Rotual Rt. (fy Baa))
Rebuat RL (f El-Eippo))
2022-05-08
2022-47-15
2022-38-21
2012-09-20
2022-11-02
2022-12-99
Figure 3: Performance comparison of trading strategies the META stock from June 9 December 9, 2022, under simulated market impact. The left panel compares the final portfolio values with (in red) and without market impact (in green). illustrating the robustness of each method to execution- related slippage. The right panel shows the cumulative returns over the evaluation period, tracking the performance of the four strategies in Table 1, alongside the baseline performance of the META stock.
risk-adjusted return (Sharpe Ratio). These experiments validate the following key understandings: (i) While robust RL with symmetric uncertainty sets significantly mitigates the effects of market impact (as illustrated in the left panel of Figure 3), it often produces overly conservative strategies that compromise profitability by taking implausible perturbations into consideration; (ii) the non- robust RL usually suffers greater risk exposure, resulting in the highest Max Drawdown among all methods; (iii) in contrast, the proposed fp-ellipse uncertainty set effectively captures the directional non-symmetry of market impact, allowing the agent to achieve more favorable trade-off between robustness and return.
4.2 Robustness to the Market Impact Scaling the Trading Volume
In this subsection, show that policy trained in a low-volume environment continues to mitigate market impact when transferred to portfolios with significantly larger volumes. consider a multi-asset portfolio allocation task, modeling the realistic setting where large volumes are traded over short periods to maintain low-variance portfolio. The same Gym-like RL environment and evaluation period from the previous experiment are used. We evaluate the robustness the market impact using the relative portfolio gap, the normalized absolute difference in final portfolio value with and without market impact:
Port. Value with - -I-Moro
Value without MI
Relative Port. Gap
Initial Cash
where MI represents the market impact. Additional experimental details provided in Appendix C.

---

Indanl Coech 5100/0000
31,000000
52.0000
Gopp vs. Cash
38
V108 Can
Rize Moo-ooooulo
Sa Xu-Cllie
Figure 4: Robustness of RL agents market impact under increasing trading volumes. The left three panels show normalized portfolio values over time across initial cash levels, with dashed and solid lines indicating performance with and without market impact, respectively. The right panel shows the relative portfolio gap, which increases sharply for the non-robust agent but remains small and stable for the robust RL agent with e--eliiptic uncertainty sets.
Results As shown in Figure 4, the robust RL agent with f--ellipse uncertainty set consistently outperforms the non-robust RL method both in return and in mitigating the effects of market impact. While the performance gap is small at low volume, the non-robust agent degrades rapidly as volume increases, suffering from instability and larger drawdowns. In contrast, the robust agent remains stable and profitable even at high volume (~ 200 M), demonstrating strong scalability.
5 Conclusion & Broader Impact
This paper focuses on the market impact appearing in quantitative trading, where an agent's 's actions affect prices. By modeling the training environment as the nominal transition kernel, the proposed novel f,-ellipse uncertainty sets better captures the non-symmetric nature of price responses compared to traditional symmetric sets. We established the theoretical tractability of this approach by deriving implicit and explicit closed-form solutions for robust policy evaluation within this framework, enabling efficient robust TD-learning algorithms that account for the market impact during training on the nominal historical environment. Experiments on real historical data demonstrated that our method significantly improves robustness and iijksadjsteed returns over non-RL, non-robust RL, and symmetric robust RL baselines. This work broadens the applicability of tractable robust RL and offers a more faithful modeling approach for market impact. The broader impact involves potentially more stable and profitable automated trading strategies.
References
[1] Ben Hambly, Renyuan Xu, and Huining Yang. Recent advances in reinforcement learning in finance. Mathematical Finance, 33(3):437-33) 2023.
[2] Nikolaos Pippas. Cagatay Turkay, and Elliot A Ludvig. The evolution of reinforcement learning in quantitative finance. arXiv preprint arXiv:2408. 10932, 2024.
[3] Schastian Jaimungal, Silvana M Pesenti, Sheng Wang, and Hariom Tatsat, Robust risk-aware reinforce- ment lcarning. SIAM Journal on Financial Mathematics, 13(1):213-226, 2022.
[4] Pengqian Yu, Joon Sem Lec, Ilya Kulyatin, Zekun Shi, and Sakyasingha Dasgupta. Model-based deep reinforcement learning for dynamic portfolio optimization. arXiv preprint arXiv: UrXiv:/901.08444 2019.
[5] Xiaochuan Shi, Yihua Zhou, Lei Wu. Robust reinforcement learning for portfolio management via competition and cooperation strategies, 2023. ICLR 2024 Conference Withdrawn Submission.
[6] Philip Ndikum and Serge Ndikum. Advancing investment frontiers: Industry- deep reinforcement learning for portfolio optimization. arXiv preprint arXiv:2403.07916, 2024.
[7] Carlos Betancourt Wen-Hui Chen. Deep reinforcement learning portfolio management markets with a dynamic number of assets. Expert Systems with Applications, 164:1 14002, 2021.
[8] Amine Mohamed Aboussalah and Chi Chi-Guhn Lee. Continuous control with stacked deep dynamic recurrent reinforcement learning for portfolio optimization. Expert Systems with Applications, 140: 12891, 2020.
10

---

[9] Min-Yuh Day. Ching-Ying Yang, and Yensen Ni. Portfolio dynamic trading strategies using deep rein- forcement learning. Soft Computing, 2815))8773-87330 2024.
[10] Yuh-Jong Hu and Shang-Jen Lin. Deep reinforemmett learning for optimizing finance portfolio management. In 2019 amity international conference on artificial intelligence (AICAI), pages 14-20. TEEF, 2019.
[11] Angelos Filos. Reinforeement learning for portfolio management. arXiv preprint arXiv: 1909.0957, 2019.
[12] WARAMETH NUIPIAN, PHAYUNG MEESAD, et al. Dynamic Portfolio Management with Deep Reinforcement Learning. PhD thesis, King Mongkut's University of Technology North Bangkok, 2025.
[13] Farzan Soleymani and Eric Paquet. Financial portfolio optimization with online deep reinforcement learning and restricted stacked deteeeccnder-ddebbrahhh Expert Systems with Applications, 156: 13456, 2020.
[14] Hyummin Cho and Hyun Joon Shin. Trading strategies using reinforcement learning. Journal of the Korea Acadtmia-Induutrial
Industrial cooperation Society, 22(1): (2())1133-33,, 2021.
[15] John Moody and Matthew Saffell. Learning to trade via direct reinforcement. IEEE transactions on neural Networks, 12(4):875 889, 2001.
[16] Yang Li, Wanshan Zheng, and Zibin Zheng. Deep robust reinforcement learning for practical algorithmic trading. IEEE Access, 7:108014-108022, 2019.
[17] Ji- Heon Park, Jae-Hwan Kim, and Jun-Ho Ho Huh. Deep reinforcement learning robots for algorithmic trading: Considering stock market conditions and us interest rates. IEEE Access, 12:20705-20725, 2024.
[18] Yasmeen Ansari, Sadaf Yasmin, Sheneela Naz, Hira Zaffar, Zeeshan Ali, Jihoon Moon, and Seungmin Rho. A deep reinforcement learning- based decision support system for automated stock market trading. IEEE Access, 10: 10:127769- 127501, 2022.
[19] Xing Wu, Haolei Chen. Jianjia Wang, Luigi Troiano. Vincenzo Loia. and Hamido Fujita. Adaptive stock trading strategies with deep reinforcement learning methods. Information Sciences, 538:142-1111 142-158, 2020.
[20] Olivier Guéant, Charles-Albert Lehalle, and Joaquin Fermandez-Tapia Dealing with the inventory risk: solution to the market making problem. Mathematics and Financial Economics, 7(4::477-507, 2013.
[21] Joel Hasbrouck. Empirical Market Microstructure: The Institutionns Economics, and Econometrics of Securities Trading. Oxford University Press, Oxford, UK, 2007.
[22] Zihao Zhang. Stefan Zohren, and Stephen Roberts. Deep reinforcement learning for trading. arXiv preprint arXiv:1911.10107 2019.
[23] Hans Buehler, Lukas Gonon, Josef Teichmann, and Ben Wood. Deep hedging. Quantitative Finance. 1978)1127- 2019.
[24] David Wu Sebastian Jaimungal. Robust risk-aware option hedging. Applied Mathematical Finance, 30(3):153-174, 2023.
[25] Robert Almgren and Neil Chriss. Optimal execution of portfolio transactions. Journal of Risk, 3:5-40. 2001.
[26] Anna A Obizhaeva and Jiang Wang. Optimal trading strategy and supply/demand dynamics. Journal of Financial markets, 16(1):1-32. 2013.
[27] Benee Tóth, Zoltán Eisler, and J-P Bouchaud. The square-root impace law also holds for option markets. Wilmott, 2016(85):7--33, 2016.
[28] Yan Li, Guanghui Lan, and Tuo Zhao. First-order policy optimization for robust markov decision process. arXiv preprint arXiv:2009, 105799 2022.
[29] Yike Li, Yunzhe Tian, Endong Tong, Wenjia Niu, and Jiqiang Liu. Robust reinforcement learning via progressive task sequence. In IJCAI, pages 455 463, 2023.
[30] Guanlin Liu, Zhihan Zhou, Han Liu, and Lifeng Lai. Efficient action robust reinforcement learning with probabilistic policy execution uncertainty. Transactions on Machine Learning Reseurch, 2024.
[31] Yue Wang and Shaofeng Zou. Online robust reinforvement learning with model uncertainty. In Advances in Neural Information Processing Systems, volume 34, 2021.

---

[32] Shangding Gu, Laixi Shi, Muning Wen, Ming Jin, Eric Mazumdar, Yuejie Chi, Adam Wierman, and Costas Spanos. Robust gymnasium: A unified modular benchmark for robust reinforcement learning. In International Conference on Learning Representations. 2025.
[33] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Cupta. Robust adversarial reinforcement learning. arXiv preprint M.Xivv1003.020000 2017.
[34] Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. In Advances in Neural Information Processing Systems, volume 35, 2022.
[35] Minghong Fang, Xilong Wang, and Neil Zhenqiang Gong. Provably robust federated reinforcement learning. arXiv preprint 250000220000..3.... 2025.
[36] Wei Shen, Xiaoying Zhang, Yuanshun Yao, Rui Zheng, Hongyi Guo, and Yang Liu. Robust rihf with noisy rewards. In International Conference on Learning Representations, 2025. Withdrawn submission.
[37] Shaocong Ma, Ziyi Chen, Shaofeng Zou, and Yi Zhou. Decentralized robust v-learning for solving markov games with model uncertainty. Journal of Machine Learning Research, 24(371):1-40, 2023.
[38] Pierre Clavier, Laixi Shi, Erwan Le Pennec, Eric Mazumdar, Adam Wierman, and Matthieu Geist. Near- optimal distributionally robust reinforcement learning with general I_p norms. Advances in Neural Information Processing Systems, 37:1750-1181 2024.
[39] Navdeep Kumar, Kfir Levy, Kaixin Wang, and Shie Mannor. An efficient solution to s-rectangular robust markov decision processes. arXiv preprint arXiv:2301. 13642, 2023.
[40] Navdeep Kumar, Esther Derman, Matthicu Geist, Kfir Y Levy, and Shie Mannor. Policy gradient for rectangular robust markov decision processes. Advances in Neural Information Processing Systems, 36:59477-59501. 2023.
[41] Runyu Zhang, Yang Hu, and Na Li. Soft robust MDPs and risk-sensitive MDPs: Equivalence, policy gradient, and sample complexity. In The Twelfth International Conference on Learning Representations, 2024.
[42] Zifan Wu, Chao Yu, Chen Chen, Jianye Hao, and Hankz Hankui Zhuo Plan to predict: Learning an uncereainty-foreseeinng model for model-based reinforcement learning. Advances in Neural Information Processing Systems, 15849-1115 2022.
[43] Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, PR Kumar, and Chao Tian. Natural actor-critic for robust reinforcement learning with function approximation. Advances in nearal information processing systems, 36:97-133, 2023.
[44] Qiuhao Wang, Chin Pang Ho, and Marek Petrik. Policy gradient robust mdps with global convergence guarantee. In International Conference on Machine Learning, pages 357977. PMLR, 2023.
[45] Zhongchang Sun, Sihong He, Fei Miao, and Shaofeng Zou, Policy optimization for robust average reward mdps. Advances in Neural Information Processing Systems, 37:17348-17372, 2024.
[46] Shaocong Ma, Ziyi Chen, Yi Zhou, and Heng Huang. Rectified robust policy optimization for model- uncertain constrained reinforcement learning without strong duality. Transactions on Machine Learning Research, 2025.
[47] Ruihong Huang and Tomas Polak. Lobster: Limit order book reconstruction system. Available at SSRN 1977207, 2011.
[48] Jean- Phanlppilippe Bouchaud, Julien Kockelkoren, and Zoltan Eisler. The price impact of order book events: Market orders, limit orders and cancellations. Quantitative Finance, 93):283-2977 2009.
[49] Jim Gatheral, Alexander Schied, and Aleksey Slynko. Transient linear price impact and fredholm integral equations. Mathematical Finance, 22(3):44-4744 2012.
[50] Leonardo Berti, Bardh Prenkaj, and Paola Velardi. Trades: Generating realistic market simulations with diffusion models. arXiv preprint 2700022502.07077, 2025.
[51] Robert Almgren, Chee Thum, Emmanuel Hauptmann, and Hong Li. Direct estimation of equity market impact. Risk, 18(7):58-6) 02,88-6,,770 2005.
[52] Anastasia Bugaenko. Empirical study of market impact conditional on order- flow imbalance. arXiv preprint arXiv:2004.08244 2020.
12

---

[53] Michaël Karpe, Jin Fang, Zhongyao Ma, and Chen Wang. Multi-agent reinforcement learning in a realistic limit order book market simulation. In Proceedings of the first ACM international conference on Al in finance, pages 1-7, 2020.
[54] Andrew Todd, Peter Beling, William Scherer, and Steve Y. Yang. Agent-based financial markets: A review of the methodology and domain. In 2016 JEEE Symposiun Series on Computational Intelligence (SSCT), pages 1-5, IEEE, 2016.
[55] Shen Gao, Yuntao Wen, Minghang Zhu, Jianing Wei, Yuhan Cheng, Qunzi Zhang, and Shuo Shang. Simulating financial market via large language model based agents. arXiv preprint arXiv:2406. 19966. 2024.
[56] Junjie Li, Yang Liu, Weiqing Liu, Shikai Fang, Lewen Wang, Chang Xu, and Jiang Bian. Mars: a financial market simulation engine powered by generative foundation model. In The Thirteenth International Conference on Learning Representations, 2025.
[57] Martin D. Gould, Mason A. Porter, Stacy Williams, Mark McDonald, Daniel J. Fenn, and Sam D. Howison. Limit order books. Quantitative Finance, 13(1 1):1 1740,, 2013.
[58] Avraam Tsantekidis, Nikolaos Passalis, Anastasios Tefas, Juho Kanniainen, Moncef Gabbouj, and Alexan- dros Iosifidis. Forecasting stock prices from limit order book using convolutional neural networks. In 2017 IEEE 19th Conference on Business Informatics (CBI), volume 1, pages 7-12. IEEE, 2017.
[59] Vincent Ragel. Reinforcement Learning, for systematic market making strategies. PhD thesis, Université Paris-Saclay, 2024.
[60] Zhenglong Li, Vincent Tam, and Kwan L. Yeung. Developing a multi-agent and self-adaptive framework with deep reinforcement learning for dynamic portfolio risk management. arXiv preprint arXiv:2402.005115 2024.
[61] Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint
2019.
[62] Yufei Kuang, Miao Lu, Jie Wang, Qi Zhou, Bin Li, and Houqiang Li. Learning robust policy against disturbance in transition dynamies via state-conservative policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36-7, pages 7247-7254, 2022.
[63] John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):137--1406, 2021.
[64] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[65] Yanwei Jia and Xun Yu Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research, 23(275):1-50, 2022.
[66] Huaqing Xiong, Tengyu Xu, Lin Zhao, Yingbin Liang, and Wei Zhang. Deterministic policy gradient: Convergence analysis. In Uncertainty in Artificial Intelligence, pages 2159-2169. PMLR, 2022.
[67] Kaiqing Zhang, Zhuoran Yang, and Tamer Basar. Networked multi-agent reinforcement learning in continuous spaces. In 2018 IEFF conference decision and control (CDC), pages 2771-2776. IEFF, 2018.
[68] Bin Hu, Kaiqing Zhang, Na Li, Mehran Mesbahi, Maryam Fazel, and Tamer Başar. Toward a thcoretical foundation policy optimization for learning control policies. Annual Review of Control, Robotics, and Autonomous Systems, 6 6()):123-158, 2023.
[69] Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. arXiv preprint arxiv: 08708, 2019.
[70] Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. Advances in Neural Information Processing Systems, 36:66845-6666 66859, 2023.
[71] Yiting He, Zhishuai Liu, Weixin Wang, and Pan Xu. Sample complexity of distributionally robust off- dynamics reinforcement learning with online interaction. In forty-second International Conference on Machine Learning, 2025.
13

---

[72] Yan Li and Alexander Shapiro. Rectangularity and duality of distributionally robust markov decision processes, arXiv preprini arXiv:2308. 1/139, 2023.
[73] Zhishuai Liu and Pan Xu. Distributionally robust off-dynamics reinforcement learning: Provable efficiency with linear function approximation. In International Conference Artificial Intelligence and Statistics. pages 2719-27227 PMLR. 2024.
[74] Miao Lu, Han Zhong, Tong Zhang, and Jose Blanchet. Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithms, Advances Neural Information Processing Systems, 37:1 12580, 2024.
[75] Wolfram Wiesemann, Daniel Kuhn, Berç Rustem. Robust markov decision processes. Mathematics of Operations Research, 38( 15:355-183, 2013.
[76] Carlo Zarattini, Andrew Aziz, and Andrea Barbon. Beat market: An effective intraday momentum Strategy for s&p500 etf (spy). Available at SSRN 4824172, 2024.
[77] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wajciech Zaremba. Openai gym. arXiv preprint 1rXiv:1606.01544 2016.
[78] Mark Towers, Ariel Kwiatkowski, Jordan Terry John Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning eavironments. arXiv preprint arXiv:2407.17032 2024.
[79] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[80] Stephen Roman. Advanced Lineur Algebra. Graduate Texts Mathematics. Springer, third edition, 2008.
[81] Neal Parikh, Stephen Boyd, et Proximal algorithms. Foundations and trends0 in Optimization, 13)):227-239 2014.
[82] Jonathan M. Borwein and Adrian S. Lewis. Convex Analysis and Nonlinear Optimization: Theory and Examples. Springer, edition, 2006.
[83] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
[84] Shaocong Ma, Yi Zhou, and Shaofeng Zou. Variance- reduced off off-policy tde learning: Non- aon-asymptotic convergence analysis. Advances in neural information processing systems, 13:14796-14806, 2020.
[85] Shaocong Ma, Ziyi Chen, Yi Zhou, and Shaofeng Zou. Greedy- gg with variance reduction: Finite-time analysis and improved complexity. arXiv preprint arxiv:2103. 16377, 2021.
[86] Zhou, Shaocong Ma, al. Stochastic optimization methods for policy evaluation in reinforcement learning. Foundations and Trends in Optimization, 63))145-192, 2024.
[87] Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. Sample complexity of variance-reduced distributionally robust q glaaminn. Journal of Machine Learning Research, 25(341):1-77 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we have clearly listed the main contributions in the introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We have provided the full proof in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We include the source code and reproducing instructions in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: We include the source code and reproducing instructions in the supplementary material. The data accessed using the third-party API.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
oolPoliiooo iicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ weeiii/ggides/ooooo
ssisnsoiiccyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: Yes]
Justification:
We explicitly specify the details in the code and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
Our evaluation environment is deterministic; that is, there is no randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Included in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://eeurp../.eee httpps//arrrpp..cppppt/tt///eee/ ceccihhescceeelleie ines?
Answer: [Yes]
Justification:
We have reviewed this code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have included this in the conclusion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The source code for reviewing purpose only and not considered as the released new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https //tp:///ee....eeeee Seeeurips.cc/ooffeeeee /LLM) for what should or should be described.
21

---

Appendix
Table of Contents
A Backgrounds
22
A. Common Uncertainty Sets in the Literature
22
The Signed Permutation Group
23
B Worst-Case Uncertainty under EElliss Uncertainty Sets
24
B.I
Supporting Lemmas
24
B.2 Implicit Solution
31
Explicit Solution
C Experiment Details
35
C.I Hardware and System Environment
35
Task Descriptions
................................
35
C.3 Reinforcement Learning Framework
36
C.4 Parameter Details
37
C.S Omitted Visualization
37
Other Implementation Details
37
D Limitations
38
E Additional Supplementary Materials
38
Convergence Robust TD-Leannin
E.2 Parameter Setting and Grid-Search
39
E.3 Other Implementation Details
40
Backgrounds
A.1
Common Uncertainty Sets in the Literature
When evaluating the robust Bellman operator
Σ π(als) a(s,a) + min Σ P(s'ss, PEP a.. PeeS
OEA
the uncertainty set U (Ps.a plays crucial role. Certain structures in l enable efficient
robust policy evaluation. Below, we summarize several widely adopted constructions.
f-divergence The f-divergence family [62, 63] generalizes statistical distances between distribu- tions using convex function f (0,00) R with f(1) 0. For distributions IP and Po such that & Po. the f-divergence is defined as
P(s)
D)(P)Po) = Po(s))
ds.
Po(s)
Special cases include the Kullback Leibler divergence (f(t) t too t). total variation distance (f(t) = b - 1)), and x² -iveeregee (f(t) = (t 1)²). In robust RL, the the f-divergence ball around the nominal transition kernel Po(-|s,00 a) yields the uncertainty set
Pa.a- P()ss,)) A (P(((-) a) ())(PP((|sss & 83,0
The inner minimization in Eq. (7) becomes a distributionally robust optimization problem over Po- As the result, the robust policy evaluation under the Kl.-divergence often requires to repeatedly solve an additional convex program.
22

---

The R-contamination model [31] assumes that the true transition kernel lies within convex mixture of the nominal model Fo and an arbitrary distribution P1:
R-contamination Model
Ps.0 { ( P-) RPa(-ls, a) PP(()s,, E A
where R E [0, 1] quantifies the contamination level. This model leads to closed-fore solutions for the robust Bellman operator, with the worst-case distribution taking mass at the minimum the value function As result, this setup enables efficient and model-free learning algorithms, including robust variants of Q-learning, TD learning, and policy gradients. It is particularly well-suited for online learning, where Po evolves with the observed data.
---norm These sets constrain the deviation from the nominal model Po(-|s, a) using the ---nrm::
p(p) (((ls, E
(P(-sss a) - Po(-1s, & 3s.a
When p 1, the constraint corresponds to total variation distance, while p = bounds the largest single-coordinate deviation. These sets are commonly used due to their interpretability and explicit analytical solution given in [39, 40]. However, their axis-aligned geometry can lead to overly conservative policies in high dimensions.
Integral Probability Metric (IPM) The IPM measures the discrepancy between distributions through expectations over function class F:
(P) == sup [p]l]] - Bp.l/]I. feF
The corresponding uncertainty sets are:
Ps.a (() A dy Po(-sss) & Sea)
The IPM-based uncertainty sets particularly useful when the state space is extremely large or continuous, as explicitly solve the minimization problem in Eq. (7) does not requires to access values all states [43].
Wasserstein Distance The Wasserstein distance [61], grounded in optimal transport theory, ac- counts for the geometry of the state space. Given cost function d : S x S R4 and p 1, the p- o--aaseerstein distance between P and Po
d(s, ((d,,))) 1/p (FeT(P JSxs
(T Po)
inf
where T(P, PP) denotes the set of joint distributions (couplings) with marginals PP and Po- The uncertainty set is then
Ps.a (P a) e s (P(-ss, Po(-ss, < $
Despite their strong theoretical properties, solving the inner minimization often requires dual formu- lations or approximation techniques.
General Uncertainty Sets There also many techniques developed handle the situation where the uncertainty set is general. For example, However, [44] proposes bilevel approach that iteratively solves the worst-case transition kernel approximate the robust value function. However, as demonstrated in [28, 44, 75], solving robust RL problems in the general case is NP-hard.
A.2 The Signed Permutation Group
The signed permutation group plays central role in characterizing the symmetry structure of uncertainty sets in our robust RL framework. Informally, this group consists of all matrices in [SS satisfying the following conditions:
1. Each entry is either 0, 1, -1.
23

---

2. Each row and each column contains exactly one nonzero entry.
In other words, every element this group is a matrix obtained by permuting the standard basis vectors of [R|5] and possibly flipping their signs. Each such matrix can be expressed the product DP, where D is a diagonal matrix with diagonal entries in {+1), and P is permutation matrix representing an element of the symmetric group Sis. This leads to the following formal definition:
Definition A.1 (Signed Permutation Group), Let Sis) denote the permutation group over |S] elements, and let (21))8) be the direct sum of |S] copies of the cyclic group of order 2. Then the signed permutation group, denoted by Signed [ ss), is the semidirect product:
Signed (S
(Zz) ) S(s)
where the action of Sis) on (Z2) $ is given by permuting the order.
In this work, we define the "symmetry" of sets as the invariance under the group action induced by the signed permutation group. Specifically, we say a set B IR $1 is symmetric under a group action by G if B _ B for all g G (or G B == {g b) geG,teB _ B). This notion of symmetry leads to the following structural property of f--norm balls:
Proposition A.2. Let B == (Bp(3)) p33) 13003 be the family of f--norm balls, where Bp( == {u R |ulpp & 3). If there exists group G such that all elements in B are symmetric under the group action by G, then G must be isomorphic to a subgroup of (Sa)
Proof. All elements in B are symmetric under the group action by G; that is, for every p 1 and every 3 > 0,
g(B,(3)) (3) Vg egeG. G.
Therefore, the act Rd - Rd is a worm-preserving bijection; by the Mazur-Ulam theorem, must be affine. Then as it preserves 0, it must be linear.
In particular, taking p = 1 and 3 = 1, each E G is an (invertible) linear isometry of the I-norm unit ball
(1 = {u Rd ||ula & 13.
whose extreme points are exactly
te..., tea}.
Because a linear automorphism of a polytope must permute its extreme points, for each and each g G there must exist a sign E {+1} and an index (i)) e (l,..., d) such that
g(e:) (t))
Thus in the standard basis is represented by signed permutation matrix:
= DP,
where D diag(E Ea) and P is the permutation matrix corresponding to a Sa. Hence every E G lies in the signed permutation group Signed Sa). In other words G s Signed(Sa), which equivalently shows G is isomorphic subgroup of Signed(Sa).
This result provides a useful insight in designing the f--ellipse set: the signed permutation group the largest group under which all fn-norm balls are symmetric. Consequently, to construct a set with less symmetry than the standard f norm balls (as we aim to do with our ép-ellipse sets), it is necessary to enlarge the family include non-ball shapes.
B Worst-Case Uncertainty under l,-Elliiss Uncertainty Sets
B.1 Supporting Lemmas
Definition B.1 (Minkowski sum). Given two sets A, B Rd, the Minkowski sum + 24* 28 23d is defined
A B (a + a A,be B).
24

---

Definition B.2 (Fenchel conjugate [79]). Let 9 R I be function over R.. Its Fenchel conjugate is denoted as g* I2 R and is defined as
g (v) = sup { g(x)}. ZER
We include the following famous Hölder's inequality without providing the proof.
Lemma B.3 (Hölder's inequality). Let p.q [1,+ satisfy ; + For every f. g e IR
fT g & 11
Moreover, equality holds if and only if
g Jp(f),
9=0 or
1964
where Jp(f) denotes any lp-unit vector that attains the maximum inner product with f:
Jp(f) = arg max f u.
(8)
Proof.
The proof can be found in [80].
Lemma B.4. For any U Rd and rudii r,8 2 O, the Minkowski sum of the two ly-norm balls
(p 2 1)
Bp( {u E Idd I|ue æl| = r). Bply, 8) (ur E Rd |l vll & s)
again ball, namely
Bp(x,r) + Bp(v, 8) Bp(r + v.
Proof. For convenience, we omit at the subscript in this proof. It suffices to show two inclusions.
B(xrr)+ B(y.s) B(x yr v.r+ + s).
Take any
z u= u, We B(er)))) B(y,s). Then by the triangle inequality,
2 - (x + w) (uu
(u v\ < lu uu x1 + ||V
wll & ss.
Hence B(x + y, s). proving the first inclusion.
B(x v s) C (z, r) + Bly s).
Let ((x x + y, + 8), so 1z (x w)ll E r s. Set
s
a= r+s
8
(if s = then r = s and the statement is trivial).
5 s
Defin a(2 (x v)
and v = y ( (x y)) Then
u utr (a + 3) - (z + v)) + y + (x y) 2,
and
Iu æ al
(x + v)ll &
(r + 5) =r,
r
|| - yl 3| (xr + y)
(r + 8) r
Thus B(x,7) r) and u B s), so u u t B(x,r) + B(y,s). This proves the reverse inclusion.
Combining (1) and (2) gives the desired equality B(x,r) ) B(y,s) B(x v. + +s.
25

---

Lemma B.5. Let g(u) = -i= [ uplp. Then the Fenchel conjugate of R - R is
inl infz, v-=y Ei 11il.
M1
us, I < 1,
g²(u)=
+00;
otherwise.
| wilp is sum of N "shifted-norms," and the Fenchel conjugate (Definition B.2) sum is the infimal convolution of the conjugates. proceed two steps.
Proof. The key is that g(u) ببك
Let f(x) = | be a single b- normm mapping and q be the dual of & satisfying 1/ + 1 =
By [79]. is standard that
00
Iull < 1, otherwise.
(*()) sup {yT x |
100,
rgRd
Then we consider its "shift" by us Let si(u) = ||u -ullp- By the translation rule for Fenchel conjugates [81, 82).
1 (v) sup(u Tu l uillpt suply' T (I u&) 110]p] vJu, us + (*)
1=4- 4
Hence, (1) = Yuu llvll < 1, otherwise.
+00
As the Fenchel conjugate of ||u u.lpp has been evaluated, the Fenchel conjugate of their sum is given by
infy wi- E, v.
us
l|lll 1,
9 (v)
1illl +00,
otherwise.
Applying this infimal convolution requires each component ft I - IR is proper, convex, and lower semicontinuous, which is automatically satisfied by the fp-norm.
Lemma B.6. Let Po, Aos Se e Rdd and %o 0 be given constants. Let 1 p optimization problem
Then the
d. = inf (po + 2o) w
wi w|
has the unique minimizer given by
Jp + 2o)
W+
Po + Aoolp
sign(ps (po + 2o) O Ioo + A-1
Illo + dol
where O represent the coordinate-wit product and | the coordinate-wise absolute value. The optimal value is solved as
9. = -C'lpo + Xollp-
Proof. By Hölder's inequality,
(po Ao) 3 -lppo + Aolpl
To make the Hölder's inequality achieve the equality, we choose = -1. + Xx)) for some I,
where J, the q-unit vector defined in Eq. (8). Then by letting [w], C, we obtain the final result.
D
26

---

Lemma B.7. Suppose that E RR, fuili=1 C R., and 2 O, and the nomm exponent + = 1
(for 1). Let f1l and C = wlllg Then the optimization problem
min {wr CRd
W
Σ
s.7.
Wi
1=1
|2lllg $ C
feasible and solves the minimizer
sign u, + x) O lu4 App-1
Wi.h 11 p11,
lea + A11
for .,,, N. where x E Rd is given by
N
us + Š11p ). X arg min XT w + |wllg Σ
Proof. We consider the constrained Lagrangian function
N
[((ws), 2)
Σ
u wi + x ws/ x w.
where l|wella & C': = ||w]la Let the dual function 0(²) == inl( [(twss 2). Define
0,(2) = inf ui 2) w0
Mi
By Lemma B.6, it solves
Ju 2)
W/
11 + A1lp
sign us + X) 2 lui + 7IP-1
14s + AB 1--
As the result,
max 0(1) min i
C
Σ
Uu Ālle
Recall that w u wll E Rd is a given vector. Denote
N >
arg min Nw + C
us + Allp g.
Put it back to Wi we obtain the final result.
Lemma B.8. Suppose that (ug Rd, B O, and u Rdd. Let
N
((A, p)) = A3 + inf (u ul) + A Σ
1u - usllp
where p E [1, +00 and 1 p + 1 1. Then
sup (up (4) -1l0 11 1 + A
Σ
w
Wis
i-1
27

---

where
sign(u) uu + )x) 2 14 + Ax p=l
12 + alo}.
Wi, -110 |1l
X arg min
X w + C
Σ 2-1
144 + Xx 1p-1
Consequently, the optimal (** of achieving supa, (0(,, pi) is given by
4= arg max
-1le + 11ll w ui. and X = 1|0 + 1 1
Proof. We take the transformation w = v + μ1l Then
p)) A + inf
For any convex g, by the defimition of Fenchel conjugate (Definition B.2), we have
inf[ Ag(u)] -À
(-w-/A)
sup (- (/)) y g(v)}
Here g(u) E, l/u uillp- Its conjugate is given by Lemma B.5,
inff E, 2 u4 12 < 1,
(²)
(000
otherwise.
where 1 + = We put it back to infu[w a Ag(w)] -xg"(--w/Nh which leads to
p 9
inf[wT u Ag(u)] -> inf
Σ = m
=- w//
12:451
where 1l, & 1. As the result, we take wi - Az; to obtain
infs, wm.-m:/w.ls. En w/ ue
| & X, clse.
inf[w u + Ag(w)]=
-20,
Thus, the full dual becomes
())
X3 infy, mn pm En w.
ues
|w| 4 $ X else.
00
For fixed μ, we need X 2 |wily to keep finite, and (p,,, (1) is decreasing in X. Hence the best choice
x ||l|l ||0 + p1ll
giving
N wi u4.
s μ) -10 + 1111, + x
inf Σ, =. Io+ad
It leads to another optimization problem infy, w,- Imn he Ie ly 2i-1 wi We construct another Lagrangian function solve it. Denote w as the minimizer given by Lemma B.7. Then we obtain
max (0,, u) -1 + 11l A
Σ 1=1
w Ui.
recovers the optimal dual variable is given by
sign(u us + 5ex) O |ui Agl
Wiss
24
|lu, + 2411b-1
where
N TT w + Σ
x arg min
uu + Allo
28

---

0, E IR is non-zero, E [1, +00], and Lemma B.9. Suppose that {u:)-11 = R, A O. B
1. Define
2 Σ
min weR
ww + A
2 uzl
Then when |w| & 2X, the problem is solved as
60
u2
(A 2
|w|
)/i u2/.
Proof. We start from the general case. Define f(u) = ww + A A u - uzl. If [2] & ANV, then the sub-gradient is given by
N
Of(u) w sign(u - u4)
1, O,
where sign(t)
0, 0,
Write {ui] R in the increasing order:
t 0.
u(1) & U12) & ... $ ((N)
Define h* ;
N-w/AA
Then
u ((x*))
the explicit solution. To prove it, we consider u (u(K*), +1); larger than exactly k* us's. That is,
@ʃ(u) w + A(k* (N b+)) 0.
Whenever u( the sign of sub- sub- gradient becomes negative. As the result, f(u) is decreasing when u < u(x*) then increasing when u > u(k*). Now we set N 2. The problem gives
u2 sign(w)lur u2
2
When w 3 0, sign(w)
= +1 and
U1 U2
[nq - U2 min(u1, u2} = u(1).
u*
2
When w 0, sign(w) and
Wi U2
U2l
max(ui, u2) U(2))
Therefore, this formula recovers the original general case solution. Putting it back to yp solves this problem.
The following lemma connects the sum-of-distance description to the quadratic form of an ellipse.
Lemma B.10. Suppose that {ui}²_ Rd. O, and ||ul u2ll2. The ellipse set is given by
= {u Iu u1l2 + uzllz < B}.
Then there exists matrix such that
8 8= (u (u K) A(u ū) < 1),
where ū. ustua More explicitly, the matrix A has the form
4
[12
u1)(u2 u.
32
122
2
32(32
U2 - u
2
29

---

Proof. Define
U1 + U2
u22 -
Wu1l2
3
b v aa - f2,
U2 41
a 2
c
2
2
U2 U1 1
and decompose each R by
2 = u u.
Then #1 ū fe, u2
ū fe, and
|lu 21ll2
||u u222 || + fell2 | fell2-
Hence
| u1l22 la u2l12 & 3
| fell2 1| fell2 & 2a.
Now decompose into
a = J
< = | a²
so that
x fell2 Vla + f)²
The inequality V(a +
+
(ax - -1² + g & 2a equivalent, after two squarings,
a²
5²
+
$ 1, where 8² = a² - f².
a²
b
Finally, observe that
a² x
)ær,
=²= ce )
so
a²
2
I ce + 1 11 (I - ce T
TT((1
a
b2
Setting A
ec
1
1/4
ec 1) implies (u ū) Alu ū)
$ 1. It exactly characterizes
{u u
un
11112 + ||u
U2
2
& 33. Simplifying the form of A leads to
4
== 32 U2
32132 | - u1 (u2 un)(u2 un) T.
u1
This completes the proof.
Lemma B.11. Let Rd, let A E Rdxd be symmetric positive definite, and let P Rd. Define
F(R) = -
(0 -V T A- p1)
(u T p, μed R.
Further set
B-UJA-1 1=
7= A 8= 1_p
82 a, then F attains unique maximizer
4 B
- =
a ava 82 -
= 0 A1
L5p
AAA -TA-I1/TA--11-0 (1 11 A-1o)- T A--1)11)
Proof. We begin by computing the derivative of
F(R) = - (v (v+ A
A- n1) (v p1) Tp.
Using the notation a 1TA -" uT A- and d 1 p. one checks
(w+ +1) A ( 1 u1) ayا² + 28p + 7,
aji + 3
F'(n)
+ 8.
Vap² + 2,3 + 7
30

---

Setting F'()) = 0 gives the stationarity condition
aj + 3 = 8vapp² 28p + +Y,
which upon squaring yields the quadratic equation
(a² axd2) 1 1 + 2,( - 5²) μ (32 132-8% -827) 0.
Let 8 < a. Here 8² > 0, so dividing by 82 gives
8² - 83y 0, a 82
ap² + 2Bp +
whose two roots are
8
1
a7 52.
ava-
82
One verifies by inspecting y-m ('( sign from + to -, and hence is the unique global maximizer.
8 /a that exactly the "+" choice yields change of
B.2 Implicit Solution
In this subsection, recap and prove the full version of Theorem 3.4. Theorem B.12. Let d == [S] be the cardinal of state space and + q 1 for P. E [1, +o0]
Suppose that ( Rdd for each (8, a) E S x A, the uncertainty size 3 > 0, and v Rdd The solution of
min weR
u u
N
s.l.
Σ
lu
uullulllp 8,
(9)
i=1
1 T u 0.
is given by
"* = arg arrggmmnn min[(n + 11* 1=*)) T u + X Σ |uu wilpp]
where
111 u1l 11ll/33 S.N w. J
1x* arg max,
x Iv + 11 1111| 1
(1) = arg ming
At u1) ml l Ei
lus 7llo3.
(10)
sign(u; Aa 10lu p-1
Wi. (11) l μ1
1u
Remark (The procedure of solving "*) ) obtain u² it suffices to solve jr and x as v and all ut's have been given. The first step is to solve
(1) arg ming { (AT(U pl) ||e + ul 11 DN llu 71p3-
(u) - μ1 sign(u, Aa 10/u, +5
uu +
for 1, N. Both variables depend the variable μ and other values known. The next step is to solve
4* = arg max, -111 u + w1lln33 + Σn= w. u.
x* = 1 1
Once (u², (2) is solved, the primal variable μ² is obtained immediately.
31

---

Proof. Our goal is to solve the following constrained optimization problem:
min wER
JJu
v
u
N
s.t.
Σ
u
uellp
& 8
=u=0.
As a constrained optimization problem, the standard approach of solving this problem using Lagrangian multipliers. introduce the Lagrangian multipliers À > for the inequality and j IR for the equality. The Lagrangian function
L(u, = A
3 I uilp - B) +n(11 u), u))
with > O, f E Because this optimization problem is standard convex optimization problem with satisfying the Slater's condition, we have the strong duality
inf sup L(u A, = sup inf L(u, A.). A,
original opt. prob.
Then turn the original optimization problem into solving its dual-form problem. We let the dual function be ( μt) == inf L(u, H.). Then
(() ー3 + inf (v + ul) u + X Σ 1
Us
The above formulation plays the crucial role in our proof. For the implicit solution, we will follow Lemma B.8 to complete the remaining calculation. For the explicit solution, this dual form can be significantly simplified in some cases.
By Lemma 8.8, the dual form can be simplified as
max (p(,, 14) -lv 1 111 +
Σ
T wa Uis
where
sign u4 + Xx) 2 114 + 7o p-1
l2s + 5lp3
Wi, -1l0 μ1ll
d arg min
x TT w + C >
It + 1x1 1p-1111
i=1
and
n² arg max
-1lv + 11l1
3+ + Σ
w j. = = |/ + 1= 11
The optimal primary variable is given as
N
u arg mig + u + * Σ llu - uelp
1=1
B.3 Explicit Solution
In this subsection, we recap and prove the full version of Theorem 3.5.
Theorem B.13. Lerd d = (S] be the cardinal of state space. Suppose that {u,) N C Rdd for each (s, a) E S A, the uncertainty size 3 2 0, and RRd. The optimization problem defined by Eq. (4) is explicitly solved in the following cases:
32

---

(a) Ler N = 1. then
nºah U1 sign(e 1 11 O 1 1 1--
1|0 + 1 1 191
(b) Letp and N = 2. Define ú "1 t-
2
μ² Umax
Umin
and
1 =1
11
Vmax
Umin
2
Suppose that 3 > I22 uilli.
Then the explicit solution to the optimization problem Eq. (4) is
given as
u 8
U1
u2l1
(ign(( +
2
(c) Letp 2 and N 2. Define =1
1 - 1 (u2 un)(u2 u1)
Ω = Q(u1, U2 3)
(11)
A ((v + 1xa 1 1*1), and nº
(12)
1T2-11
Suppose that 3 > I22 u1l2. Then the explicit solution to the optimization problem Eq. (4) given as
is
32
1/2
11
1
0 (v 11
2
1
Proof. We follow the standard routine used in proving Theorem B.12.
(a) When = 1 the objective optimization problem is given by
min weR
s.t.
| U1 & 1,
1 uu= 0.
We take the transformation u/ -- u1. It still satisfies 1 TTu 0. Then the problem become
min a'ERd
v TTu'
s.t.
| $ 1
1 TT 0.
This transformation has turned this problem into the standard fo-norm structure, which has been explicitly solved in [39, 40]. The optimal u' is given for arbitrary p > 1 as
B sign(u + 1 11)1/ 11
w44
10 + n 1
where n²* arg meR || + ulllpp As the result.
u u/ + u1 u1 + 3 sign C + 2 1)/1 + --
|/ 1 9-1
(b) When p = 1, we define the order
((1)s & U(2)
$ u(M
33

---

We follow the same routine as Theorem B.12 and derive the dual function çp\,, p4::
N
(0(A - inf (v + p1) T 2
u
2zl1
d
N
d
① AB + inf
Ug + p1
Mayl
j-1
d
N
= -8 +
Σ
inf (v) + m)u + A
Σ
[2g
uny1
where (i) decomposes the & nnoo by coordinates. When 2, by Lemma B.9, the dual function is solved as
p) = - +
Σ
inf (gg + prug + X
Ug Misl
j=1
1=1
=-18 + Σ
Dv + [1)
U1 + U2j
+ (1
10 + Hl
Juu1 - u2yl
2
AB + (v 1p) T ū
221 - (v 14]
T
2
As the smaller À is, the larger c ni) is. achieves the supremum at x max, + 1pler a- Then we solve
sup (A) A
v 14 + (v + 14) T a + I| + 14||0l
U1
d2
1
(v + 1u)TT
U1 02
2
2
Then we have
supsup p(u, A) 3 122
ui 1l inf 10 + 11 +
(v + 11)
|101
12
+ 1 u.
2
B
U2
As 3 - U22
Un > 0, it solves
R Umax Umin
1 111 + 1 1100
Umax
Umin
and
x
2
2
4
Now we consider the KKT condition of the original Lagrangian function. We solve
ong Tu,,,,,,, 14) ssign( uıj) + ssign(u; u22) + oy 3 0.
For inactive coordinate, the optimal value attained for arbitrary uy (u(n) "(2)): these cases, we simply take Uy M1y u1y+ There are exactly two active coordinates matching the corner-case condition |0g + 1+ 2 j arg max fy and j+ arg min In these cases we take Uj as #ny+ subtracting a drift. It finally solves
2
2
u1 u2lli
Umax + Umin
sign(v
2
(Iu-
Mimns +-min
min
The magnitude coefficient is used to ensure that u* belongs to ||u || < B.
(c) By Lemma 10 there exists a semi sosiivositive definite matrix A such that
{ul ||u un l ||u uzlla < 5) {u (u a) A(u ū) 1). where 22, As the result, the objective optimization problem can be simplified as
min
TTu
WE
s.t.
(u ū)
A(u a) < 1,
T u = 0.
34

---

We follow the same routine as Theorem B.12 and derive the dual function g pi:
((). - inf (v p1)
X(u ū) A(u a)
1
= = (v p1)
ū
(u+ μ1 A (u m1),
41
where the infimum is attained at u 2x
A
(1 + μ1l. Then
sup
(1) = sup X,p
- A + (u+ n1)
T ( n1) A (o wnm
A
41
(u p1) T A p1) (u + u1)Tu
sup
where (i) applies the optimal choice (²) V μ1) A A- + μ1) and n is given by
Lemma B.11 to solve this maximization problem. As the result,
(u 11)
2
where X (U 1)
2
(v + 1s 1* and
1 DTA 1TA-11 1TA
1 1
(I (1 ( - (vTA 1-11)2
Iv 1 A (1
Here, the matrix A is determined by (M1, u2) and the vector in Lemma B.10. We recall that 0. Therefore, n* is further simplified as
2 A--1
1 A-111
It completes the proof in the part (c).
C Experiment Details
In this section, we include the omitted details of Section 4. All source codes and pereme-prreeter settings are available in the supplementary materials.
C.1 Hardware and System Environment
We conducted our experiments on a laptop running Windows Home. The device is equipped with 32GB of RAM, 1TB SSD, an AMD Ryzen 9 7940HS processor and NVIDIA GeForce RTX 4070 Laptop GPU, Our implementation was tested using Python version 3.10.10. Additional dependencies are listed in the supplementary requirements. txt file.
The actual hardware requirement for running our implementation is significantly lower than the specification listed above. Most experiments can be reproduced on consumer- consumer-grade machines with 8-16GB RAM and any CUDA-compatille NVIDIA GPU.
C.2 Task Descriptions
Multi-Asset Portfolio Rebalancing This task captures the setting where institutional investor reallocates capital across multiple assets over discrete time intervals to maintain desired risk-return profile. It reflects strategic portfolio management, such as end-of-day rebalancing or tactical asset allocation. The task emphasizes robustness to market impact under varying capital scales, which is critical for large-volume institutional strategies. In our experiment, we select five representative ETFs, SPY, TLT, GL.D, EFA, and VNQ, and use historical data from May 2021 to May 9, 2022 for training. Evaluation is conducted the out-of- sample period from June 9 to December 9, 2022.
35

---

Table Example of sske execution from an AMZN order book snapshot (21 June 2012). This table shows how our simulator determines the execution price for market buy order. Instead of using the last trade price, the simulator consumes liquidity by filling shares at each ask level in order, starting from the best price. It calculates the VWAP based on the prices and quantities filled, and uses this VWAP as the final executed price.
Level
Price (USD)
Depth (sh)
Exec. 100 sh
Exec. 000 sh
Level
223.95
100
100
100
Level 2
223.99
100
0
100
Level
224.00
220
220
Level 4
224.25
100
O
100
Level 5
224.40
547
O
480
VWAP paid
223.95
224.21
Single-Asset Intra-Day Trading This task models the decision- makinng process of an agent that repeatedly buys sells single asset over short time intervals, such minutes or seconds. reflects the setting of mid-frequency or algorithmic trading, where even small trades can shift market prices. The goal maximize the risk-adjusted cumulative return while accounting for execution slippage, making it standard benchmark for evaluating RL-based trading strategies. We follow the exactly same training and evaluation period as the multi-asset portfolio rebalancing task. In our experiments, we use minute-level historical data for the assets including META, MSFT, and SPY. The observation includes stock price, trading volume, implied volatility, and current portfolio return. During evaluate, we reconstruct the LOB dynamics using the tick-level trade data; the execution price is then simulated as illustrated in Table 2. Same as the multi-asset portfolio rebalancing experiment, the training data covers the period from May 9th, 2021 to May 9th, 2022, and the evaluation period is from June 9 to December 9, 2022. All data are accessed via the Polygon.io Stock Market API.
C,3 Reinforcement Learning Framework
We implemented Gym-like RL trading environment [77, 78] using the historical data including the stock price, trading volume, the implied volatility, and the current portfolio return. Instead of using single timestep data, set lookback_period 30 to account for 30 previous timestep; as the result, the observed state contains Lookback_perice described above, we assign the transition probability from the current state to the next state 8t+1 I as the probability 1, independent with the action. The action is implemented as single float value that determines the position size relative the maximum possible position size based the available capital. The reward is a Sharpe- ratio that consists of two components:
4 = 120 dimension. Given the state
reward- current_retury
current__olatilityy 8 |current_poo poerrentppositin previous_positioon
max_ max_shares
where is small constant to ensure numerical stability, and 8 a tunable coefficient controlling the strength of the transaction cost penalty. The first term encourages higher risk-adjusted return, while the second term discourages frequent or drastic position shifts, which aligns with practical trading considerations under market impact or slippage. We also include 0.1% transaction cost throughout the experiment.
Uncertainty Restriction to Execution Prices Instead of perturbing the whole transition probability, restrict the perturbation the execution price. The formulation of restriction is given follows: we denote the state 8, as two components (pt, ft), and we hope perturb the transition on only. We re-write the transition probability using the conditional probability
P(s St, az) = P(pt+1, fe+1 St, at)
PP(:41 st, a))P(f+1 Pc+1, at,
Then we set the uncertainty u as the perturbation on the kernel IP(p++1 86, at) instead of the whole transition kernel (s Sc. oz).
36

---

Momentum Strategy The momentum strategy is classical intra-day minute-level algorithmic trading method that exploits intraday time-series momentum generate trading signals. We follow the classical implementation presented by [76]. The core idea of the momentum strategy is based on the empirical observation where assets that have performed well the recent past are more likely to continue performing well in the near future, while poorly performing assets are likely to continue underperforming.
C.4 Parameter Details
Detailed descriptions of all parameters are provided in the separate supplementary materials.
C.5 Onitted Visualization
In Figure 3, we have visualized the return curve of the META stock. In this subsection, we include the visualization of the other two assets.
140000
115000
110000
Finel Portote Value
130000
Partolia Value
105000
120000
100000
110000
95000
Nav- Roto Rt.
Momentum
100000
90000
Ratual R. Ma-dat Ellooo)
85000
NSFT
90000
Nomentum
Non Ron-Robusl RL.
Robust RL. (f,-Bei)
Robust RL. (f, (-Ellipaa)
2022-08-43
2022-07-1
2022-08-21
2022-39-26
2022-11-02
2022-12-09
113000
140000
Momenttm
Robust R.. Bal)
Final Portollo Value
130000
110000
Elioso)
Portfolla Value
120000
105000
110000
100000
95000
90000
Nomentum
Non Non-Rotuust RL
Robust (f,- (G-Bai))
Rotust RL (f, Ellippo)
2022-08-49
2022-07-15
20222-8--11
2022-09-26
2022-11-022
20222 12-09
Figure 5; Performance comparison of trading strategies on the MSFT stock and SPY ETF.
C.6 Other Implementation Details
In this subsection, we discuss the practical implementation considerations and provide the details for reproducing our experiments. The source codes also provided in the supplementary material.
RL Algorithm and Value Function Approximations We use the standard Robust Actor -Citic algorithm [43] to train the RL agent with replacing its IPM uncertainty set or doubly-sampling
uncertainty set with the fp-ellipse uncertainty set and the l -orm uncertainty set. We adopt an Actor-Critie architecture with shared feature extractor and separate heads for the actor and critic. The shared backbone consists of three fully connected layers with ReLU activations, mapping the (flattened) input state high-level representation. The actor head outputs the vector with the same dimension as the action through a two-layer MLP followed by sigmoid/tanh activation, depending
37

---

on the underlying task. The critic head predicts the state value using similar two-layer MLP. All linear layers are orthogonally initialized promote stable training. When updating the Actor network, we apply the classical PPO algorithm [83].
To apply the robust RL framework, apply the discretization to the execution price. We introduce two additional hyper-parameter to control the discretization level: 2NV + represents the number of total discretization the execution prices and represents the strength of each level. Given the execution price we have 2. potential execution prices .N8] in total. This discretization approach allows to directly
Discretization of Execution Prices
[p- - ........ - 6, P. 5, ... p apply the closed-form form solution to solve the optimal u*.
D Limitations
Despite the promising results, this work has several limitations. First, the theoretical analysis largely relies on the assumption of finite state and action spaces for tractability, which is primarily due to the limited development of deep learning theory. Second, the market impact simulation during the evaluation stage, while based on trade-level data, remains an approximation and may lack accuracy for extreme volumes. fact, when the volume sufficiently high, often triggers momentum- based strategies deployed by other institutions in the market, resulting in higher market impact than reflected VWAP. Third, although our experiments demonstrate the robustness of the ép-ellipse uncertainty set under increasing volumes, we do not test its robustness under increasing trading frequency. Designing experiments this setting more challenging, as the behavior of RL agents varies significantly across different time scales.
E Additional Supplementary Materials
In this additional supplementary material', we provide several components omitted in our previous technical appendix. It includes: (1) The theoretical analysis of the robust TD-leamnngg (2) The further breakdown of the parameter setting and the grid-search strategy to optimize the optimal parameter. (3) The other implementation details that are related robust reinforcement learning setting.
E.1 Convergence of Robust TD-Learning
Theorem (Convergence of Robust TD-Leaming). the uncertainty set is defined as the elleppipse uncertainty set, then the worst-case transition probability P- (+(ss, can be represented as
(+)s,,)) = Po(-ss, + u*
where 3 is the radius of the uncertainty set and u* solved in Theorem 3.4 and Theorem 3.5.
Remark. For non-robust policy evaluation, variance-reduction techniques are often employed to accelerate training [84-86]. For robust policy evaluation, the most recent progress can be found in [87]. For simplicity, we only present a naive robust TD-learning approach.
Proof. We apply the following T'D-lerniing update rule:
V(s) V(s) +n r(s,a) +(s,a)+(((s)--- v(s)) +yVT x
(13)
stand. em. Po
'This section was originally submitted as separate supplementary material, We include it here for the reader's convenience.
38

---

It is easy to observe that this update rule is equivalent to the non-robus, TD-learnin over the worst-case transition probability:
le +V+)
=r(s,0) 0) + 7 Po P Σ (
=r(s,a) + 7 Pols ( + Σ [(als)u
=r(s,a) + Σ [[dss][P((s1s.0)
a( + 7 Σ ((lls)
-r(s, a) + V(s'),
where (i) applies the worst-case transition probability we have derived. Therefore, by applying existing TD-leamnnng convergence analysis, we obtain that the convergence rate is also VK"
E.2 Parameter Setting and Grid-Search
We apply grid-search to select hyperparameters to balance learning performance, computational efficiency, and robustness to uncertainty. In this section, we detail the architectural and training hyperparameters used across our three model variants: standard RL, elliptic uncertainty robust RL, and ball-shaped uncertainty robust RL.
E.2.1
Model Architecture
All models share the same actor-critic network architecture, which consists of:
Feature Extractor: Three fully-connected layers with ReLU activation functions, each containing 256 neurons.
Actor Network: Two fully-connected layers (256 and 128 neurons) with the final layer using tanh activation function to bound actions within [-1, 1].
Critic Network: Two fully-connected layers (256 and 128 neurons) with a linear output layer.
Weight Initialization: Orthogonal initialization with gain of /2 for linear layers to improve training stability.
This architeture selected by default and we did not further the model architecture.
E.2.2 Training Parameters
We employ the Proximal Policy Optimization (PPO) algorithm with the following hyperparameters:
Table 3: PPO Training Hyperparameters. The PPO clip parameter regulates the range of policy updates ensure stability. Policy update epochs indicate how many times each batch reused for learning. Batch size is the number of trajectories used per update.
Parameter
Standard RL
Robust RL (Elliptic)
Robust RL (Ball)
Learning Rate
3 x 10
3 x 10-4
3 x 10-4
Discount Factor (7)
0.99
0.99
0.99
PPO Clip Parameter (e)
0.2
0.2
0.2
Policy Update Epochs
10
10
Batch Size
64
64
64
We apply the grid-search on the learning rate and the PPO clip parameter; however, the gear-search not applied to improve the performance. Instead, we apply the grid-search to identify default
39

---

parameter group to ensure the algorithm convergence. In the remaining experiments, we will use the same parameter in the robust RL method for fair comparison.
E.2.3 Robustness Parameters
For our robust RL implementations, we include another group of hyper- parameters.
Table 4: Robustness Parameters. The robust type is the string used in codes to determine the different types of uncertainty sets, The parameter 3 used determine the size of uncertainty set.
Parameter
Robust RL (Elliptic) Robust RL (Ball) PIN2
Robust Type
P1
Uncertainty Set Parameter (5)
1 x 10
x 10
Uncertainty Dimension
3
Epsilon
x 10-3
1 x 10-3
We grid-search he parameter 3 from [0.1, 0.01, 0.001, 0.0001, 0.00001] and use the best result on the training period with adopting the early stopping. Then the best model is evaluated in the testing period to obtain the final performance.
E.3
Other Implementation Details
The learning rate scheduler employs a ReduceLROnPlateen strategy with factor of 0.5 and patience of 5 episodes. This adaptive approach reduces the learning rate when performance plateaus. Addi- tionally, gradient clipping with a threshold of 0.5 is applied to prevent exploding gradients and ensure stable training.
The choice of 3 = 1 x 10 4 for both robust methods represents a balance between model performance and robustness. Too large value would overly emphasize worst-case scenarios, while too small a value would not provide sufficient robustness against uncertainty. Through empirical testing, this value demonstrated the best trade-off between trading performance and resilience to market fluctuations.
Advantage normalization is applied prior to the policy update to stabilize training and improve convergence. The PPO clip parameter e = 0.2 prevents excessively large policy updates, maintaining proximity to the previous policy while allowing sufficient exploration.
To apply the robust RL framework, we apply the discretization to the execution price. We introduce two additional hyper-paramette level: 2N + represents the number of total discretization in the execution prices and ó represents the strength of each level. Given the execution price p. have 2.V potential execution prices e, P.P.P pappt c.....P apply the closed-form solution solve the optimal u². The uncertainty dimension and the epsilon in the robustness parameters are used to determine the discretization level of execution prices. For example, when the discretization level is given by
Discretization of Execution Prices
control the discretization
[p- - Ne,...,,
P + Ne] in total. This discretization approach allows us to directly
have three potential execution prices [p+e, ep,t, P,p_e which also corresponds to the uncertainty dimension. In the nominal transition kernel, we assume the distribution over this discretized set as [0.25, 0.5, 0.25], which provides sufficient flexibility for us to choose the uncertainty u.
Uncertainty Restriction to Execution Prices Instead of perturbing the whole transition probability, we restrict the perturbation on the execution price. The formulation of restriction is given as follows: we denote the state 8, as two components ft), ) and we hope perturb the transition on Pt only. We re- re-write the transition probability using the conditional probability
P(( St, a.) PPP:+1 f(+1 11, az)
= P(p++1 St. P Pt41,,
Then we set the uncertainty w as the perturbation on the kernel P(p++1 81, 88, a:) instead of the whole transition kemnel [P(8+1 8t, 0z). Given that the transition of the execution price Pt+ is discretized, the transition probability P(p++1 8t, 0t) a discrete distribution. For example, when taking the uncertainty dimension 3, we in fact obtain: P(p(+1 p + st,a,) 0.25 wt.
40

---

P(pt-1 =p st. at) = 0.25 + u², and P(p++1 - | = -..5 + ua. By default, in the
PIN2-type robust RL, we choose the shift parameter in [u u. u²1 = [0.1 [001=101-1 10.1-1/3,- 1/33 the action is buying a stock and 01 [u' u2, a stock.
u2, u0] [-1/3, 1 1/3] if the action is selling
Solving the Worst-Case Uncertainty After restricting the domain to the discrete execution price, we turn the original worst-case Bellman equation into the following form:
(s) = r(s. a) + min ΣΣ (s
a = r((s,)) + mim Σ (s) 8, P. a,
=r(8,a) + mim
V (=((s))(aas)) [Pol'' (8,a) un P(S 11,
a -((8,0) + + ΣΣ V (s)) + mimin ΣΣ Una ( (,
Moreover, we use P( p', s, a) as a deterministic transition. is common when training in the historical data, (I) the observed feature f' comes from the existing dataset and the action does not change its value (in our implementation, the action a will only affect the execution price), and (2) many features are directly calculated based on the current state, action, and the execution price such as the remaining cash. As the result, we obtain
(*( a() ΣΣ VT (
min E u(p')V'(s))
=r(s,o) ? ΣΣ (s + min V
Here we still write u V for convenience, while the value function V represents the 2N + 1 dimensional vector with value taken on s' ("), where takes 2N 1 values and f' is deterministically determined by (p', s, a). After making this modification, using Theorem 3.4 and Theorem 3.5, or existing l, norm formula [39, 40].
solve min, u V
41


Reviews:
[{"review_id": "MVxxUxnYdt", "reviewer": "Reviewer_73NP", "summary": "This paper proposes a robust RL approach particularly motivated by the lack of consideration about market impact in finance applications. The authors generalize the uncertainty set of missspecficiation the transition kernel from a unimodal definition to \"elliptic\" by introducing multiple foci and respective sizes. Based on this new uncertainty set, this paper then develop theoretical analysis of the Bellman update LP problem, and develop a TD learning algorithm. Numerical experiments on minute-level single-asset strategy and large-volume portfolio rebalancing are conducted.", "questions": "- Can the authors comment more on how the foci and set size can be decided? Ideally, can it be selected in a data-driven fashion, yet, not computationally intensively as cross validation? \n- Can the authors provide a more theoretical characterization of how the elliptic uncertainty sets overcome the limitations of simple uncertainty sets other than the simple 5-dim examples?\n- Why specifically focus on TD-learning algorithm? Any particular benefit?", "limitations": "- The authors mention that their analysis only works with finite state and action space, which may have some discrepancy with real-world financial applications.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "*Strength*\n- The writing is very clear and polished. I like how the problems are well motivated and cleanly defined. Figure 2 is a nice explanation. \n- The theorem presentation and theoretical analysis seem very rigorous with details.\n- The numerical experiments are very detailed.\n\n*Weakness*\n- The overall methodology is quite straightforward and thus technical novelty is limited. \n- The motivation of market impact, though interesting, is a bit niche. \n- It is particularly unclear why the proposed robust optimization approach is a good solution to the proposed problem of market impact. Intuitively, the environment change due to market impact is potentially structured and directional, and it’s not immediately clear whether a worst-case robust approach, even one with an elliptic structure, captures that structure optimally. It would be helpful to better justify why robustness is a suitable solution and not overly conservative.", "was_revised": true, "final_justification": "I commend the authors' detailed and thoughtful response, and increased my score to reflect the improvement.", "rebuttal": "We deeply appreciate the reviewer’s thoughtful and constructive feedback. Below, we provide our point-by-point responses to each of the concerns raised.\n\n## Questions\n\n> **Q1:** Can the authors comment more on how the foci and set size can be decided? Ideally, can it be selected in a data-driven fashion, yet, not computationally intensively as cross validation?\n\n**Response:** In our experiments, the foci and set size are treated as hyperparameters. Specifically, we fix one focus at \\\\$0, representing the unperturbed case, and the other at \\\\$$\\pm$0.1, representing a scenario where the execution price is impacted by \\\\$0.1 (as the $(s,a)$-uncertainty set is action-dependent, we use different foci for the buying and the selling action). For the set size, we select from a predefined set of candidate values $\\\\\\{ 0.1, 0.01, 0.001, 0.0001, 0.00001 \\\\\\}$ and choose the one that yields the best performance on the training period (May 9th, 2021 to May 9th, 2022).\n\nWe acknowledge the reviewer's concern regarding the computational cost of hyperparameter tuning described above. To address this issue, we believe the selection process can be made more data-driven by leveraging established techniques from finance and deep learning. In particular, financial models [1] such as Kyle’s Lambda or the Square-Root Impact Model can be used to estimate the expected market impact, which can then inform the choice of foci. The corresponding uncertainty around this estimate can be derived from traditional statistical confidence intervals. Alternatively, modern deep learning models could be employed to learn these parameters directly from data, offering a potentially more scalable and adaptive solution. As the result, we believe this direction can naturally connect to many existing works in data-driven finance and market microstructure modeling.\n\n* [1] Hasbrouck, Joel. \"Trading costs and returns for US equities: Estimating effective costs from daily data.\" *The Journal of Finance* 64.3 (2009): 1445-1477.\n\n\n\n> **Q2:** Can the authors provide a more theoretical characterization of how the elliptic uncertainty sets overcome the limitations of simple uncertainty sets other than the simple 5-dim examples?\n\n**Response:** From a theoretical perspective, we characterize the conservativeness of an uncertainty set using the robust value function. That is, if the robust value function is smaller, then the uncertainty set is more conservative.  In the ideal scenario, suppose the true market impact is precisely represented by a specific perturbation vector $u_0$. If we define  $\\mathcal{U}=\\\\\\{ u_0\\\\\\}$ as the uncertainty set; then robust RL over this singleton set exactly recovers the MDP induced by the true market dynamics. In this case, optimizing over the robust MDP is equivalent to training directly on an environment that accurately simulates the market impact.\n\nHowever, in practice, we cannot know the exact value of $u_0$. We set $\\beta = \\\\\\|u_0\\\\\\|+\\epsilon$ as the size of the uncertainty set. Then both the ball $\\mathcal{U}\\_b=\\\\\\{ u: \\\\\\|u\\\\\\| \\leq \\beta\\\\\\}$ and the ellipse $\\mathcal{U}\\_e=\\\\\\{u: \\\\\\|u\\\\\\|+\\\\\\|u-u_0\\\\\\| \\leq \\beta \\\\\\}$ can contain the true value $u_0$, and moreover, we have $\\mathcal{U}\\_e \\subset \\mathcal{U}\\_b$. This inclusion relation leads to the inequality \n$$\\min\\_{u \\in \\mathcal{U}\\_b} V^\\pi\\_u\\leq \\min\\_{u \\in \\mathcal{U}\\_e} V^\\pi\\_u \\leq V\\_{u\\_0}^\\pi$$\nThis suggests that the robust value function for the ball uncertainty set $\\min\\_{u \\in \\mathcal{U}\\_b} V^\\pi\\_u$ is theoretically more conservative (more precisely, it is not less conservative) compared with the ellptic uncertainty set $ \\min\\_{u \\in \\mathcal{U}\\_e} V^\\pi\\_u$. \n\n\n\n> **Q3:** Why specifically focus on TD-learning algorithm? Any particular benefit?\n\n**Response:** Typically, in traditional RL, there are two methods to estimate the value function: (i) *Monte-Carlo method*, also known as REINFORCE. A complete (or truncated) trajectory is sampled, and its discounted return is used as an unbiased but high‑variance estimate of the value. (ii) *TD-learning method (and its variants)*.  The agent bootstraps from a one‑step look‑ahead, repeatedly minimizing the TD‑error. This yields lower‑variance, more sample‑efficient updates.\n\nIn robust RL, the TD-learning method  become almost unavoidable. Recall that our goal is to solve the robust value function which is defined in the form of $\\min\\_u V\\_u^\\pi$. The $\\min\\_u$ term make it challenging to directly apply the Monte-Carlo approach, as sampling from the worst-case transition requires to take an additional step to approximate it before sampling from it. As the result, the TD-learning method is much more preferable. After obtaining an estimate of the robust value function via TD learning, we can plug it into any standard policy‑gradient scheme (e.g. the PPO algorithm) using the generalized advantage estimate (GAE).  \n\n\n\n## Weaknesses\n\nWe thank the reviewer again and would like to take this opportunity to clarify several points regarding the identified weaknesses.\n\n> **W1:** The overall methodology is quite straightforward and thus technical novelty is limited.\n\n**Response:** We thank the reviewer for recognizing the clarity of our methodology. We would like to emphasize that our work goes beyond a straightforward combination of robust RL and financial applications. Specifically, we introduce a novel perspective on modeling market impact. Prior studies have typically approached this by either (i) approximating the underlying MDP using fine-grained limit order book (LOB) data, or (ii) simulating market dynamics through large numbers of interacting agents. These methods aim to learn a direct mapping from observed data to execution prices. In contrast, our approach models market impact through uncertainty sets within the robust MDP framework, which provides a new perspective for understanding the market impact. Furthermore, we extend this framework by proposing an elliptical uncertainty set tailored to financial settings, addressing the drawback of traditional uncertainty set used in the RL literature when applied to the financial application. Lastly, we extend the closed-form solution of $\\ell_p$-norm uncertainty set to our generalized ellipstic uncertainty set, which is also a new result even in the robust RL literatue.  \n\n\n\n> **W2:** The motivation of market impact, though interesting, is a bit niche.\n\n**Response:** Instead of being niche, we tend to believe the topic is still emerging and growing in importance. \n\n* (i) On the finance side: With the field of modern quantitative trading is becoming increasingly crowded and competitive, understanding market impact has become a central concern in algorithmic trading and portfolio optimization.  By proposing a robust RL framework that better and easier models market impact, our work contributes to this growing body of research and provides tools that are under-explored but potentially applicable to real-world trading systems.  \n* (ii) On the RL side: We are also interested in extending the application of the elliptical uncertainty set to other domains where RL has already played a central role, such as robotics and control. The geometric flexibility of the ellipse allows for modeling more directional or correlated uncertainties, which may benefit certain real-world systems with structured noise or anisotropic dynamics. \n\n\n\n> **W3:** It is particularly unclear why the proposed robust optimization approach is a good solution to the proposed problem of market impact. Intuitively, the environment change due to market impact is potentially structured and directional, and it’s not immediately clear whether a worst-case robust approach, even one with an elliptic structure, captures that structure optimally. It would be helpful to better justify why robustness is a suitable solution and not overly conservative.\n\n**Response:** We deeply appreciate the reviewer's insightful question. Capturing the optimal structure is indeed a challenging question and our approach cannot (and is not designed to) solve it. Instead, our method tells how to make it better when we cannot accurately obtain the execution price. Let's break down this idea in pieces:\n\n* The robust value function $V^\\pi:=\\min_{u\\in\\mathcal{U}} V_u^\\pi$ models the worst-case discounted future return. If the uncertainty set $\\mathcal{U}$ is larger, it is more conservative (as it gives smaller value). If the set $\\mathcal{U}$ is smaller, it is less conservative.\n* The ideal case is that there exists the uncertainty $u_0$ exactly characterizing the MDP induced by the ground truth market impact and we directly use $\\mathcal{U}=\\\\\\{ u_0\\\\\\}$ as the uncertainty set. In this case, training a RL agent on the robust MDP defined over $\\mathcal{U}=\\\\\\{ u_0\\\\\\}$ is equivalent to train over the real LOB data.  \n* The less ideal case is that even if the uncertainty set $\\mathcal{U}$ contains some additional uncertainties; e.g. $\\mathcal{U}=\\\\\\{ u_0, u_1\\\\\\}$, the minimum value $\\min\\_{u \\in \\mathcal{U}}$ is still achieved exactly at $u\\_0$. In this case, the robust value function recovers to the previous ideal case. Otherwise, the robust value function gives a strictly smaller value than $V^\\pi\\_{u\\_0}$, which makes it more conservative. \n\nUnfortunately, our approach cannot theoretically rule out this “otherwise” scenario. However, it is still guided by the above intuition: the smaller the uncertainty set, the less likely it is to include the conservative uncertainty. Our method contributes by trimming down the traditional uncertainty sets used in robust RL, thereby reducing the risk of being overly conservative."}, {"review_id": "0TN1nDGZ9x", "reviewer": "Reviewer_aH4Q", "summary": "This work analyzes the limitations of traditional symmetric uncertainty sets and proposes a novel elliptic uncertainty set that better captures real-world market fluctuations. Based on this new formulation, the paper studies its theoretical properties and derives closed-form solutions under different norms. Furthermore, it introduces a TD method to evaluate policies within this uncertainty framework. Finally, simulation results are provided to validate the proposed algorithm.", "questions": "My main concerns focus on the following two points:\n\n1. When determining the worst-case transitions under an $l_p$ norm uncertainty set, as proposed in the paper, it is unavoidable that the resulting worst-case transition may not be a valid probability distribution. In particular, the closed-form solution can include negative probability values, which are not meaningful. This issue could significantly degrade the model's performance in practice. In financial applications—where interpretability and robustness are critical—such behavior is particularly problematic. Could the authors elaborate on the practical reasonableness of such results? How might this issue be mitigated or avoided?\n\n2. Compared with uncertainty sets defined via $f$-divergences, the elliptic uncertainty set does not guarantee that the resulting distributions are absolutely continuous with respect to the nominal transition distribution. This raises a concern: could it be the case that, under certain conditions, the worst-case transition does not exist or is not meaningful in real-world applications? Are there specific and common conditions under which this lack of absolute continuity is acceptable or justified?", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "***Strengths***\n1. The paper is clearly written. It highlights the limitations of traditional uncertainty sets and provides illustrative examples, which are especially helpful for readers who may not be familiar with the financial context.\n\n2. The paper proposes a novel elliptic uncertainty set and offers solid and significant analytical results under this formulation.\n\n***Weaknesses***\n1. While the elliptic uncertainty set can mitigate transition mismatches observed in practice, it may introduce new types of mismatch—particularly when applying the closed-form solution.\n\n2. The formulation in the reinforcement learning (RL) section could be more rigorous. For instance, in Line 92, the proper term is \"$s,a$-rectangular uncertainty set.\" In Line 108, the correct terminology is \"visitation distribution\" to distinguish it from the stationary distribution. Additionally, a formal mathematical definition of this concept is missing. I recommend that the authors refer to the latest edition of the textbook [1] and the recent paper [2] to improve the precision of the presentation.\n\n[1] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. Vol. 1. No. 1. Cambridge: MIT press, 1998.\n\n[2] Agarwal, Alekh, et al. \"On the theory of policy gradient methods: Optimality, approximation, and distribution shift.\" Journal of Machine Learning Research 22.98 (2021): 1-76.", "was_revised": true, "final_justification": "According to the review comments and rebuttal, I keep my borderline accept rating.", "rebuttal": "We deeply appreciate the reviewer’s positive evaluation and constructive feedback. We have many thanks to reviewer's helpful suggestions on improving  rigor and clarity in the formulation; we will replace the imprecise phrasing with the correct terms and formal mathematical definitions.  We will also consult the recommended references to align our notational conventions and definitions with current standards in the literature.\n\nBelow, we provide our point-by-point responses to each of the concerns raised.\n\n> **Q1:** When determining the worst-case transitions under an $\\ell\\_p$ norm uncertainty set, as proposed in the paper, it is unavoidable that the resulting worst-case transition may not be a valid probability distribution. In particular, the closed-form solution can include negative probability values, which are not meaningful. This issue could significantly degrade the model's performance in practice. In financial applications—where interpretability and robustness are critical—such behavior is particularly problematic. Could the authors elaborate on the practical reasonableness of such results? How might this issue be mitigated or avoided?\n\n**Response:** We appreciate the reviewer's careful reading and insightful question. Simply speaking, we will need to make an assumption to ensure each $u$ in the uncertainty set is well-defined; that is,  their induced transition probability $\\mathbb{P}=\\mathbb{P}_0+u$ presents no negative values).  This assumption is stated on Lines 100–101 and is consistent with prior theoretical work on $\\ell_p$-norm uncertainty sets [kumar2023a,kumar2023b].\n\n* [kumar2023a] Kumar, Navdeep, et al. \"An efficient solution to s-rectangular robust markov decision processes.\" *arXiv preprint arXiv:2301.13642* (2023).\n* [kumar2023b] Kumar, Navdeep, et al. \"Policy gradient for rectangular robust markov decision processes.\" *Advances in Neural Information Processing Systems* 36 (2023): 59477-59501.\n\nWe agree with the reviewer that our presentation on this assumption in our paper could be unclear.  In the revised version, we will explicitly label it as a formal assumption and clarify that the uncertainty set should be restricted accordingly to ensure the all uncertainties are well-defined.\n\n> **Q2:** Compared with uncertainty sets defined via $f$-divergences, the elliptic uncertainty set does not guarantee that the resulting distributions are absolutely continuous with respect to the nominal transition distribution. This raises a concern: could it be the case that, under certain conditions, the worst-case transition does not exist or is not meaningful in real-world applications? Are there specific and common conditions under which this lack of absolute continuity is acceptable or justified?\n\n**Response:** We thank the reviewer for raising this important point. Indeed, when using uncertainty sets that are not defined via $f$-divergences, the resulting worst-case transition distributions are not guaranteed to be absolutely continuous with respect to the nominal distribution. This can lead to pathological cases where no positive $\\beta$ yields a valid (non-negative) transition probability. For instance, if the nominal transition is a delta distribution (i.e., assigns probability 1 to a single next state), then any perturbation under an $\\ell_p$-norm may fail to produce a valid probability vector, as noted by the reviewer in the previous question.  \n\nThis limitation is intrinsic to the $\\ell_p$-norm uncertainty set and has not been fully addressed in prior work (e.g., [kumar2023a, kumar2023b] mentioned above). As a result, we follow the convention in the literature and assume the nominal transition distribution is fully supported over the state space, which also ensures that the uncertainty set contains at least one valid perturbation, and hence the worst-case transition remains well-defined. In practice, this assumption is often satisfied. For example, in financial modeling, stock prices are commonly modeled using geometric Brownian motion (i.e., the Black–Scholes model), which yields a strictly positive density for the normalized price and implies full support over the state space. Therefore, the issue of non-existence due to lack of absolute continuity may not pose a practical limitation in financial domains."}, {"review_id": "n6ii79avx4", "reviewer": "Reviewer_E9d3", "summary": "This paper tackles the market impact challenge in financial RL by introducing elliptic uncertainty sets that capture directional price shifts. \nContributions include \n1) Developing a novel class of elliptic uncertainty sets better to capture the empirically observed directional nature of market impact. \n2) Deriving implicit and explicit solutions for robust policy evaluation.\n3) Evaluating on real-world financial data, results show that the new robust TD learning algorithm outperforms baselines in Sharpe ratio and remains robust under high volumes, offering a more effective RL framework for financial markets.", "questions": "1) In Example 3.1, the paper states that symmetric uncertainty sets fail to capture directional uncertainty. How do the proposed elliptic uncertainty sets capture the directional nature of environmental shifts in financial markets? It would be helpful to have intuitive examples or analyses similar to those for symmetric sets to better demonstrate the superiority of elliptic uncertainty sets.\n2) The paper proposes an ellipse with N foci based on the classical ellipse. How is N determined in real financial market experiments, and what value of N achieves the best performance in practical applications?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros: \n1) The paper investigates financial market characteristics and identifies market impact as a key challenge in robust RL, proposing elliptic uncertainty sets to address the limitations of traditional symmetric sets, thus focusing on a critical financial issue with innovative solutions.\n2) The paper offers a thorough analysis of common uncertainty sets in the literature, complemented by comprehensive theoretical derivations of both implicit and explicit solutions for elliptic uncertainty sets.\n3) The paper is well-structured and concisely written, ensuring clarity in organizing backgrounds and ideas and presenting technical content and proving.\nCons:\n1) The paper lacks detailed derivations and development of robust RL-related methods, such as the robust Bellman operator and robust TD learning algorithm, which might be better if presented in the background.\n2) The experiments use relatively few baselines, insufficient to fully demonstrate the method's superiority, and the analysis of results is overly simplistic.\n3) The theoretical analysis relies heavily on finite state and action spaces, which may not hold in real-world financial markets with continuous dynamics, potentially limiting generalizability.\n4) Experiments focus on historical data and standard market conditions, lacking validation under extreme scenarios, which limits real-world risk assessment.", "was_revised": true, "final_justification": "I appreciate the authors for providing a comprehensive response, and I support the acceptance of this paper.", "rebuttal": "We deeply appreciate the reviewer’s positive evaluation and constructive feedback. Below, we provide our point-by-point responses to each of the concerns raised.\n\n## Weaknesses\n\n> **W1:** The paper lacks detailed derivations and development of robust RL-related methods, such as the robust Bellman operator and robust TD learning algorithm, which might be better if presented in the background.\n\n**Response:** We appreciate the reviewer’s helpful suggestion. In the revised version, we will include additional background on robust RL. \n\n> **W2:** The experiments use relatively few baselines, insufficient to fully demonstrate the method's superiority, and the analysis of results is overly simplistic.\n\n**Response:** We appreciate the reviewer's suggestions. We would like to clarify that our experiments are conducted to validate the effectiveness of reducing the market impact instead of proposing a new RL trading strategy; therefore, adding more baselines may not be helpful in validating this statement. \n\n> **W3:** The theoretical analysis relies heavily on finite state and action spaces, which may not hold in real-world financial markets with continuous dynamics, potentially limiting generalizability.\n\n**Response:** We appreciate the reviewer’s insightful comment. We agree that the discretization may not fully capture the continuous dynamics of real-world financial markets. Bridging this gap between discrete theoretical models and continuous environments is an important and active area of research in modern RL, and we consider it a valuable direction for future work. \n\n> **W4:** Experiments focus on historical data and standard market conditions, lacking validation under extreme scenarios, which limits real-world risk assessment.\n\n**Response:** We appreciate the reviewer’s suggestion. We agree that evaluating performance under extreme scenarios is important for comprehensive risk assessment. However, such scenarios often exhibit significant distributional shift from the training data, making model performance in these regimes less representative or informative. Therefore, we didn't include experiments in this setting.\n\n## Questions\n\n> **Q1:** In Example 3.1, the paper states that symmetric uncertainty sets fail to capture directional uncertainty. How do the proposed elliptic uncertainty sets capture the directional nature of environmental shifts in financial markets? It would be helpful to have intuitive examples or analyses similar to those for symmetric sets to better demonstrate the superiority of elliptic uncertainty sets.\n\n**Response:** We thank the reviewer for the thoughtful question and helpful suggestion. Below, we clarify how elliptic uncertainty sets capture directional uncertainty in financial markets, using the following simplified example:\n\n* Assume we are **buying** a stock at the price $100$. The execution price would be $\\\\\\{99.8, 99.9, 100.0, 100.1, 100.2 \\\\\\}$, which corresponds to the uncertainty set $\\\\\\{-0.2, -0.1, 0.0, 0.1, 0.2\\\\\\}$.  We recall the symmetric uncertainty set $\\\\\\{u: |u|\\leq 0.2 \\\\\\}$: if it needs to include $0.2$ in this set, we must also include $-0.2$ by the symmetry. \n* However, in our case, we may hope model the execution price as $\\\\\\{100.0, 100.1, 100.2 \\\\\\}$ instead of $\\\\\\{99.8, 99.9, 100.0, 100.1, 100.2 \\\\\\}$, as when buying a stock, it is more likely to push the price up, leading to the non-symmetric behavior. In this case, we need to use the elliptic uncertainty set $\\\\\\{u: |u|+|u-0.2|\\leq 0.2 \\\\\\}$. We can easily verify that both $-0.2$ and $-0.1$ are not in this set. \n\nMore generally, we treat execution prices as random variables and define uncertainty sets over their probability distributions. The elliptic set incorporates a directional prior by filtering out the unlikely execution prices such as $-0.2$, introducing prior knowledge into the RL agent; such knowledge would be difficult to learn from historical data alone, as the historical data does not contain information about the agent's own market impact. We will revise the paper to include this intuitive example and clarify the modeling advantage of elliptic uncertainty sets.\n\n> **Q2:** The paper proposes an ellipse with N foci based on the classical ellipse. How is N determined in real financial market experiments, and what value of N achieves the best performance in practical applications?\n\n**Response:**  In our experiments, we fix the number of foci as $2$ and do not tune the number of foci; specifically, we fix one focus point at \\\\$0, representing the unperturbed case, and the other at \\\\$$\\pm$0.1, representing a scenario where the execution price is impacted by \\\\$0.1 (as the $(s,a)$-uncertainty set is action-dependent, we use different foci for the buying and the selling action). We found that this two-point elliptical uncertainty set is sufficient to capture the directional nature of price impact in our setting, and using a larger $N$ may result in difficulty of solving the worst-case uncertainty (this is also the reason why we further derive the explicit closed-form solution in Theorem 3.4); therefore, we did not consider configurations with more than two foci."}, {"review_id": "JYcaRcMtyk", "reviewer": "Reviewer_SFEg", "summary": "The simulator built with historical data has a huge limitation in that the actions of the agent cannot influence the stock price. To address this issue, this paper has proposed a novel approach that model the market impact with elliptic uncertainty sets. The proposed approach has solid theoretical guarantees, and the empirical results have shown the effectiveness of the proposed approach.", "questions": "The authors are encouraged to clarify the theoretical analysis.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper deals with an important problem, the market impact issue.\n\nThe proposed approach has a solid theoretical grounding. The authors have derived closed-form solutions under the proposed uncertainty sets. Beyond that, the authors have presented explicit solutions under certain conditions.\n\nThe experiments are conducted on real-world financial data, and the experiment results show improvement over existing RL baselines.\n\nThe theoretical analysis is a little hard to follow, and it is unknown whether the uncertainty in the theorem is consistent with that often occurs in the real market, i.e., are the assumptions in the theorem realistic?", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s positive evaluation and constructive feedback. To improve the clarity and readability of the theoretical analysis, we will provide more detailed explanations at each step. Below, we address the remaining questions raised by the reviewer:\n\n> **Question:** The theoretical analysis is a little hard to follow, and it is unknown whether the uncertainty in the theorem is consistent with that often occurs in the real market, i.e., are the assumptions in the theorem realistic?\n\n**Response:** We thank the reviewer for this insightful question. Rather than referring to it as an \"assumption\", we prefer to name it as a \"model\"; that is, a model for describing the market impact. Obviously, all models are inaccurate on describing the real market. Therefore, we interpret the question as: *Does the theoretical model proposed in our work offer a more realistic representation of market impact compared to existing models?* \n\nWe answer this question by splitting it into the following three parts:\n\n* *How to model the market impact with uncertainty sets?*\n\n    When we hope buy an asset with the price $p$, the actual execution price $p_e$ is typically different from $p$ due to the market impact. Instead of trainning a neural network to predict the execution price, the uncertainty set is used to model this difference by giving a range of it. Specifically, in traditional robust RL, we set $\\beta=|p_e -p|$ and consider the ball $u\\in [-\\beta, \\beta]$. It means the true execution price $p_e$ would belongs to the set $[p - \\beta, p +\\beta]$.  However, we observe that market impact tends to be directional. For instance, buying usually drives the price up, while selling pushes it down. As a result, it may be more realistic to model the execution price as $[p, p + \\beta]$ when buying and $[p - \\beta, p]$ when selling. Our elliptical uncertainty set captures this asymmetry while a ball can never make it.\n\n* *Is this approach more realistic compared to existing models?* \n\n    Yes. By incorporating directionality, our model better reflects typical market behaviors: When buying a stock, the actual execution price is usually higher than the current price; therefore, we can filter out some less realistic moves in our uncertainty set to make it more realistic. Moreover, the elliptical uncertainty set generalizes the ball-shaped uncertainty set, making it capable of modeling both symmetric and asymmetric impact patterns. This flexibility enhances its applicability to a wider range of market conditions.\n\n* *Which part is still unrealistic?*\n\n    We acknowledge that our model still simplifies certain aspects of market behavior. Two key limitations include: (i) The extreme market move. Even large trades may have negligible impact in highly liquid markets with abundant counter-orders.  In highly liquid markets with abundant counter-orders, even large trades may have negligible impact. For instance, during widespread selling pressure, a large buy order may still execute at a decreasing execution price. (ii) The discretization. We discretize execution prices using a finite grid, which introduces approximation error compared to continuous-valued prices. \n\n    Despite these limitations, we still believe the elliptic uncertainty contributes meaningfully to bridging the gap between simulated environments and real-world financial markets by addressing the intrinsic disadvantage of classical ball-shaped uncertainty set. \n\nWe appreciate the reviewer again for the positive evaluation. We hope our explanation resolves your concern."}]

Author rebuttal:

Meta-review: Motivated by the mismatch in financial RL between training on historical data and deployment in live markets (where trading actions cause market impact), the authors introduce a class of elliptic uncertainty sets that better capture the directional nature of market impact, deriving efficient closed-form solutions for robust policy evaluation. Experiments on single- and multi-asset trading tasks show improved Sharpe ratios and robustness under higher trade volumes, offering a scalable approach for financial RL.

The paper is very well-written and the results are solid. There were initial concerns centered around the motivation, technical novelty, and the limitation on tabular MDPs, but the reviewers remained (or became more) positive about the paper after the discussion period. Overall, I recommend a borderline accept.
Decision: Accept (poster)