OPHR: Mastering Volatility Trading with Multi-Agent Deep Reinforcement Learning
Authors: Zeting Chen
Xinyu Cai
Molei Qin
Bo An
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Quantatitive Trading
Option Trading
Primary area: applications
TL;DR: We introduce OPHR, a multi-agent reinforcement learning framework that outperforms traditional approaches in volatility trading by dynamically selecting options positions and optimizing hedging strategies.

Abstract: Options markets represent one of the most sophisticated segments of the financial ecosystem, with prices that directly reflect market uncertainty. In this paper, we introduce the first reinforcement learning (RL) framework specifically designed for volatility trading through options, focusing on profit from the difference between implied volatility and realized volatility. Our multi-agent architecture consists of an Option Position Agent (OP-Agent) responsible for volatility timing by controlling long/short volatility positions, and a Hedger Routing Agent (HR-Agent) that manages risk and maximizes path-dependent profits by selecting optimal hedging strategies with different risk preferences. Evaluating our approach using cryptocurrency options data from 2021-2024, we demonstrate superior performance on BTC and ETH, significantly outperforming traditional strategies and machine learning baselines across all profit and risk-adjusted metrics while exhibiting sophisticated trading behavior. The code framework and sample data of this paper have been released on https://github.com/Edwicn/OPHR-MasteringVolatilityTradingwithMultiAgentDeepReinforcementLearning

OPHR: Mastering Volatility Trading with Multi-Agent Deep Reinforcement Learning
Zeting Chen"
Xinyu Cai Nanyang Technological University Singapore xinyu009@e wtt.ttu.ee eccu.edd....
Nanyang Technological University Singapore schen0910es ntu.tuuue edu.ag
Molei Qin
Bo An Nanyang Technological University Skywork AI Singapore
Nanyang Technological University Singapore molei001@e ntu edu.
boan@ntu. edu.
Abstract
Options markets represent of the most sophisticated segments of the finan- cial ecosystem, with prices that directly reflect market uncertainty. In this paper, introduce the first reinforcement learning (RL) framework specifically de- signed for volatility trading through options, focusing on profit from the difference between implied volatility and realized volatility. Our multi-agent architecture consists of an Option Position Agent (OP-Agent) responsible for volatility tim- ing by controlling long/short volatility positions, and Hedger Routing Agent (HR-Agent) that manages risk and maximizes path-dependent profits by select- ing optimal hedging strategies with different risk preferences. Evaluating our approach using cryptocurrency options data from 2021-2024. we demonstrate superior performance on BTC and ETH, significantly outperforming traditional strategies and machine learning baselines across all profit and risk-adjusted met- ries while exhibiting sophisticated trading behavior. The code framework and sample data of this paper have been released on http::/gttthbb.om//lleen//////// Mastering Volatility Teter//innniiiiiiiiieeeeeeeeeeee
Introduction
Options markets represent one of the most sophisticated segments of the financial ecosystem, offering traders the ability construct complex, non-linear payoff's and express nuanced views market direction, volatility, and timing. More recently, cryptocurrency options markets have emerged as rapidly developing segment, with Bitcoin (BTC) [Nakamoto] 2008 and Ethereum (ETH) [Buterin and Others 2013) options gaining significant traction, which provide unique research opportunities due their transparency, 24/7 operation, and comprehensive data accessibility that often unavailable in traditional markets.
Options Kariya and Liu 2003 serve the insurance of financial markets, with prices that directly reflect market uncertainty. While traditional option pricing models like Black-Scholes-Mern (BSM) BBackk and Scholes) 1973 Merton 1973) provide a theoretical framework for option valuation, they rely on idealized assumptions that rarely hold in practice: continuous trading, no transaction costs, constant volatility, and the ability hedge risk exposures perfectly. These simplifications,
"Equal contribution. Correspondence to: Xinyu Cai xinyu009@c .tu. edu.eg
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

while mathematically elegant, fail to capture the empirical realities of options markets, where the difference between implied volatility (IV) and realized volatility (RV) creates persistent volatility
trading opportunities [Carr and Madan 1998 Christensen and Prabhala 1998 Ni et al.
2008 Sinclair
2013
Tan et al.) 2024].
IV is the volatility priced in the option price [Canina and Figlewski, 1993 Dumas et al. 1998), and RV is the true price fluctuations in the underlying asset (nndersn al. 2003 McAleer and Medeiros 2008]. IV is higher than RV most of the time because the market needs to pay a premium to option sellers as a reward for bearing tail risk. Option sellers typically need trade the underlying asset to hedge risk. Options aren't 't just insurance; during periods of significant market volatility, traders can often profit by buying options and properly hedging. In these instances, RV typically exceeds IV. Therefore, the volatility trading involves two challenges to beat the market: i) Good volatility timing: sell options to collect premium when markets are calm, and buy options to profit when markets are about to become volatile; ii) Select a proper hedging strategy to manage risk when selling options and take profit when buying options as illustrated in Figure
RL (Sutton and Barto 1998 Mnih et al. 2015 Hasselt et al. 2016 offers a compelling alternative by learning optimal trading policies directly from market data. Unlike traditional approaches relying on option pricing models with restrictive assumptions and RV forecasting [Andersen et al. 2003]. RL frames option trading as sequential decision problem, naturally incorporating practical con- straints such as transaction costs, discrete hedging intervals, and market dynamics. This data-driven methodology eliminates the need for explicit pricing models, instead allowing RL agents to discover mispriced volatility through experience.
Although RL has been successfully applied many trading tasks [Jiang and Liang 2017 Conegundes
and Perciral 2020 Yang et al. 2020 Briola et al. [021] Nagy et al.) [023 Takara et al. 2023 Zong al. 2024 Qin et al.j 20244, options trading is an untouched area due to its complexity. In this
paper, we introduce the first RL framework that addresses the complexities of volatility trading in a data-driven manner. Our framework is composed of 2 parts: Option Position Agent (OP-Agent) controlling the long/short of volatility, and ii) Hedger Routing Agent (HR-Agent) selecting the optimal Hedgers with different risk preferences to perform dynamic Delta hedging based on positions and market conditions. To make they work coordinately with each other, we first distill a sub-optimal Oracle policy to OP-Agent, then alternatively train HR-Agent and Op-Agent.
We evaluate our approach using historical cryptocurrency options data on BTC and ETH, leveraging the transparency and data accessibility of these markets conduct comprehensive experiments. Our findings demonstrate that the RL model significantly outperforms traditional rules-based strategies in both long and short Gamma implementations, with particular emphasis on managing tail risk when selling options and identifying optimal timing for buying options during market dislocations.
The primary contributions of our work are:
The first RL framework specifically designed for trading volatility through options, moving beyond previous RL applications that focused solely on option pricing and hedging.
A novel multi-agent architecture comprised of two specialized agents working in concert: OP-Agent, identifying volatility trading opportunities, and HR-Agent, tuning risk preference.
An effective training methodology that enables the two agents to learn collaborativlyy from market data, balancing the competing objectives of profit maximization and risk management.
Comprehensive empirical results demonstrating the framework's 's ability to properly manage tail risk when selling options and identify advantageous timing for buying options.
2 Background & Related Works
This section introduces the fundamental concepts of options, option pricing models, and volatility trading strategies that form the basis for our RL approach to trade volatility.

---

Price Oscillation within Range
Price Trending Single Direction
Straddle at 100
Straddle at 100
After First Hedgg at
After First Hedge at 110
After Second Hedge 110
After Second Hedge
Proftt & Loss
Profit & Loss
85
95 100
105
110
115
100
105
110
115 Underlylnng Price
120
125
130
Underlying
Figure I: Importance of proper Hedging strategy: In this figure, red arrows show the PnL path with hedging, and blue arrows show the PnL path without hedging. In the oscillation market (left), the price path is 100 + 90 + 110 + 100. In the trending market (right), the price path is 100 110 120 130. These figures ignore the time value decay for simplicity. All curves in the two figures will drop as time passes.
2.1 Options and the Black-Scholes Model
Financial options are derivative contracts that give the holder the right, but not the obligation, to buy (call option) or sell (put option) an underlying asset at a specified price (strike price) on or before a specified date (expiration date). Options characterized by three key attributes: option type (Call or Put), strike price (K) at which the underlying asset may be transacted, and expiry time (T) by which the option must be exercised. The payoff structure of European options is defined mathematically as:
V (ST) = [(s - K) (
if Call
1 (T)+S))+ Put
where ST is the underlying asset's price at expiration and (x)+ mean max(x,,
The Black-Scholes-Merton Model (BSM) represents the foundational framework for pricing options. While real markets violate several of its assumptions, the model provides crucial theoretical insights and practical tools for options traders. The BSM assumes constant risk-free interest rate r, an underlying asset price following geometric Brownian motion (dS, = pS,dtt + &S:WW,), frictionless markets with no transaction costs, and no arbitrage opportunities. These assumptions lead to the
Black-Scholes partial differential equation: 8t equation yields the theoretical price of European option V 952
2V 52 By + +r as - rV = 0. Solving this S1)
Beyond pricing, BSM provides essential risk exposure metrics known as "Greeks". Delta (A = av
quantifies an option's 's price sensitivity to changes in the underlying asset price. Gamma (I'
g"V
25
measures the rate of change in A relative movements in the underlying asset. Theta (Ô
aV measures time decay-how much value option loses time passes. Vega ( quantifies sensitivity to volatility changes. These Greeks provide critical insights for risk management and volatility trading strategies: traders can hedge those risks they do not want and keep those they are willing to take. The details of BSM and Greeks are presented in Appendix
A.I A.I
2.2 Volatility Trading
There are many volatility trading strategies due to the complexity of options. One of them is the long/short Gamma strategy, which aims to profit from the difference between IV-the volatility priced into options by the market, and RV-the actual price fluctuations experienced by the underlying asset.
The Gamma-Theta Relationship. Options positions with positive I' experience convex payoffs (accelerating gains) as the underlying price moves in either direction. However, this advantage comes cost: negative O or time decay. This fundamental relationship informs the core mechanics of volatility trading. Using Taylor's 's theorem, we can approximate how an option's 's price changes:
1 V(S + dS, dt) = V(S.o.t) + ds x A (ds)² x X dt
2
3

---

When Delta- Delta-hedging is applied (neutralizing the dS x A term through offsetting positions in the underlying). the theoretical PnL becomes:
PnI. 6- S ro? - -2) dt, 2
where o, = (5: )2 is the instant RV (profit) and a is IV which determines the magnitude of the time decay rate è (loss). The formula shows that the expectation of (V is higher than the IV(a) priced in options, we are expected to profit from a long Gamma trade and vice versa.
Dynamic Delta Hedging for Gamma Strategies. Dynamic Delta hedging (DDH) (Hull and White 2017] serves distinct functions in the Gamma strategy. For long Gamma positions, it acts as a profit mechanism through I' laaiinn-ssttmttcaalllll
buying low and selling high as price fluctuations alter A values. For short I' positions, it primarily mitigates risk by neutralizing directional exposure. As illustrated in Figure the effectiveness of hedging depends on market conditions. In oscillating markets, hedging locks in profits when prices return to original levels, while in trending markets, hedging may limit potential gains as prices move consistently one direction. This demonstrates why proper hedging strategy selection is crucial despite similar RV in both scenarios. sale-bbsed Hedger is commonly used, which monitors portfolio Delta against predefined thresholds, Alternatively, RL approaches can be employed to train hedging strategies with varying risk-aversion levels through actor-critic methods using risk-adjusted reward functions [Buehler et al. 2019 202 2022 Murray
et al. 2022]. While recent work such as DLOT [Tan et al. 2024 applies deep learning to option portfolio management by constructing long/short positions across multiple stocks' straddles without underlying hedging, our work addresses fundamentally different problem: single-asset volatility trading with dynamic delta hedging. This distinction is crucial-our our approach tackles the complex challenges of gamma scalping, continuous hedging decisions, and path-dependent profit optimization that emerge from maintaining isolated volatility exposure, requiring delta-hedged straddle PnL. as the optimization target rather than unhedged portfolio returns.
3 Problem Formulation
We first present several financial concepts that are necessary to trade options. Then we describe how to formulate the volatility trading problem as Cooperative Markov Decision Process (MDP).
Financial Concepts for Volatility Trading. To build an RL framework for volatility trading, we first define the following financial concepts: Underlying asset, denoted as S,, refers to the financial asset from which the options are derived. In our context, it primarily includes the perpetual future of the underlying asset, which is used to hedge the A exposure of our volatility trading positions. Straddle is combination of a call and a put option with identical strike prices. At the money (ATM, strike K equals S,) straddle is a common strategy to trade volatility due to its zero A exposure and substantial I' and Vega exposure. Position, denoted as Pt, consists of option positions: the holding of option contracts, and hedging positions: the holding of the underlying asset. Net value V/2 represents the total value of our account, which consists of the options positions value, underlying position value, and cash: V22 = Voptions + yuundellying + Cash Balance. Market features F, are features derived from options and underlying market data analyze price trends and volatility. Greeks, G. A r, θ Vega), are risk measures derived from the BSM model, indicating the sensitivity of the net portfolio value (V₁) different variables, which have been described in Section 2.1 Hedgers are hedging policies that take information like Greeks and time-to-expires as input, and decide how many hedging instruments trade at each step, which have been described in Appendix B.2
Markoy Decision Process Formulation. We formulate volatility trading as a Cooperative MDP, where two specialized agents collaborate to optimize trading performance. The OP-Agent identifies and executes volatility trading opportunities, while the HR-Agent selects optimal hedging strategies every N steps. This cooperative relationship is critical: the OP-Agent's position decisions directly influence the HR-Agent's state space, while the HR-Agent's 's hedging strategy affects the returns of positions established by the OP-Agent. Both agents share the common objective of maximizing portfolio net value while managing risk exposure. Specifically, in our framework, the problem can be formulated as (MP, M )
MDP =< 500 A TOP
R
(4)
MD =< Shy. AK The Rbb hr
>

---

Env
Market Features
Portfolio
Greeks
Phase
Phase
Hedgers
Distill
Train
Oracle
OP-Agent
HR-Agent
The overview of OPHR. The upper section depicts the trading environment containing market features, portfolio status, and Greeks measurements: both market and portfolio will affect the Greeks. The lower section shows the two-phase training process: In Phase I, sub-optimal Oracle policy distills market knowledge to initialize the OP-Agent. In Phase II, the OP-Agent is trained alternatively with the HR-Agent, who selects proper Hedger based on Greeks and market features.
Figure 2:
OP State is the market feature Fr denoted as = Fr. OP Action, denoted as at E A {+1, -1,0,, determines the target position (long/short/neutral) of options at each step. HR State, denoted as s (F', Pe, Gz), consists of the market feature Fr, position information Pt. and Greeks Gr Hedgers are set of hedging policies {T(() Ji=1 with different risk-aversion levels, which the hedger
method proposed by Murray et al. [2022] trains.. A rule-based Hedger hedger is used as baseline our framework. They take the portfolio and Greeks (Pr, G.) as input and output the final trading amount of the underlying asset to perform hedge. HR Action, denoted a."" A involves selecting the optimal Hedger from the set of predefined Hedgers In Figure [ the two Hedgers are price-based Hedgers. The red one has finite threshold (less risk-seeking), and the blue one has an infinite threshold (never hedging, extreme risk-seeking). Details of Hedgers are presented in Appendix B.2 The HR-Agent makes a decision every N steps, and the selected Hedger executes hedging at each step. Transition: The OP State contains market features F; only, so its transition $441 (s,") is the market dynamics, which is not affected by the action. However, the positions and Greeks in s are affected by the two agents: the OP-Agent determines which options to hold, and the HR-Agent's s selection of Hedger affects how the underlying position evolves under varying market conditions. Thus, the transition hhr T (shr ai at takes the action of two agents into account. Reward function: The reward for OP-Agent is defined as the change in portfolio net value op r++1
Vi+1 Vú: the reward of HR-Agent is defined as the advantage of the selected Hedger n hedger a
over rule-based baseline Hedger E+tt
Vetnper
Vi Vengh..
4 OPHR: A Two-Phase Multi-Agent Framework for Volatility Trading
In this section, we demonstrate our OPHR framework as shown in Figure 2 to solve the volatility trading problem. We first introduce the training of the option part, consisting of an OP-Agent, which involves an n-step TD error update. Then we introduce the hedging part, which consists of a set of Hedgers with different risk preferences and an HR-Agent selecting the optimal Hedger based on positions and market conditions. To make the two parts work coordinately with each other, we design a joint training method, which first initializes the OP-Agent with a sub-optimal Oracle policy, then trains the HR- HR-Agent and the Ap-Agent alternatively.
5

---

4.1 OP-Agent Optimization
OP-Agent is responsible for determining the holdings of options at each time step. Its primary objective is to adjust the overall payoff structure of the portfolio as a function of the price of the underlying asset, according to current market conditions, enabling the agent to systematically capture profits from price fluctuations of the underlying asset.
n-step TD Learning. In options trading, portfolio returns are typically realized over time, as long Gamma strategies rely on large price movements, and short Gamma strategies accumulate theta over time. In addition, frequent hedging incurs transaction costs, introducing noise into short-term rewards. These issues render the reward signal both delayed and noisy, limiting the effectiveness of one-step TD updates to capture the true value of positioning decisions. To address this, we adopt n-step TD learning, which incorporates a sequence of future rewards and bootstrapped value estimation, enabling the agent to better estimate the long-teny value of current positioning decisions.
27
( = E
Σ CTtr
+00 arg arg max m Qo(s),a1)
(5)
a
The algorithm for online learning is presented in Algorithm
Algorithm 1: OP-Agent Online Training via stPp TD Error
1: Reinitialize trading environment Env
2: for E Range(0, (0, T,, pPP) do for & E Range(O, 7t 1) do
3:
4: if f mod n = then Get HR Action a arg max, (s a), set Th Hedger
5:
6:
With probability e, choose a random action, otherwise a; arg max, Qo(x, a) op Fxecute actions and get next State 8t1 and Reward r41
7:
8:
Store step transition: (s,, a; fri=1 . rr st+ ) in RTP Sample batch of transitions from Rop and update with n TD error Eq.
9:
return Q
4.2
HR-Agent Optimization with DT--Hedger Baseline
Baseline Hedger. As we mentioned in Section 2.2 DDH plays a vital role in volatility trading, which manages risk and realizes profit by adjusting the underlying position. The Delta threshold hedging strategy is a commonly used rule-based method that monitors the Delta exposure of the current portfolio and trades the hedging instruments make the Delta exposure 0 when the Delta exposure exceeds the threshold. In this work, the Baseline Hedger heddger is used to compute the reward for the HR-Agent and other baseline experiments.
Deep Hedgers. A key drawback of the Baseline Hedger is that it only considers the snapshot of A exposure. However, all Greeks (A, θ and Vega) are functions of time and IV and the underlying price S,, which change dynamically as time passes and the market evolves. Therefore, DRL-based methods have been applied to train better so-called Deep Hedgers to optimize the long- term performance. Murray et al. 2022 provides an actor-critic alternative to train a set of Deep hedger J.K with different risk aversions on simulated data.
Hedgers (T)
Ji=1
Hedger-Routing Agent. However, as illustrated in Figure I it is important to consider the market condition when making the hedging decisions, but the Hedgers do not take the market conditions into account, and it is difficult to incorporate the real-world market data into the training protocol of Deep Hedgers by simulation. further optimize the hedging behavior, we employ the HR-
bodger based on hedger the portfolio and market conditions every n! step. We execute the selected Deep Hedger т ((al))
Agent to select Deep Hedgers. The HR-Agent chooses an optimal Deep Hedger π. ((af)
in Env and the Baseline Hedger &hedger in a twin environment, Env duplicated from Ere, for nitr steps. Then the reward for a!! is computed as the difference of net values in the two environments:

---

phr "Htk Vinni - Veenrr. This relative reward design helps the training of the AR-Agent. The training
algorithm of the HR-Agent is demonstrated Algorithm 2
Algorithm 2: Agent Online Training via 1-step TD Error
1: Initialize Ene and En
2: for E Range (0, T, nhr) do
3: With probability c, choose a random action, otherwise a!" = arg max, (u( ahr)
4: for t E Range(0, nz'h
5: Get OP Action at = (s,)s)
6:
Use selected Hedger "(a?")
hedger in Env, and baseline Hedger hedger in Enve
7:
Execute actions and get the next State 8141
op
8:
phr ghr ) in Ro, and synchronize Ene with Frc Get phr "ttkk store (sh" at" "ttpl (tnnl))
9:
Sample batch from RoP and perform DQN update vl with Eq. (5) soft update vp
10:
return
4.3
OPHR Training
Phase 1: Offline Initialization. Long trajectories in n-step TD increase the variance of target returns, often leading to unstable training and slow convergence. So we employ sub-optimal Oracle policy to generate initial experience for guidance in the initialization phase. The Oracle OP policy T generates long/short signals by comparing the future RV and current IV and employs the Baseline Hedger he to make hedging decisions. Specifically, if the future RV 2 a + (9) places long position, if the future RV S a - (3) places short position; otherwise, place a neutral position. While inherently sub-optimal, its action aligns with desirable and profitable trading behaviors, significantly reducing the exploration burden, accelerating convergence.
Phase 2: Iterative Online Training. To further improve the trading performance, we alternatively train the AR-Agent and OP-Agent. The necessities iterative training lie in folds. i) From the MARL perspective: iterative training can reduce complexity, improve stability and convergence. ii) From the option trading perspective: the OP-Agent can take more aggressive positions if the hedging policy is more sophisticated, and the HR-Agent also needs to learn how handle these more aggressive positions. The detailed implementation is presented in Algorithm 3 in Appendix [B.]
Experiment
5.1 Experiments Setup
Datasets. To comprehensively evaluate the proposed algorithm, we conduct experiments on BTC and ETH options data obtained from Deribit. The dataset splitting is shown in Table [ This period covers diverse market conditions, including bull markets (e.g.. 2019 and 2021), bear markets (e.g., 2022), and periods of elevated volatility (e.g., the 2020 COVID-19 pandemic and the 2022 crypto market crash). We utilize hourly-level data to capture intraday price dynamics and volatility structures critical to volatility trading. The dataset includes complete options chains across wide range of strikes and expirations (weekly to quarterly), as well as comprehensive market indicators such as implied volatility surfaces, open interest, and trading volume.
Table I: Dataset Splits for Experiments
Dataset BTCUSD 19/04/01 22/12/31
Train
Validation
Test
23/01/01 23/06/30 23/07/01 - 24/007001
ETHUSD 19/04/01 22/12/31 23/01/01 23/06/30 23/07/01 24/07/01
Evaluation Metrics. We evaluate our method using financial metrics: 1 profit, risk-adjusted profit, 2 risk and 3 trade criteria. Returns are aggregated into day-level before calculation. Total Return (TR) the overall return rate of the test period. Annual Volatility (AVOL) is the standard deviation of daily returns annualized. Maximum Drawdown (MDD) measures the largest loss
7

---

from any peak to show the downside risk of the strategy. Annual Sharpe Ratio (ASR) is the profit adjusted by volatility risk. Annual Calmar Ratio (ACR) measures profit adjusted by downside risk. Annual Sortino Ratio (ASoR) applies downside deviation as the risk measure. Win Rate (WR) is the percentage of trades that result in profit. Profit/L.os Ratio (PIR) measures the average profit of winning trades relative to the average loss of losing trades. Holding Period (HP) is the average holding time per trade. The detailed definitions of these metrics are presented in Appendix C.I
Table 2: Performance comparison Crypto markets with 8 baselines. Pink, green, and blue results show the best, second-best, and third-best results.
Profit
Risk Risk-Adjusted Profit
Risk Metrics
Trade Metrics
Data
Model
TR(%)t
ASR1
ACR1
ASoR
AVOL(%))
MDD(
WR(%))
PLR
Long
-33.05
-1.32
-0.77
-2.13
24.90
42.70
21.74
1.82
Short
2.90
0.09
0.10
0.09
24.55
21.10
73.91
0.37
MR
19.80
-1.32
-0.74
-1.54
15.40
27.45
41.94
1.18
MOM
-3.30
-1.85
-0.85
-1.61
20.45
44.80
46,99
0.78
GARCH
-40.83
-4.29
-0.95
-5.54
9.51
42.76
26.32
0.78
BTC
DoepVol
-14.24
-1.30
-0.65
-2.44
10.92
21.85
36.36
1.33
GBDT
-30.45
-2.88
-0.91
-4.75
11.22
35,25
35.16
1.15
MLP
-74.55
4.73
-1.13
-5.14
18.36
76.80
26.39
1.20
LSTM
-21.78
-1.99
-0.86
-3.26
11,46
26.64
40.00
1.13
DLOT
4.91
0.52
0.55
0,66
21,40
8.92
47.97
1.11
OP
21.43
1.19
1.46
2,03
17.01
13.84
44.50
1.86
OPHR
33.10
1.87
3.35
3.27
16.83
9.41
45.93
2.05
Long
28.25
-0.72
-0.53
-0.83
37.55
50.95
34.78
0.71
Short
-34.75
-1.12
-0.69
-1,25
33.00
53.05
56.52
0.48
MR
-19800
-1.32
-0.74
-1.54
15.40
27.45
41.94
1.18
MOM
-3.300
-1.53
-0.84
-1.46
23.90
43.65
44.44
0.87
GARCH
-52.59
-3.79
-0.90
-4.18
13.85
58.10
12.50
0.62
ETH
DeepVol
-22.82
-1.79
-0.96
-3.03
12.71
23.76
42.86
1.15
GBDT
-48.54
-3.76
-1.06
-4.55
14.22
50.28
30.00
MLP
-70.66
-4,83
-1.15
-5.37
16.80
70.59
29.01
1.03
LSTM
-62.67
-4.22
-1.11
-5.03
15.42
64.44
31.31
0.75
DLOT
1.19
0.07
0.09
0.09
17.14
13.53
42,80
1.34
OP 23.49
0.85
0.77
0.79
26.85
29.45
52.74
1.29
OPHR
44.89
1.76
1.58
1.67
24.42
27.25
55.61
1.43
Baselines. To comprehensively evaluate our method, we implement multiple baseline approaches categorized into three major groups: Directional Volatility Strategies: These include pure Long Volatility and pure Short Volatility strategies. The Long Volatility strategy purchases at-the-money straddles when volatility conditions favorable, while the Short Volatility strategy sells at-the- money straddles when IV exceeds historical RV. Single Factor Models: These strategies allocate long short positions based
reversion (MR) (Poterba and Summers, 1988 Wong and Lo Wood et al. 2022 or momentum (MOM) (Mskkowtz et al. 2012 Lim et 2019 Wood et al.
2009
2022 Tan al. 2023 Heston al. 2023 Wood et al. 2023) factors. Machine Learning Models:
We implement GBDT Dorogush et al. 2018). MLP[[LeCun et al. 2015). and LLSTM[Hochriite and
Schmidhuber 1997 predict optimal long/short volatility positions by leveraging their respective strengths in capturing non-linear relationships, complex feature interactions, and temporal dynamics volatility patterns. We also include GARCH[Bollersley 1986), the seminal econometric model for volatility forecasting that captures time-varying conditional heteroskedasticity through autoregressive dynamics, serving the classical benchmark in volatility prediction, and Deep (oreno Zohren 2024]. a deep learning framework for volatility forecasting, representing the current frontier in neural network-based volatility modeling. Additionally, we create an end-to-end DL baseline following DLOTTTan et al. [224]. which applies deep learning to directly predict the options' PnL. instead volatility. Our Method: To evaluate the effectiveness of different parts of our method, OP uses the AP-Agent and the rulc-based Hedger.
and

---

Underfying
Hedge
Underlying Price with Trades
Open
Close
Thets
50%
Delta
Vega
RV
Decomposition
Figure 3: Trade examples of OPHR on BTC. The left panel illustrates a long trade, which primarily generates profits from I and Vega exposure, while incurring theta costs. The right panel presents short I' trade, where the strategy mainly profits from theta accrual over the holding period and gains from decline in implied volatility, with the primary risk stemming from I' exposure.
5.2 Baseline Comparison
As demonstrated in Table 2, our proposed OPHR approach exhibits consistent superiority across both BTC and ETH markets. The method achieves the highest total returns while ranking first in all risk-adjusted performance metrics (ASR, ACR, and ASoR). These results underscore OPHR's effectiveness in risk management and profit optimization through its adaptive hedging framework.
In contrast, conventional directional volatility strategies display notable limitations due to their inherent tail risk profiles. Short Gamma positions achieve high WR but poor PLR because they collect small premiums consistently while remaining exposed to rare but catastrophic losses during volatility spikes. Conversely, Long Gamma strategies offer better PLR by capitalizing on large market movements, but suffer negative returns due to low WR and theta decay during market stability.
Traditional bastd approaches (MR and MOM) demonstrate suboptimal performance, as they fail to adapt abrupt volatility regime shifts that characterize cryptocurrency options markets. Similarly, MI. baselines (GBDT, MIP, LSTM, Deep GARCH) struggle with profitability despite occasionally achieving lower volatility, due two fundamental limitations: first, they fail to bridge the gap between forecasting RV and optimizing path-dependent PnL outcomes in options trading; second, they consistently underperform in capturing the long- Gammaa opportunities hidden in the fat tails of volatility distributions, precisely where the most significant profit potential exists, as discussed in Appendix [.2]] The PnL. curves are also presented in Figure 4 and 5S) in the Appendix. DLOT significantly outperforms RV prediction-bel based methods, benefiting from an end-to-end design that avoids model assumptions similar in spirit to our RL approach. However, lacking adaptive hedger selection and further RL optimization, it cannot match our method's 's performance.
5.3 Closer Look
Table 3: The long/short trading behaviour
Long
Short
Data
Model
HP
WR(%)
PLR
HP
WR(%)1
PLR
OP
20.94
31.96
2.21
52.94
55,86
1.60
BTC
OPHR
21.29
33.68
2.28
51.12
56.64
1.87
OP
8.91
44.25
1.51
47.56
60.98
1.08
ETH
OPHR
9.16
47.17
1.71
50.59
63.79
1.18
Table reveals how the HR-Agent systematically enhances trading performance across position types. The distinct holding periods between long and short positions (approximately 9-21 hours for long versus 50-51 hours for short) reflect OPHR's sophisticated volatility timing capability. This

---

pattern aligns with the market reality that cryptocurrencies typically experience extended periods of relative stability (suitable for short volatility positions with longer holding periods), punctuated by brief episodes of extreme volatility (ideal for long volatility positions with shorter holding periods). OPHR effectively adapts to these regime shifts, maintaining short positions during calm periods while quickly capitalizing on and exiting from volatility spikes through tactical long positions.
This volatility timing ability translates directly into performance improvements, with OPHR en- hancing WR by 1.72-2.92% for long positions and 0.78-2.81 % for short positions compared to the base OP model. The PLR improvements are equally notable, particularly for short positions where OPHR's hedging refinements prevent catastrophic losses during unexpected volatility surges while minimizing unnecessary hedging costs during stable periods. These trade-level enhancements directly drive OPHR's ' superior overall performance metrics in Table [ including higher total returns and significantly improved risk-adjusted metrics.
To further illustrate OPHR's ' trading strategies, we visualize representative trade examples in Figure 3 In long I' position, OPHR actively hedges to capture I and Vega gains, which typically outweigh theta costs, leading to positive returns. In contrast, the short I' position relies on theta accrual and benefits from declining implied volatility, while I' risk is managed through reduced hedging frequency. In both cases, the strategy balances risk and reward to achieve a net profit.
Table 4: Transaction costs percentage of total PnL of OPHR
Strategy Options Cost (%)
Underlying Cost (%) Total Cost (%)
BTC 4.15%
5.21%
9.36%
ETH
2.97%
2.78%
5.75%
Transaction Cost Analysis. OPHR's profitability remains robust after accounting for transaction costs. Table T shows that total transaction costs represent 9.36% and 5.75% of P&L for BTC and ETH respectively. We incorporate realistic commission fees following Deribit's fee structure: 0.05% for perpetual futures and 0.03% for options (capped at 12.5% of option price), with bid-ask spread costs implicitly captured through market order execution prices in backtesting.
These modest cost ratios result from OPHR's measured trading frequency, with average holding periods of 9-51 hours (Table that minimize unnecessary transactions. The HR-Agent's sophisticated hedging decisions further reduce excessive rebalancing during stable periods while maintaining effective risk management during volatility spikes. This cost efficiency confirms that OPHR's 's superior performance reflects genuine alpha generation rather than unrealistic cost assumptions.
Conclusion
In this paper, we introduced OPHR, a novel reinforcement learning framework for volatility trading through options. Our OP-Agent excels at volatility timing by dynamically identifying market regimes where implied volatility is mispriced relative to expected RV. Complementing this, our HR-Agent optimizes performance by selecting appropriate hedging strategies that lock in profits during volatility spikes while minimizing costs during calmer markets. Empirical results on cryptocurrency options markets demonstrate that OPHR consistently outperforms traditional approaches across key metrics, validating that our multi-agent reinforcement learning framework effectively captures the complex patterns essential for successful volatility trading. Future work could explore additional volatility trading opportunities, such as volatility smile skewness and term structure anomalies, well as further optimizing hedgers to incorporate market features for more precise hedging decisions. These enhancements would enable even more sophisticated volatility trading strategies while maintaining the data dataen adaptive nature of our reinforcement learning framework.
Acknowledgements
This research supported by the National Research Foundation Singapore and DSO National Laboratories under the Al Singapore Programme (AISGAward No: AISG2-GGC2023-0009.
10

---

References
G. Andersen, T. Bollerslev, X. Diebold, and P. Labys. Modeling and Forecasting Realized Volatility. Econometrica, 2003.
Black and M. Scholes. The Pricing Options and Corporate Liabilities, Journal of Political Economy, 1973.
Tim Bollerslev. Generalized autoregressive conditional heteroskedasticity, Journal of econometrics, 1986.
A. Briola, J. Turiel, R. Marcaccioli, and T. Aste. Deep Reinforcement Learning for Active High Frequency Trading. arXiv preprint arXiv:2101.07107, 2021.
H. Buehler, L. Gonon, Teichmann, Wood, B. Mohan, and Kochems. Deep Hedging: Hedging Derivatives Under Generic Market Frictions Using Reinforcement Learning. Technical report, Swiss Finance Institute, 2019.
H. Buehler, P. Murray, M. S. Pakkanen, and B. Wood. Deep Hedging: Learning To Remove The Drift Under Trading Frictions With Minimal Equivalent Near-Martingale Measures. arXiv preprint arXiv:2111.0788 2021a.
H. Buehler, P. Murray, and B. Wood. Deep Bellman Hedging. arXiv preprint arXiv:2207.00932, 2022.
Hans Buchler, Phillip Murray, Mikko Pakkanen, and Ben Wood. Deep hedging: Learning risk- neutral implied volatility dynamics, arXiv preprint arXiv: 103. /948, 2021
V. Buterin and Others. Ethereum White Paper. GitHub repository. 2013.
L. Canina and S. Figlewski. The Informational Content of Implied Volatility. The Review of Financial Studies, 1993.
Carr and D. Madan. Towards A Theory of Volatility Trading. Volatility: New estimation techniques for pricing derivatives, 1998.
B. J. Christensen and N. R. Prabhala. The Relation Between Implied and Realized Volatility. Journal of financial economics, 1998.
L. Conegundes and A. C. M. Pereira. Beating The Stock Market with Deep Reinforcement Learning Day Trading System. In 2020 International Joint Conference Neural Networks (IJCNN), 2020.
A. V. Dorogush, V. Ershov, and A. Gulin. CatBoost: Gradient Boosting with Categorical Features Support. arXiv preprint 1.Xxv:8110.113666 2018.
B. Dumas, J. Fleming, and R. E. Whaley. Implied Volatility Functions: Empirical Tests. The Journal of Finance, 1998.
H. Van Hasselt, A. Guez, and D. Silver. Deep Reinforcement Learning with Double Q-Learning, In Proceedings of the AAAI Conference on Artificial Intelligence, 2016.
L. Heston, C. Jones, M. Khorram, S. Li, and Mo. Option Momentum. The Journal of Finance, 2023.
S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural computation, 1997.
Hull and A. White. Optimal Delta Hedging for Options. Journal of Banking & Finance, 2017.
Z. Jiang and J. Liang. Cryptocurrency Portfolio Management with Deep Reinforcement Learning. In 2017 Intelligett Systems Conference (IntelliSys), 2017.
T. Kariya and R. Y. Liu. Options, Futures and Other Derivatives. Asset Pricing: Discrete Time Approach, 2003.
Y. LeCun, Y. Bengio, and G. Hinton. Deep Learning, nature, 2015.

---

B. Lim, S. Zohren, and S. Roberts. Enhancing Time-Series Momentum Strategies using Deep Neural Networks. The Journal of Financial Data Science, 2019.
M. McAleer and M. C. Medeiros. Realized Volatility: A Review. Econometric Reviews, 2008
R. C. Merton. Theory of Rational Option Pricing. The Bell Journal of Economics and Management Science, 1973.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, and Others. Human-Level Control through Deep Reinforce- ment Learning. nature, 2015.
Fernando Moreno Pino and Stefan Zohren. Deepvol: Volatility forecasting from high- gigh-freuueccy data with dilated causal convolutions. Quantitative Finance, 2024.
J. Moskowitz, Y. H. Ooi, and L. Pedersen. Time Series Momentum. Journal of Financial Economics, 2012.
P. Murray, B. Wood, H. Buchler, M. Wiese, and M. Pakkanen. Deep Hedging: Continuous Reinforce- ment Learning for Hedging of General Portfolios across Multiple Risk Aversions. In Proceedings of the Third ACM International Conference on Al in Finance, 2022.
Nagy, J.-P. Calliess, and S. Zohren. Asynchronous Deep Double Duelling Q-Learnin for Trading- Signal Execution in Limit Order Book Markets. arXiv preprint arXi::2301.0888 08868888.888,,8 2023.
S. Nakamoto. Bitcoin Whitepaper. URL: https://bitconn. org/bitcoin. pdf-(: 17.07. 2019), 2008.
S. Ni, J. Pan, and A. M. Poteshman. Volatility Information Trading in The Option Market. The Journal of Finance, 2008.
J. M. Poterba and L. H. Summers. Mean Reversion in Stock Prices: Evidence and Implications. Journal of Financial Economics, 1988.
M. Qin, S. Sun, W. Zhang, H. Xia, X. Wang, and B. An. Earnhft: Efficient Hierarchical Reinforcement Learning for High Frequency Trading. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
E. Sinclair. Volatility Trading. John Wiley & Sons, 2013.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT press Cambridge, 1998.
L. A. Takara, A. A. P. Santos, V. C. Mariani, and L. dos S. Coelho. Deep Reinforcement Learning Applied A Sparse-Reward Trading Environment with Intraday Data. Available at SSRN 4411 2023.
W. L. Tan, S. Roberts, and Zohren. Spatio-temporal momentum: Jointly learning time-series and cross-sectional strategies. The Journal of Financial Data Science, 2023.
W. L. Tan, S. Roberts, and S. Zohren. Deep Learning for Options Trading: An End-To Ti--Edo- Approach. In Proceedings of the 5th ACM International Conference on Al in Finance, 2024.
H. Y. Wong and Y. W. Lo. Option Pricing with Mean Reversion and Stochastic Volatility. European Journal of Operational Research, 2009.
K. Wood, Roberts, and Zohren. Slow Momentum with Fast Reversion: A Trading Strategy using Deep Learning and Changepoint Detection. The Journal of Financial Data Science, 2022.
K. Wood, S. Giegerich, S. Roberts, and S. Zohren. Trading with The Momentum Transformer: An Intelligent and Interpretable Architecture, 2023.
H. Yang, X.-Y. Liu, S. Zhong, and A. Walid. Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy. In Proceedings of the First ACM International Conference on Al in Finance, 2020.
C. Zong, C. Wang, M. Qin, L. Feng, X. Wang, and B. An. MacroHFT: Memory Augmented Context- aware Reinforcement Learning On High Frequency Trading. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2024.
12

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency. research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we provide a clear contribution list in introduction which aligns our paper and abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
13

---

Justification: As we mentioned in conclusion, this work is only specific volatility trading strategy. More possibilities are still to be study with RL.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper is about RL in financial applications, no theoretical analysis is needed.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experiment details are presented in the paper and the Appendix.
Guidelines:
14

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We open-source the code and example data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https//nips.cc/public guides/CooeSubbmssioooiiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (htips: hecpppbbilgguiide//ooeeeeuiiiieu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
15

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e-g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are presented the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: The statistical significance not applicable for quantitative trading research, because we only experienced one history and cannot access to other parallel universe.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: Yes]]
Justification: They are presented in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
16

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:://euuii.ccc//lll///ttitcue
Answer: [Yes]
Justification: We do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: They discussed in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Comprehensive test needed before bringing this strategy to real trading. Guidelines:
The answer NA means that the paper poses such risks.
17

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e-g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
We do not use any of them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com cotphiaaeeecmm/atttttttte has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not open-source our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human involved in this research.
18

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human are involved in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: This research is not related LLM, and we only use LLM to polish the writing this paper.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htttss//oeuunssccccoootteeeessss for what should or should not be described.
19

---

A Cryptocurrency Options
A.1
The Black-Scholes-Merton Model and Greeks
The BaackSccoles-Merton (BSM) Model is a seminal framework for pricing European-style options. Despite the fact that real-world markets often violate its assumptions, the model offers fundamental insights and practical tools for derivatives pricing and risk management.
A.1.1
Model Assumptions
The BSM model relies on several idealized assumptions:
Markets are frictionless: there transaction costs taxes.
Trading is continuous, and investors can borrow and lend at constant, risk-free interest rate 7.
The price of the underlying asset S; follows geometric Brownian motion, governed by the stochastic differential equation:
dSe pSSdt S,dWe,
where:
14 is the expected return of the asset,
is the volatility of the asset,
We a standard Wiener process.
There are arbitrage opportunities.
The option is European.
A.1.2 Black-Scholes Pricing Formula
Under these assumptions, the value (tt, of a European option satisfies the Black-Scholes Partial Differential Equation (PDE):
QV Ōt
RV as?
OV rV 0.
o² 5
+r.
as.
Solving this PDE with appropriate boundary conditions yields closed-form pricing formulas for European call and put options:
C(1, (di( - Ke- ((T-
(6)
St), = Ke -r(--) N(222) - S,N(-d1).
(7)
where
di In(S//K) + (r 102) (T - t)
(8)
oVT
d2 -----ovT-
and N-) is the cumulative distribution function of the standard normal distribution.
A.1.3 Greeks
The BSM model also provides a set of sensitivity measures known as the Greeks, which quantify how the option price responds to changes in key variables:
Delta: Measures sensitivity to the underlying price.
0V
A
as
A N(d),
Apu = N(d)) - 1.
It is essential for directional risk management.
20

---

Gamma: Measures the rate of change in Delta with respect to the underlying asset.
r 0.v
05
r (d1)
S
where (d1) = 1 2/22// is the standard normal probability density, High Gamma
V2= V2
indicates that Delta changes rapidly with small movements in S..
Theta: Measures the effect of time decay on option value.
av
+ Ōt
Gull (d)) Ke (T (d))))
2VT (,N'
2 KreeerlT-- (T-) N(-d2)
put
It is usually negative for long options, reflecting value loss over time. Vega: Measures sensitivity to volatility.
Ov
Vega
Oo
Vega (d)
Rho: Measures sensitivity to interest rate changes.
Peall K(T - t)e -1) ( N(d2),
Ppu K(T l)e -T( N(-d2)-
A.2 Inverse Option
Inverse options are European-styy style, cash-settled option contracts denominated in the underlying cryptocurrency (e.g., BTC ETH), representing the dominant format in contemporary crypto derivatives markets. These contracts can only exercised automatically at expiry, climinating the possibility of early exercise. Instead of requiring the physical delivery of the underlying asset, settlement occurs in the form of a payout denominated in the base cryptocurrency. calculated as the difference between the delivery price and the strike price. Option pricing based the Black-Scholes framework, wherein the forward price, rather than the spot index, is employed as the principal input for valuation. Trading is generally conducted range of order types. Furthermore, the system incorporates risk management measures, including price correction mechanisms, to enhance market integrity and operational resilience. Inverse options are particularly well-suited for market participants seeking to hedge or speculate on crypto-asset exposures without engaging spot or fiat transactions.
continuous 24/7 basis and accommodates diverse
A.3 Portfolio Margin
Traditional margin systems typically apply fixed margin requirements to individual positions, without accounting for the potential offsetting risk of multi-leg strategies. For instance, in short straddle positions-when where both a call and a put option are sold with the same strike price and expiration the worst-case loss is expected to occur in only one direction of price movement, not both simultancously. Charging full margin on each leg independently leads to a substantial overestimation portfolio risk, thereby reducing capital efficiency.
To address this inefficiency, the Portfolio Margin (PM) framework adopts risk-based approach that estimates the required margin based on the portfolio's 's net risk exposure. Instead of evaluating each position in isolation, PM performs a series of stress tests on the entire portfolio, simulating potential profit and loss (P&L) under a range of underlying asset price and implied volatility scenarios. The maximum projected loss from these scenarios is then used as the basis for the maintenance margin requirement.
21

---

Calculation Procedure. The PM system simulates the portfolio P&L under multiple discrete asset price shifts (e.g.. 15%; 12%, .... 0%..... +15%) and volatility adjustments (e.g., +45%, 0%, -30%). For each price scenario X%, the total portfolio P&I. is computed as:
((x%) - A(X + B(%%)
where A(X%) represents the P&I of futures or perpetual contracts, and ((%%) is the worst-case loss of the option positions under the three volatility assumptions the given price level. The **maximum potential loss (ML)*$ is determined as:
ML min ((%%) for all scenarios}
An additional contingency margin is added account for uncovered short options and large directional exposure:
Contingency =
Σ
Net Short Options at Strike x a%) + (|uutures Position x 8%) x Spoo Price
Finally, the total maintenance margin is given by:
Maintenance Margin M Contingency
This methodology ensures that the margin reflects the true tail risk of the portfolio while improving capital efficiency for hedged and risk-reducing strategies.
Margin Control in Simulation Environment In experiments, the simulated trading environ- ment adopts Portfolio Margin (PM) framework to estimate margin requirements. For all conducted trading episodes, the required margin constrained remain below fixed threshold of 50% of the agent's 's total cash holdings. This constraint ensures the feasibility and solvency of all trading actions throughout the simulation, preventing excessive leverage and guaranteeing compliance with realistic capital limitations.
B
Method
B.1
Oracle OP-policy Design
As introduced in 4.1 the Oracle OP- OP-policy, denoted as "'gacee is designed to provide structured cetemaatioonssccfiiccai
supervision for reinforcement learning agents by leveraging privileged future RV. This section presents the detailed implementation of the policy.
The strategy generates trading signals by comparing the future RV with the current IV, Fimps subject to predefined tolerance threshold B. Formally, the policy operates as follows:
If RV 2 iimp (1 B), then initiate a long position.
If RRViuture < Timm (1 B), then initiate a short position.
Otherwise, maintain neutral position.
In practice, the Oracle OP-policy serves as an oracle demonstrator by generating high-quality sequential action-value trajectories through direct interaction with the environment. When filling the experience buffer, the Oracle OP-policy utilizes future RV information future, across multiple time horizons, along with varying tolerance thresholds 3e, construct diverse set of oracle strategies. This diversity facilitates better coverage of the policy space and enables the OP agent to approximate near-optimal behavior more effectively during the early stages of training.
While inherently sub-optimal due to the unrealistic assumption of future information availability, this rule-based strategy aligns with desirable trading behaviors and effectively serves structured demonstration policy. By doing so, it reduces the exploration space and helps the learning agent to converge rapidly toward profitable regions of the policy space. Hedging decisions under this policy are executed using a predefined baseline Hedger Hedger-
It is important to note that, although the Oracle OP-policy utilizes future RV from the training set-a type of privileged information unavailable during deployment- this signal used solely to generate offline trajectories for training purposes. These trajectories serve high-quality demonstrations to
22

---

guide the learning process. Thus, no future information is exposed to the agent during inference, ensuring the integrity of forward-looking decision-making, As such, the policy improves sample efficiency and accelerates convergence without compromising generalization to out-of-sample, real- world environments.
Fundamentally, the Oracle OP policy functions as a proxy for an optimal strategy based on historical data, providing structured supervision in the form offline expert demonstrations. Similar methods in imitation and offline reinforcement learning, its purpose is to aid training rather than serve as deployable policy. Since all supervision derives from past data, the approach remains consistent with realistic deployment constraints.
B.2 Hedgers
To hedge the A risk of the position and to lock in I profits, employ a series of hedging instruments. These Hedgers are constructed using various methods to adapt to different market conditions. We simulate the use of three types of hedging instruments under different market regimes and position directions. Based on the overall return and risk exposure, we select the optimal Hedgers to construct Hedgers pool, which serves the candidate set for the HR-agent's hedging decisions.
Delta-based Hedger: This type of Hedger determines whether to hedge based on the A value of the current position. A decision threshold Aphres is predefined. When the absolute A of the position exceeds this threshold,
A
A thress
a full Delta hedge is executed to neutralize the position's A exposure.
Price-based Hedger: This Hedger triggers hedge based on significant price movements. Let Pt denote the current underlying price and Plst hodge the underlying price at the last hedge. threshold Pheres used determine whether a new hedge is necessary. When the relative price change satisfies
Se
Sthees
Slast hodge
a full hedge is performed to adjust the position accordingly.
Deep Hedgers: This class Hedgers leverages deep reinforcement learning to learn adaptive hedging strategies in a data-driven manner. Unlike rule-based Hedgers, deep Hedgers dynamically determine hedging actions by interacting with the environment, optimizing for risk-adjusted objective. Specifically, we use the Actor-Critic algorithm to train these agents under various market regimes and position profiles.
Each Deep Hedger receives the current market state as input, which includes the underlying price, option Greeks, historical price volatility, and the current position. The output is continuous hedging action between 0 and I, representing the proportion of A exposure to hedge.
The training objective balances hedging cost and residual portfolio risk through utility- based loss:
Lhddge Rs +
log TEle-AAAy -AX,],
where R2 is the immediate hedging reward (typically negative cost), X, is the wealth with post-hedging residual risk, and the risk-aversion parameter. larger encourages conservative (risk-sensitive) behavior. By training under different simulated market conditions and position directions, Deep Hedgers learn to generalize across regimes and exhibit context-award hedging behavior.
To enhance the HR-agent's hedging effectiveness across di- verse market regimes, we construct a comprehensive Hedger pool composed of the three aforementioned types of Hedgers: Delta-based Hedgers, Price-based Hedgers, and Deep Hedgers.
Hedger Pool Construction:
We generate a diverse set Delta-based and Price-based Hedgers by varying their respective decision thresholds. Mcanwhile, Deep Hedgers are trained under range of simulated market
23

---

environments, position direction, and risk-aversion parameters A. The resulting collection of Hedgers is then evaluated via backtesting across segmented market regimes-caleeorrizee categorized by volatility levels and directional dynamics (e.g., trending vs. mean-reverting)- and position types (e.g., long or short T).
To ensure diversity in hedging behaviors and risk exposures, we retain only the top-k performing Hedgers that achieve favorable trade-off between hedging cost and risk control, primarily in terms of A exposure. These selected Hedgers collectively form the candidate Hedger pool, which serves the basis choices for dynamic Hedger selection by the HR- agent within the full framework. This enables the construction of diverse and efficient hedging strategies adapt varying market conditions.
B.3 Joing Training
Algorithm 3: OPHR Joint Training
Require: Oracle OP-Policy Toructe: Baseline Hedger HHddgr Env and Twin Env
Initialize replay buffer RTP, Rhr Q-networks Qo Qu and target Q-networks Qør, Qui
2: Phase I: OP-Agent Offline Pretraining with Oracle OP-Policy and Baseline Hedger
oP 3: Collect trajectories using "Orale and o Hoüger store transitions in Rop
4: for Offline Training Iterations do
5: Sample a batch transitions from RoP and update á with n steep TD error Eq.
6: Phase 2: Joint Online Learning
7: for Joint Training Epochs do
8: Fix Qs and train HR-Agent Qs using Algorithm for N episodes
9: Fix Q and train OP-Agent Qs. using Algorithm for 'P episodes
10: return Qs. Q
C Experiment
C.1 Evaluation Metrics.
We evaluate our method using 8 financial metrics: 1 profit, 3 risk-adjusted profit, 2 risk and 2 trade criteria. Returns are aggregated into day-level before calculation.
Total Return (TR) is the overall return rate of the period, defined as TR Y- where V, is the final margin balance and Vi is the initial margin balance.
Vi
Annual Volatility (AVOL) is the the standard deviation of daily returns annualized defined
as o [rett] x y to measure the volatility risk, where ret (rel,, rel2, red,) a vector of daily return, af is the standard deviation function, and m is the annualization factor 365.
Maximum Drawdown (MDD) measures the largest loss from any peak to show the down- side risk of the strategy.
Annual Sharpe Ratio (ASR) is the profit adjusted by volatility risk, defined as: SR
Elreti drrot]
x /m, where E[ret] is the expectation of daily return.
Annual Calmar Ratio (ACR) defined as ACR
F(ret] MDD
x m, measuring profit adjusted
by downside risk.
Annual Sortino Ratio (ASoR) applies downside deviation as the risk measure. is defined
as: ASoR E(ret]x / where downside deviation(DD) is the standard deviation of the DD
negative daily return rates.
Win Rate (WR) is the percentage of trades that result in profit, defined WR
Nuin. x 100%, where Nuci the number of profitable trades and Nistal is the total trades.
Niotal
Profit/Loss Ratio (PLR) measures the average profit of winning trades relative to the [E[retwin]] where E[rctwinl is the
average loss of losing trades, defined as PLR = [
average return of profitable trades and E E'rettoss] is the average return of losing trades.
24

---

EN'cptal hy
Holding Period (HP) is the average holding time per trade, defined as HP= where h; is the holding period of the i-th trade and Ntotal is the total number of trades. It reflects the trading frequency.
Nistad
C.2 Hyperparameters
In Section 4.1 we introduce the N-step remporal-differenee Double DQN algorithm applied within rolling training framework. The main hyperparameter settings for this algorithm summarized in Table In our experiments, we adopt rolling training of every 10 days and apply the proposed algorithm historical data, and the network is updated using 12-step temporal-difference (TD) learning.
Table 5: Hyperparameters for OP-Agent Training
Parameter
Description
Value
Training window size
Training window size
10
Oracle future RV
Time horizons for Oracle future RV
(3, 6, 9, 12, 24)
Oracle threshold /
Thresholds for oracle policy
(0.1, 0.2, 0.4, 0.6, 0.8) 0.1
Oracle exploration coracle Episodes
Oracle exploration rate
Number of episodes per training window
20,000
Updates per episode
Network updates after each episode
20
Batch size
Number of samples per training batch
512
Hidden layer size Learning rate
Maximum hidden layer size in OP-Agent
1024
Step size for gradient updates
x 10
N-step TD n
Steps used in n-step TD learning
Discount factor 7
Reward discount factor
0.99
Soft update coefficient т
Rate for target network soft updates
0.005
Target update frequency
Frequency of target network updates
10
Dropout rate
Dropout rate in network layers Initial exploration rate Final exploration rate Decay rate of (steps)
0.2
Estat
0.9
fend
0.01
<decay
10,000
As described in Section B.2 we utilize three types of Hedgers, each trained under different market regimes and position conditions. For each setting, multiple Hedgers trained, and the top 30 performing ones are selected based on validation performance. These selected models serve as candidate Hedgers for the HR-agent. The hyperparameter settings for the Hedgers are summarized in Table 6 while the preliminary training parameters for the HR- HR-agent are listed in Table
Table 6: Hedger parameters.
Hedger type
Parameter values
Delta-based Hedger
0, 0.01, 0,02, 0.04, 0.06, 0.08, 0.1, 0.12, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9
Price-based Hedger
0.5%, I %, 2%, 3%, 4%, 5%, 7%, 10%, 12%, 15%,
20%. 30%, 50%, 100%
Risk Aversion parameter À
0.1, 0.2, 0.5, 0.7, 2, 3, 5, 10, 20, 40, 60, 80, 100, 200, 400, 800
Hedger pool size
30
25

---

Table Hyperparameters for HR-Agent Training
Parameter
Description
Value
Learning rate
Step size for Q-network updates
1 x 10-4
Discount factor 7 Batch size
Future reward discount factor
0.99
Number of samples per update
512
Hidden layer size
Size of each hidden layer in the Q-network
1024
Estart
Initial exploration probability
0.9
Gend
Final exploration probability
0.01
(decay
Steps over which decays linearly
10,000
Updates per step
Number of Q-network updates per routing step
10
Routing interval nhr
Steps between Hedger selections
24
Training Setup. We conducted all experiments a server equipped with 4 NVIDIA RTX 4090 GPUs and an AMD Ryzen Threadripper PRO 5995WX CPU. The total time required for one iteration of stage I and stage 3 was approximately 12 hours for the OP-Agent optimization in Section and around 3 hours for training the HR-agent in Section 4.2
C.3 Baselines
This section presents the detailed implementation of the baselines introduced in Section 5.1 All baseline strategies adhere to the following fundamental principles:
Open Position: At the time of initiation, the strategy selects the nearest-totthe-money (ATM) straddle within predefined maturity range the underlying instrument. The notional size of the position is determined based on the allocated margin usage.
Depending on the nature of the signal, strong reversal signal may indicate either a position reversal or position closure. Alternatively, for certain signals, take-profit and stop-loss rule applied: the position is closed if the profit reaches p% if the loss reaches 1%. In addition, maximum holding period HP Paax is imposed; if neither take-proni proeit nor stop-loss conditions are met before H max. the position is force-closed upon reaching this time limit.
Close Position:
Hedger: For consistency across all baselines, apply unified Delta-based hedging scheme as described in Section B.2 where the hedging threshold set to Atres 0.1.
C.3.1 Directional Volatility Strategies
This strategy involves taking directional exposure to volatility by trading at-the-money (AT') straddles. Specifically, straddles with time to maturity between miin and mmax are selected, and each position is held until fixed rollover point Mylloove- At each rollover date, existing positions closed, and new positions are established following the same maturity selection criteria.
Two directional variants are considered:
Long: Buys ATM straddles that meet the maturity condition and holds the position until the rollover date.
Short: Sells ATM straddles under the same maturity condition and holds the position until the rollover date.
A unified rule-based Delta Hedger is applied to this strategy.
All parameters used in this strategy summarized in Table 8
C.3.2 Single Factor Models
Both the Mean Reversion (MR) and Momentum (MOM) strategies are constructed based on single indicator derived from the percentile of realized volatility (RV).
26

---

Table Parameter settings for Long and Short Volatility strategies
Parameter
Long Volatility
Short Volatility
M'max
90
90
miin
60
60
Myollover
21
21
thres
0.1
0.1
For each evaluation date, the model computes the annualized realized volatility over a backward-looking window of length Pool, denoted by op. This RV op is then compared against the historical distribution of RV computed over rolling lookback window of length Plookback. yielding percentile score.
RV Percentile
Based this percentile:
Mean Reversion (MR): Takes a short volatility position when the current Geal percentile is high (expecting reversion), and long volatility position when it is low.
Momentum (MOM): Takes long volatility position when the percentile momentum is high (expecting continuation), and a short volatility position when low. Specifically, the momentum of the percentile score is defined the relative change over lag of i periods:
Percentile,__.
-1)
Mopereentile,s =
x 100%
(10)
Percentile;--:
Positions are opened by selecting ut-the-money (ATM) straddles with maturities between manin and Mmax, conditional on the factor signal.
Position closure follows the general rule-based principles described in Section 5.1
unified rule-based Delta Hedger applied to this strategy.
All parameters used in this strategy are summarized in Table
Table 9: Parameter settings for Single Factor Models (Mean Reversion and Momentum)
Parameter
MR
MOM
Poul (window for realized volatility)
3, 6, 9, 12, 24 hours
Plaukback (lookback window for percentile)
365 days
Mmins Mmax (option maturity range)
60, 90
Attres (Delta hedging threshold)
0.1
(take-profit threshold)
5%
(stop-loss threshold)
3%
H Pmax (maximum holding period)
96 hours
SMR.long, SMR (MR threshold)
20, 80
SMOMlong SMOMshon (MOM threshold)
15%, -00%
C.3.3 Machine Learning Models
We implement three supervised learning models to generate volatility forecasts: Gradient Boosted Decision Trees (GBDT), Multi-Layer Perceptron (MLP), and Long Short-Term Memory networks (LSTM).
Each model is trained to predict the annualized RV over forward-looking window of length Post which serves as the target variable. Input features include historical volatility measures, option market variables, and macroeconomic indicators characterizing the current market environment.
The predicted volatility then compared to the current implied volatility (IV) the ATM straddle to form a relative signal. The corresponding thresholds are selected based on backtesting, using values that yield the best performance on the training and validation Sets.A rule-based decision rule applied based their ratio:
27

---

A long straddle position is opened if the predicted volatility exceeds the current implied
volatility by ratio greater than lon. i.e.,
Fpedd
SML.long-
A short straddle position is opened the predicted volatility sufficiently below the current implied volatility, i.c., IV
Spest
SML.sbon.
Positions are opened using at-the-money (ATM) straddles with maturities between Miin and max
Position closure follows the same rule-based framework described in Section 5.1
Take-profic is triggered when return exceeds 5%;
Stop-loss is triggered when loss exceeds 3%;
Maximum holding period: HPmmx 96 hours;
If predicted volatility crosses the opposite threshold, the position is closed and reversed.
Delta risk is managed using the unified rule-based Delta hedging mechanism, with threshold Aftres 0.1.
All parameters for the machine learning strategies are summarized in Table 10
Table 10: Parameter settings for Machine Learning Models
Parameter
GBDT
MLL
LSTM
Poul
(3, 6, 9, 12, 24)hours
mmin mmaa
60, 90
hres
0.1
p
5%
[
3%
H Pamxx
96 hours
SML.long: SMLshort
1.6, 0.7
1.2, 0.9
1.3,
CatBoost iterations=3000 depth=8 target='ic'
MLP
LSTM
max hidden 1024
layers=3
Model architecture
depth=5
max hidden dim=512
ReLU, dropout=0.2
dropout=0.2
D More detailed Result Analysis
D.1 Baseline Comparison
The visualization of the baseline comparison, summarized in Table 2 is presented in Figures 4 and
D.2 ML prediction
Table 11: Performance ML Forecasting Models
BTC
ETH
Model
Spearman IC
Pearson IC
R²
Spearman IC
Pearson IC
R²
LSTM
0.6133
0.6136
0.5776
0.5946
0.6192
0.5846
MLP
0.6030
0.4303
0.3353
0.5801
0.3306
0.5023
CaBBoost(GBDT)
0.6214
0.6279
0.5781
0.6056
0.6152
0.5407
Table 11
presents the volatility forecasting performance of the baseline machine learning models. From the perspective of information coefficients (IC), all models generally demonstrate reasonably good predictive ability.
28

---

Strategy Return Comparison
Return (%)
LONG
-20
MLP
LSTM
-40
GDBT
MR
60
MOM
OP
OPHR
2023-07
2023-09
2023-11
2024-01
2024-03
2024-05
2024-07
Time
Figure 4: Baseline Comparison on BTC
Strategy Return Comparison
40
Return (%)
-20
LONG
MLP
440
LSTM
MR
-60
OP
OPHR
2023-07
2023-09
2023-11
2024-01
2024-03
2024-05
2024-07
Figure 5: Baseline Comparison on ETH
Table 12: Performance of ML Forecasting Models on Top 5% Outliers
BTC
ETH
Model
Spearman IC
Pearson IC
R²
Spearman IC
Pearson IC
R²
LSTM
0.3519
0.4460
0.1459
0.3462
0.5041
0.3363
MLP
0.3770
0.1903
-0.8157
0.4101
0.4018
-0.0536
(GBD))
0.3902
0.4746
0.2557
0.3664
0.5083
0.3578
Table 13: Performance of ML Forecasting Models 95% Inliers
BTC
ETH
Model
Spearman IC
Pearson IC
R²2
Spearman IC
Pearson IC
n22
LSTM
0.6207
0.6619
0.6695
0.6032
0.6599
0.6483
MLPP
0.6172
0.6633
0.6378
0.5894
0.6531
0.6050
(GBD)))tGGBDT)
0.6285
0.6700
0.6268
0.6158
0.6599
0.5783
29

---

However, these models fail to accurately capture the sharp spikes in realized volatility (RV), which are critical for long Gamma trading strategies. Table 12 shows that model performance drops noticeably on the top 5% outliers, reflecting limited robustness under extreme conditions. Moreover, there remains substantial gap between predicting volatility and executing profitable trading decisions based on that prediction. Addressing this gap is precisely where our proposed method offers distinct advantage.
D.3 PNL Decomposition
To understand the sources of profit and loss (PnL. option trading strategies, we decompose the cumulative PnL into analytically interpretable components based on the Greeks: Delta (A), Gamma (T), Theta (θ), Vega, and residual term capturing unexplained effects. Let I,.... denote discrete trading timestamps. The total PnL at time / is approximated as:
PnL. Delta, + Theta, + Vega, Gamma; Residual,
(11)
Each component corresponds a specific source of risk exposure and computed as follows:
Delta PnI.. The contribution from changes in the underlying price S, is estimated by:
Delta, = A (S: - S
(12)
Here, 4,-1 sensitivity of the option's 's value to movements the underlying.
@V 85 is the Delta exposure at the previous time step. This term captures the linear
Theta PnI.. The time decay of the option value is given by:
Theta; 0,-1 At
(13)
where 0,-1 8V is the time sensitivity, and At is the time step.
Vega PnL. The impact of changes in implied volatility o is estimated as:
Vega, = Vega,-1 (a: - d
(14)
where Vega,-1
0V Do
is the option's sensitivity implied volatility.
Gamma PnL (Realized). Gamma measures the convexity of the option value with respect the underlying. Between two hedge timestamps frre and feurr, the cumulative realized Gamma PnL is:
teur
Gammaa realized Baur
1. .r. (S. (S,-)
(15)
t=
where r, = [T,, (T-1) is the average I exposure at time t.
Gamma PnL (Unrealized). To monitor latent convexity risk, the unrealized Gamma PnL from the last hedge point to current time t tracked as:
Gamma unrealized
Fr. (Se. S
(16)
faus bedue
This quantity is not included in total PnL but helps visualize the risk of delayed hedging.
30

---

Residual PnL. The residual term accounts for the portion of PnL not explained by the above Greeks:
Residuale PnI., (Delta, Theta, Vega, Gamma.)
(17)
Residual PnL may arise from several sources: model misspecification, transaction costs, bid-ask spreads, slippage during rebalaneing, inaccurate Greek estimates, discrete hedging errors. high residual may indicate imperfect model assumptions or unmodeled market factors.
Table 14: PnL attribution (%) for OPHR on BTC and ETH
Component
BTC Long
BTC Short
ETH Long
ETH Short
Delta
1.17
-0.55
-5.26
-23.50
Gamma
91.41
-169.15
107.60
-125,70
Theta
-5.71
66.15
-4.99
44.23
Vega
64.58
123.43
32.01
51.93
Residual
-51,45
80.12
-29.36
153.03
As shown in Table [4| we decompose the PnL of OPHR's ' long and short I' trading strategies into Greeks PNL and express each percentage of the total profit. Despite the presence of sizable residual terms in the attribution, the analysis reveals that the primary sources of profit in the long I strategy stem from and Vega exposure, while the main cost associated with Theta decay. In contrast, the short I' strategy benefits predominantly from Theta and Vega gains, while incurring substantial losses from adverse I' exposure. These observations align well with the intended design of the respective strategies.
E Broader Impact
Our reinforcement learning framework for options trading offers potential positive impacts through improved market liquidity, more accurate risk pricing, and democratization of sophisticated trading strategies. However, we acknowledge potential negative consequences including market concen- tration that could exacerbate inequality, possible herding behavior leading to market instability if widely adopted, and environmental concerns related to cryptocurrency markets where our method is tested. We emphasize the importance of responsible deployment to ensure algorithmic advances in financial markets benefit society broadly rather than concentrating advantages among a small group of sophisticated participants.
31


Reviews:
[{"review_id": "GNSXA9BCo6", "reviewer": "Reviewer_KnBe", "summary": "In this paper, the authors study the volatility trading task in the option markets. A multi-agent framework is proposed, which consists of two types of agents, OP-agent and HR-agent, respectively. Crypto option data was used in the experiments for verifying the effectiveness.", "questions": "Plz provide justifications to the above questions.\n\nAs mentioned before, this reviewer does not have solid expertise to evaluate the reported results. One extra question for clarification is: should the TWAP and VWAP be considered as baseline methods for this work? If yes, plz add those results as baselines too.", "limitations": "The reviewer is not aware of such concerns.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The strengths can be summarized as follows:\n1. The volatility trading task in the option markets is fundamentally important. It is a critical task in the finance sector. The reviewer believes that it could potentially be very useful in the real-world.\n2. It is sound to recast the task as a mdp process and then taking an RL approach. The authors provide rigorous formulation in Section 3.\n3. The two-phase multi-agent framework is sound. It is very interesting to see an agent pool for a real-world scenario.\n4. The experiment results based on real-world data is appreciated. However, this reviewer does not have solid expertise to evaluate the reported results.\n\nHowever, this paper cannot be fully appreciated due to the following weaknesses:\n1. It is NOT clearly presented why RL is the right solution for such a real-world task.  The reviewer is interested to see such an RL solution, however, it is also important to understand why it is the right one?  Is RL a unique choice here?\n2. What is an intuitive understanding of the two-phase approach?  Plz help the reviewer and readers to understand the design from a high-level.\n3. Is the Deep Hedger reproduced in the experiments?\n4. Is it possible to provide any performance guarantee?  Or it can only take an engineering approach?  \n    many RL research at NeurIPS has deep analysis.  It would be good to show the depth of this work.  \n    Is there any unique property for RL algorithms to deal with option trading data or crypto data?", "was_revised": true, "final_justification": "Thanks for the authors' responses. They address my concerns well. And I will keep my scores.", "rebuttal": "Thank you for your valuable comments and positive assessment. Please find our detailed responses below.\n\n### **Why RL is the right solution**\nTraditional approaches that rely on explicit volatility forecasting followed by rule-based trading decisions face fundamental limitations that make RL particularly well-suited for volatility trading:\n\n1. **Minimizing model assumptions**: Traditional methods depend heavily on explicit modeling assumptions - IV calculations rely on option pricing models such as BSM, while RV estimation requires choosing among different variance estimators (maximum likelihood, maximum posterior, different data frequencies, etc.). In practice, selecting the appropriate models and parameters across varying market conditions is extremely challenging and often leads to model misspecification. RL circumvents these issues through data-driven learning that incorporates all market dynamics directly into the reward signal, eliminating the need for multiple separate models to estimate volatility components.\n2. **Sequential decision-making & Path-dependent profit**: Volatility trading presents inherently sequential challenges where current decisions affect future risk exposures and hedging costs/timing - this is not a static optimization problem. As shown in Section 2.2 and Figure 1, optimal hedging decisions depend on complex interactions between market conditions, position Greeks, and future price paths. Current option positions determine future delta exposure, which influences subsequent hedging requirements and costs. In our experiments, we conducted extensive grid search optimization for rule-based strategies but still struggled to find configurations that generate stable profits across different market regimes. Traditional approaches using predictions with manually designed rules are prone to getting trapped in local optima or failing entirely. RL naturally handles these sequential dependencies by learning optimal action sequences that account for future implications of current decisions.\n\n### **Intuition of the two-phase approach**\nOracle policy initialization has been used in trading RL [1,2], but volatility trading presents unique challenges that motivate our two-phase design. The reward structure is highly path-dependent, and profitable trading trajectories are relatively sparse - making it extremely difficult to discover profitable strategies through random exploration alone. Moreover, the few profitable trajectories can be easily overwhelmed by noisy market data, leading to poor convergence.\n\nOur Oracle policy uses delta-hedged P&L signals to guide the agent toward profitable trading behaviors, significantly reducing exploration costs. This initialization ensures the agent starts from a reasonable region of the policy space, then RL training further optimizes beyond this foundation to discover superior strategies that pure exploration would struggle to find amid market noise.\n\n### **Deep Hedger reproduction**\nYes, we reproduce the Deep Hedger following [3]. Appendix B.2 describes the details of different hedgers including Deep Hedgers, and Table 5 shows the parameters used for these hedgers. Our implementation follows the actor-critic framework with risk-adjusted utility functions and varying risk-aversion parameters as described in the original work.\n\n### **Performance Guarantee**\n**Theoretical considerations**: While we don't provide formal convergence guarantees as this is an empirical RL research, in our research practice we found the following elements crucial for performance assurance:\n\n1. **Oracle policy initialization**: The Oracle policy in Phase 1 ensures we start from a profitable region of the policy space, reducing the risk of converging to poor local optima.\n2. **n-step TD learning**: Effectively reduces noise in Q-value estimation, making RL training convergence more stable compared to 1-step methods.\n3. **Lower-frequency HR-Agent switching**: The HR-Agent's less frequent hedger selection (every 24 hours) similarly improves training stability by reducing action space complexity.\n4. **Relative reward design for HR-Agent**: Using the advantage of selected hedgers over baseline hedgers as reward provides more stable learning signals than absolute performance measures.\n\n### **TWAP/VWAP baselines**\nThese execution algorithms are designed for minimizing market impact when trading large orders, which is orthogonal to our volatility trading objective. Our baselines already comprehensively cover the relevant comparison space: directional strategies (Long/Short), factor models (MR/MOM), and ML forecasting approaches (GBDT/MLP/LSTM). TWAP/VWAP would not provide meaningful comparisons for volatility timing and Greek management, as they address fundamentally different trading problems.\n\n### **References:**\n[1] Universal Trading for Order Execution with Oracle Policy Distillation, AAAI 2021\n[2] EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading, AAAI 2024\n[3] Deep Hedging: Continuous Reinforce359 ment Learning for Hedging of General Portfolios across Multiple Risk Aversions, ICAIF, 2022"}, {"review_id": "b7TuACaQxs", "reviewer": "Reviewer_vLxe", "summary": "This paper introduces OPHR, a novel multi-agent deep reinforcement learning framework for volatility trading in options markets. The core idea is to decompose the complex trading task into two specialized roles: an **Option Position Agent (OP-Agent)**, which determines the overall strategy (long or short volatility) by timing the market, and a **Hedger Routing Agent (HR-Agent)**, which dynamically selects the optimal hedging policy from a pool of pre-trained hedgers to manage risk and maximize path-dependent profit. The framework is trained in two phases: an offline initialization phase where the OP-Agent learns from a privileged \"Oracle\" policy, followed by an iterative online phase where both agents are trained collaboratively. The authors evaluate their framework on historical Bitcoin and Ethereum options data from 2021-2024, demonstrating that OPHR significantly outperforms traditional strategies, factor models, and machine learning baselines across a comprehensive set of performance and risk metrics.", "questions": "1.  The baseline ML methods (GBDT, MLP, LSTM) are designed to forecast future realized volatility, and Table 10 shows they achieve respectable forecasting performance (e.g., Pearson IC > 0.6, R² > 0.57 for BTC). However, their trading performance in Table 2 is very poor. The authors correctly state that the ML baselines \"fail to bridge the gap between forecasting RV and optimizing path-dependent PnL outcomes\" (Lines 270-272). Could you elaborate further on why this gap is so large? Is it primarily because a simple forecast-to-trade rule cannot capture the path-dependent effects of hedging and costs, or is it that these models are inherently poor at predicting the rare, fat-tailed volatility events where most of the profit from long-gamma strategies is generated (as suggested in Appendix D.2)? A deeper analysis here would strengthen the paper's core thesis.\n\n2.  Following on from Weakness #2, the reliance on BSM for Greek calculation seems to be a significant model assumption. Have you considered or experimented with more sophisticated methods to obtain more accurate portfolio sensitivities? For example, one could use a model that accounts for the volatility smile (e.g., SABR) or employ model-free Greek estimates. How would the performance of the HR-Agent, and thus the entire OPHR framework, be affected if it were provided with more accurate, market-implied Greeks?\n\n3.  The Oracle policy used for offline pre-training (Phase 1, Line 210) leverages future realized volatility to generate \"desirable and profitable trading behaviours\". This is a clever way to bootstrap the agent. Could you provide some insight into the sensitivity of the final OPHR performance to the quality of this pre-training? For instance, how does the performance change if the Oracle's `β` threshold is defined differently or if the pre-training phase is shortened or skipped entirely?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1.  **Novel and Sensible Problem Decomposition:** The paper's primary strength lies in its intelligent decomposition of the volatility trading problem into a multi-agent system. Separating the strategic decision of \"volatility timing\" (OP-Agent) from the tactical execution of \"risk management\" (HR-Agent) is a clear and logical approach that mirrors how human traders might operate. This hierarchical structure (explicitly formulated as a Cooperative MDP in Section 3, page 4) is well-suited for the complexities of options trading and represents a significant and original contribution to the application of RL in finance.\n\n2.  **Thorough and Robust Empirical Evaluation:** The experimental setup is comprehensive and of high quality. The authors test their model on a multi-year dataset (Table 1) covering diverse market regimes (bull, bear, high volatility). They compare OPHR against a strong and varied set of baselines, including directional strategies, single-factor models, and standard ML forecasting models (Section 5.1). The results, particularly in Table 2, show a consistent and substantial outperformance of OPHR, not just in total returns but, more importantly, in risk-adjusted metrics like Sharpe, Calmar, and Sortino ratios. The ablation study (comparing `OPHR` to `OP`) effectively demonstrates the value added by the adaptive HR-Agent.\n\n3.  **High-Quality Presentation and Reproducibility:** The paper is well-written and clearly structured. Figures 1 and 2 provide excellent visual intuition for the core challenges (hedging) and the proposed architecture. Furthermore, the appendices are exceptionally detailed, providing full transparency on the experimental setup. The inclusion of detailed hyperparameter tables (Tables 4, 5, 6), baseline implementations (Appendix C.3), and evaluation metric definitions (Appendix C.1) significantly strengthens the paper and provides a clear path for reproducing the work.\n\n### Weakness\n\n1.  **Omission of Transaction Costs:** The authors correctly state that \"frequent hedging incurs transaction costs\" (Line 181), identifying this as a key real-world challenge. However, the simulation environment appears to be frictionless, omitting trading fees, bid-ask spreads, and potential slippage. This is a critical limitation. A core component of the OPHR strategy is dynamic hedging, which can involve frequent trades. In real markets, these transaction costs can accumulate rapidly and significantly erode, if not completely eliminate, the reported profits. The omission undermines the practical viability of the strategy and weakens the claim of outperformance.\n\n2.  **Unrealistic Assumption for Greek Calculation:** The framework's state representation and risk management rely heavily on option Greeks (Δ, Γ, Θ, Vega), as stated in Line 134 and used by the HR-Agent. The paper details the standard Black-Scholes-Merton (BSM) formulas for these Greeks in Appendix A.1.3. Using BSM, which assumes a single, constant volatility, is highly unrealistic for cryptocurrency options markets that exhibit pronounced volatility smiles and skews. This means the Greeks used as input to the HR-Agent are likely inaccurate. The very high \"Residual\" PnL reported in the PnL attribution analysis (Table 13, e.g., -51.45% for BTC Long and 153.03% for ETH Short) serves as evidence that the BSM model fails to explain a large portion of the portfolio's returns, directly highlighting the model's inadequacy in this context.", "was_revised": true, "final_justification": "The concerns were addressed by author's rebuttal.", "rebuttal": "Thank you for your thorough and insightful review. Please find our responses below.\n\n### **Omission of Transaction Costs**\nWe apologize for not clearly stating this in the paper. We **do** consider transaction costs in our simulation, which are deducted from the cash balance when trades are executed. Our implementation follows Deribit's fee structure: **Commission fees**: 0.05% for perpetual futures and 0.03% for options (capped at 12.5% of option price). We also consider the **Bid-ask spread costs**: We use market orders in backtesting, so spread costs are implicitly included through realistic execution prices.\n\nImportantly, our trading policy is **not high-frequency**. As shown in Table 3, average holding periods range from 9-21 hours for long positions and 47-51 hours for short positions. The HR-Agent selects hedgers every 24 hours (Table 6), making transaction costs a relatively small component of total P&L. \n\nThe following table shows transaction costs as a percentage of total P&L in our experiments:\n\n| Strategy | Options Cost (% of P&L) | Underlying Cost (% of P&L) | Total Cost (% of P&L) |\n|----------|-------------------------|---------------------------|----------------------|\n| OPHR BTC | 4.15% | 5.21% | 9.36% |\n| OPHR ETH | 2.97% | 2.78% | 5.75% |\n\nWe will add this crucial information to the revised manuscript.\n\n### **Unrealistic Assumption for Greek Calculation & Advanced Vol Smile Models**\nWe acknowledge that BSM makes simplifying assumptions, but our framework is designed to be robust to Greek calculation imprecision for several key reasons:\n\n**1. Greeks as reference values, not precise inputs**: Our method doesn't rely on perfect Greek precision but uses them as reference signals. BSM remains widely used by market participants despite its limitations - experienced traders understand these constraints and adapt accordingly.\n\n**2. Data-driven error correction**: The Deep Hedgers take Greeks as input features, and the HR-Agent routes among hedgers based on their actual performance during RL training. This means model misspecification is automatically \"priced in\" through the data-driven learning process, similar to how human traders compensate for known BSM limitations.\n\n**3. Residual PnL explanation**: The high residuals in Table 13 are primarily due to discrete time intervals (5-minute data) rather than Greek misspecification. Second-level data would yield much smaller residuals, as the Taylor expansion in Eq. (2) assumes continuous hedging. Our PnL decomposition serves to qualitatively analyze whether our strategy's profit sources align with expectations (short positions earning theta, long positions earning gamma).\n\n**Why advanced smile models aren't necessary**: More sophisticated volatility smile models (e.g., SABR) focus on precise pricing across the entire volatility surface, typically used for market-making or volatility surface arbitrage. Our strategy trades only ATM straddles with relatively low frequency, making precise smile modeling unnecessary for our specific application.\n\n### **Why ML-based Strategies Fail**\nThe poor performance of ML baselines despite good forecasting metrics (Table 10 vs Table 2) highlights fundamental challenges in translating predictions to profitable trading:\n\n**1. Model assumption burden**: ML forecasting requires explicit volatility modeling choices (estimators, frequencies, etc.) that introduce model misspecification risks. RL circumvents these assumptions through direct action optimization, adapting to market dynamics that traditional volatility frameworks may not capture.\n\n**2. Forecasting-execution gap**: Building on these modeling limitations, ML models excel at volatility forecasting but struggle with the sequential, path-dependent execution required for options trading. As shown in Section 2.2 and Figure 1, optimal hedging decisions depend on complex interactions between market conditions, position Greeks, and future price paths. Converting point forecasts into profitable trading sequences requires manually designed rules that are difficult to optimize across market regimes - precisely the challenge our grid search experiments revealed.\n\nThese fundamental issues (1 & 2) affect all prediction-based approaches, while ML methods also face an inherent limitation:\n\n**3. Fat-tail prediction failure**: While ML models achieve good overall forecasting performance, Table 11 shows they perform poorly on extreme volatility events (top 5% outliers) where most long-gamma profits are generated. RL's exploration and adaptation capabilities provide advantages precisely in these critical profit-generating scenarios.\n\n**End-to-end ML forecasting baseline**: We add an end-to-end ML baseline that circumvents problems 1 & 2 following [1]. Please refer to the detailed experiment results in section **Additional baseline details (DLOT)** in our response to **Reviewer JxTL**. Results show that supervised learning methods still have significant disadvantages compared to RL in this scenario, demonstrating the superiority of RL's adaptive learning approach even when modeling assumptions are minimized.\n\n### **Oracle Policy Sensitivity Analysis**\n\nThe Oracle policy serves as a crucial bootstrap mechanism during Phase 1. To ensure diverse and profitable experience generation, we adopt a range of β thresholds when constructing the Oracle policy. Specifically, as shown in Table 4, we vary the Oracle threshold β across multiple values (0.1, 0.2, 0.4, 0.6, 0.8) to produce a broad spectrum of trading signals. This diversity encourages the OP-Agent to learn a richer set of profitable behaviors during pre-training.\n\nThese varied Oracle signals are then incorporated and refined in Phase 2, where the agent learns to adaptively combine and build upon these strategies. This two-phase approach is critical due to the delayed nature of profit realization in volatility trading, providing structured guidance toward profitability early in training.\n\nBased on our design rationale, the Oracle need not be optimal - it simply needs to demonstrate profitable trading patterns that reduce exploration burden. Shortening or skipping the pre-training phase would lead to significantly slower convergence, as the agent would lack initial guidance and struggle to discover effective volatility trading strategies through exploration alone amid market noise. We will incorporate these explanations into the revised manuscript for clarity.\n\n### **Reference:**\n[1] Deep Learning for Options Trading: An End-To-End Approach. ICAIF 2024."}, {"review_id": "h42eLGlEI6", "reviewer": "Reviewer_WWUg", "summary": "This paper introduces OPHR, a multi-agent reinforcement learning framework designed for volatility trading in options markets, specifically targeting cryptocurrency derivatives. The primary trading objective is to capture the difference between implied volatility and realized volatility. The framework consists of two specialized agents: the OP-agent, which is responsible for volatility timing by selecting long, short or neutral positions in the parity cross-market. The HR-Agent selects from hedging strategies with different levels of risk aversion to optimize delta hedging under different market conditions. Training is performed in two phases: an offline initialization using an Oracle policy and online iterative joint training of the two agents. The approach is evaluated on BTC and ETH options with superior results over multiple baselines.", "questions": "1. Technical details on the Hedger pool construction (e.g., hyperparameter sensitivity of the hedger selection) could be expanded.\n\n2. Considering real-world deployment in live markets, how do you handle the latency of action selection for HR-Agent given that hedging may need near real-time responsiveness, especially during volatility spikes?\n\n3. Have you tested OPHR’s generalizability to other markets (e.g., SPX options, equity index options)?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1. Thoughtful model design addressing practical market issues like transaction costs and discrete hedging intervals.\n\n2. Novel decomposition into an OP-Agent and a separate HR-Agent. Use of Oracle policy for offline pretraining is well justified.\n\n3. Extensive experiments on real market data and OPHR demonstrates robust returns, reduced drawdowns, improved Sharpe/Calmar/Sortino ratios, and sophisticated volatility timing behaviors.\n\n### weaknesses\n1. There are no significant innovations from the perspective of RL methods, and the core contribution lies in the use of existing RL methods for volatility trading.\n\n2. Lack of comparison with more advanced or recent volatility models, such as GARCH variants or deep volatility forecasting networks, which may outperform ML baselines like MLP or LSTM.\n\n3. There is a lack of ablation studies to illustrate the performance improvement and significance of TD learning method design for both OP-Agent and HR-Agent.\n\n4. Limited theoretical analysis. This work has strong application value but lacks profound theoretical basis. However, the code of the results that pay attention to practical application is not open source, which makes it difficult to reproduce and further research, and may reduce its contribution to a certain extent.", "was_revised": true, "final_justification": "I have read the rebuttal, and it addressed my main concerns, so I have increased the score.", "rebuttal": "Thank you for your valuable comments. Please find our detailed responses below.\n\n### **Innovation from RL Perspective**\nWhile our framework builds upon established RL techniques, this work represents the **first application of RL to volatility trading in options markets**. Our contributions go far beyond naive application of existing methods - we designed a complete framework specifically for this previously unsolved problem:\n\n1. **Novel problem formulation**: We introduce the first Cooperative MDP formulation for volatility trading (Section 3), decomposing the complex task into specialized agents with complementary objectives that address the unique interdependence between position sizing and dynamic hedging.\n2. **Tailored multi-agent training framework**: The two-phase training with Oracle initialization, alternating agent updates, and relative reward design specifically addresses volatility trading's challenges - delayed rewards, noisy signals, and complex state-action dependencies that don't exist in traditional RL applications.\n3. **Specialized technical details**: Critical implementation details include n-step TD for handling noisy option trading rewards, lower-frequency HR-Agent switching for training stability. These technical choices are essential for making RL work in this domain.\n\nThe success of our approach required solving numerous domain-specific challenges from problem formulation through implementation details - simply applying standard RL algorithms would not have succeeded in this complex financial trading environment.\n\n### **Advanced Volatility Model Baselines**\nEven sophisticated volatility models like GARCH variants or deep volatility networks face the same fundamental limitations we identified:\n\n**1. Model assumption burden**: Advanced volatility models still require explicit modeling choices and assumptions that introduce misspecification risks, while RL learns optimal actions directly without these constraints.\n**2. Forecasting-execution gap**: The core issue isn't prediction quality (Table 10 shows respectable performance) but translating forecasts into profitable sequential trading decisions. Advanced models excel at forecasting but cannot capture the path-dependent execution required for options trading.\n\n**Empirical evidence**: We tested GARCH and DeepVol models:\n\n| Model | TR(%)↑ | ASR↑ | ACR↑ | ASoR↑ | AVOL(%)↓ | MDD(%)↓ | WR(%)↑ | PLR↑ |\n|-------|--------|------|------|-------|----------|---------|--------|------|\n| **BTC GARCH** | -40.83 | -4.29 | -0.95 | -5.54 | 9.51 | 42.76 |26.32 | 0.78 |\n| **BTC DeepVol** | -14.24 | -1.30 | -0.65 | -2.44 | 10.92 | 21.85 | 36.36 | 1.33 |\n| **ETH GARCH** | -52.59 | -3.79 | -0.90 | -4.18 | 13.85 | 58.10 | 12.50 | 0.62 |\n| **ETH DeepVol** | -22.82 | -1.79 | -0.96 | -3.03 | 12.71 | 23.76 | 42.86 | 1.15 |\n\nDespite their sophistication, these models still underperform significantly. The transition from forecasting capability to profitability requires complex model construction and market-specific adjustments - a process our RL method circumvents through direct action optimization without manual model tuning.\n\n### **Ablation Study on n-step TD Learning**\nWe use n-step TD because 1-step TD struggles with convergence due to noisy stepwise reward signals in options trading. The n-step approach is essential for capturing the delayed and cumulative nature of options trading rewards. In volatility trading, profits don't materialize immediately from individual actions but accumulate over multiple time steps through gamma scalping (buying low and selling high as prices oscillate) and theta decay (time value erosion). A single trading decision's profitability depends on subsequent market movements and hedging actions, creating reward signals that are inherently delayed and path-dependent. In our experiments, 1-step TD fails to adequately learn from these profitable experiences because it cannot connect immediate actions to their eventual cumulative rewards, leading to poor strategy development and convergence difficulties. The n-step TD method effectively addresses this by incorporating future rewards over longer horizons, enabling better learning from the temporal structure of options trading profits.\n\n### **Theoretical Analysis and Reproducibility** \nWhile we don't provide formal convergence guarantees, our framework incorporates principled design elements that ensure performance stability in practice: Oracle initialization, n-step TD learning for noise reduction, lower-frequency agent switching for training stability, and relative reward design for stable learning signals.\n\n**Regarding reproducibility**: We commit to open-sourcing our complete backtesting environment and RL training framework to enable full reproducibility. We will protect only the specific feature engineering components that provide competitive advantage, ensuring the core methodology remains accessible while balancing academic contribution with practical deployment considerations.\n\n### **Technical Details on Hedger Pool Construction**\nComprehensive details are provided in Appendix B.2, including:\n- **Selection criteria**: Top-30 performers based on validation performance across market regimes\n- **Hyperparameter ranges**: Tables 5-6 detail all parameters for Delta-based, Price-based, and Deep Hedgers\n- **Training methodology**: Different risk-aversion levels (λ from 0.1 to 800) ensure diverse hedging behaviors\n\n### **HR-Agent Responsiveness in Real-time Trading**\n**Training vs. deployment distinction**: During training, the HR-Agent switches hedgers at fixed 24-hour intervals (nhr = 24) to ensure RL training stability - frequent switching would cause training instability and convergence difficulties. However, in live deployment, the HR-Agent can operate in an **event-driven manner**, responding immediately to significant market moves or Greek threshold breaches. The fixed interval is a training constraint, not a deployment limitation.\n\n### **Generalizability to Different Markets**\nOur focus on cryptocurrency options markets is driven by practical advantages and research value:\n\n**Data accessibility**: Crypto derivatives provide comprehensive 24/7 data access essential for robust RL training, unlike traditional options markets with limited historical data availability.\n\n**Research value**: As an emerging market, crypto options offer higher research value and cleaner microstructure without complex market maker intermediation, allowing focus on core volatility trading dynamics.\n\n**Framework generalizability**: Our RL methodology can extend to different markets, with success primarily depending on market-specific feature engineering. Even within our study, BTC and ETH required different feature engineering approaches to achieve optimal results (though all baselines for each asset used identical features). However, feature engineering optimization falls outside this research's scope - our contribution lies in the RL framework design rather than market-specific feature selection.\n\nOur research aims to demonstrate that RL methods indeed have advantages over traditional approaches in the volatility trading domain."}, {"review_id": "C7WVEwnl6E", "reviewer": "Reviewer_JxTL", "summary": "In this paper, the authors present a two-agent framework for learning option trading using deep reinforcement learning. The authors propose using an option position agent responsible for volatility timing and a hedge routing agent to manage risk. The authors compare the proposed approach to 8 different baselines using various evaluation metrics and employ two crypto option datasets (BTC and ETH).", "questions": "- The authors claim in l. 47-49 that \"options trading is an untouched area due to its complexity,\" while citing existing papers on the topic (e.g., Tan et al., 2024) without appropriate discussion or comparison. Could the authors clarify why such prior work is not more thoroughly discussed or included in the comparisons, and whether their claims regarding novelty should be revised?\n\n- It is not clear whether commission and transaction costs were incorporated into the reward function or evaluations. Could the authors clarify whether these costs were included, and if so, how they were estimated and implemented?\n\n- The paper does not discuss the stability of the training process or the robustness of the learned policies. Could the authors comment on whether the learned policies are stable and how multiple runs are expected to affect the results?\n\n- In Table 2, almost all baselines (except one) lead to negative profits. Could the authors provide more details on how the baselines were implemented and tuned, to clarify whether these results reflect inherent limitations of the baselines or possible issues in their implementation?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper addresses an interesting topic, which has been less studied in recent years despite the wealth of literature on financial trading using DL/DRL. The authors nicely motivate their work and present all necessary background in a concise and easy-to-follow manner. I think the paper holds some value, mainly due to the proposed use of two agents for simultaneously optimizing the objective. However, I also have the following important concerns:\n\n- The authors claim in l. 47-49 that \"options trading is an untouched area due to its complexity.\" However, there are papers on topics very closely related, e.g.:\nTan, Wee Ling, Stephen Roberts, and Stefan Zohren. \"Deep Learning for Options Trading: An End-To-End Approach.\" Proceedings of the 5th ACM International Conference on AI in Finance, 2024.\namong others. This paper is also cited by the authors but is not appropriately discussed or compared against. This is a major shortcoming of the work. Therefore, I think the authors should revisit the paper and reconsider some of their strong claims and/or appropriately discuss the existing literature on the topic, which, even though limited, does exist.\n\n- It is not clear whether commission/transaction costs have been incorporated into the reward or evaluations. The authors do mention them, but it is unclear whether they were incorporated and how they were calculated. The impact of different transcation costs on the performance of the method is not quantified.\n\n- The authors do not appropriately discuss the stability of the training process and/or provide evaluations to demonstrate whether the policies learned by the agents are stable (e.g., by conducting multiple training/evaluation experiments and reporting the standard deviation).\n\n- All baselines used in Table 2, apart from one (over 16), result in negative profit. These results raise questions about how well the baselines have been implemented and/or tuned.\n\n- The presentation of the paper can be improved. There are important acronyms defined only in the abstract and not in the main text (IV, RV), acronyms never defined (PnL), formats not defined (e.g., dates in Table 1), etc. Overall, the paper requires very careful proofreading to correct such issues and to ensure that all information necessary for the reader is appropriately provided in the text. The presentation of the main method is also somewhat confusing. The authors provide an overview in Fig. 2, but the presentation order does not follow the logical flow of the method (e.g., the oracle distillation is actually presented at the end).", "was_revised": true, "final_justification": "After reading the authors’ rebuttal, I am increasing my score to Borderline Accept, as the authors have addressed most of my concerns. However, I share the view of other reviewers that the methodological novelty (e.g., in the formulation) is limited, and the paper leans more toward the application side. Therefore, I would not oppose rejecting the paper if there is a strong opinion towards this.", "rebuttal": "Thank you for your valuable comments. Please find our detailed responses below.\n\n### **Originality and Related Work Discussion**\nYou are correct in pointing out the work by Tan et al. [1]. We acknowledge this important reference and will revise our claims more precisely. While [1] does apply deep learning to options trading, there are key differences in problem formulation and approach:\n\n**Different trading objectives**: [1] focuses on **option portfolio management**, constructing long/short positions across different stocks' straddles to form diversified option portfolios without underlying hedging. Our work specifically targets **volatility trading with dynamic hedging** for single underlying assets, which involves fundamentally different challenges, including gamma scalping, delta hedging, and path-dependent profit optimization.\n\n**Methodological differences**: [1] uses supervised learning for portfolio allocation decisions, while we are the **first to formulate volatility trading as an MDP problem** and apply reinforcement learning to learn optimal volatility timing and hedging strategies jointly.\n\n**Market focus**: [1] addresses equity options portfolio construction, while we focus on cryptocurrency options volatility trading with continuous hedging requirements.\n\nWe will revise our manuscript to more rigorously discuss this related work and clarify that our contribution lies in being the first RL framework designed explicitly for **volatility trading with dynamic hedging**, rather than general options trading.\n\n### **Transaction Costs**\nWe do incorporate transaction costs in our simulation:\n- **Commission fees**: Following Deribit's structure - 0.05% for perpetual futures and 0.03% for options (capped at 12.5% of option price)\n- **Bid-ask spread costs**: Implicitly included through realistic market order execution prices in backtesting\n\nOur trading frequency is relatively low (average holding periods of 9-51 hours as shown in Table 3), making transaction costs a small component of total P&L. The following table shows transaction costs as a percentage of total P&L in our experiments:\n\n| Strategy | Options Cost (% of P&L) | Underlying Cost (% of P&L) | Total Cost (% of P&L) |\n|----------|-------------------------|---------------------------|----------------------|\n| OPHR BTC | 4.15% | 5.21% | 9.36% |\n| OPHR ETH | 2.97% | 2.78% | 5.75% |\n\nThese results demonstrate that our trading strategy is not sensitive to transaction costs, as the fees represent a relatively small proportion of overall profits. We apologize for not stating this clearly and will add this information to the revised manuscript.\n\n### **RL Training Stability**\nWe acknowledge the importance of training stability analysis. Our framework incorporates several stability-enhancing design elements:\n\n**Oracle initialization**: Phase 1 provides consistent starting points, reducing variance across training runs.\n\n**n-step TD learning**: Effectively reduces noise in Q-value estimation, making RL training convergence more stable than 1-step methods.\n\n**Lower-frequency HR-Agent switching**: The HR-Agent's less frequent hedger selection (every 24 hours) improves training stability by reducing action space complexity and preventing convergence difficulties from frequent switching.\n\n**Relative reward design**: Using the advantage of selected hedgers over baseline hedgers as reward provides more stable learning signals than absolute performance measures.\n\n### **Baseline Performance and Implementation**\nWe provide detailed descriptions of all baseline implementations in Appendix C.3, including consistent principles for position entry/exit, hedging policy, take-profit/stop-loss rules, and holding periods setup.\n\nFor each category of baseline strategies, we performed systematic grid search to optimize key hyperparameters as follows:\n\n**Single-Factor Models (Mean Reversion / Momentum)**: Construct percentile-based signals from a 365-day historical RV distribution. Realized volatility is computed over multiple time windows (3–24 hours), and trading is triggered when thresholds on the percentile or its momentum are met.\n\n**Machine Learning Models (GBDT, MLP, LSTM)**: Use predicted annualized RV over a forward window and compare it to current ATM IV to generate trading signals. Entry/exit thresholds were tuned for best validation performance. Fine-tuned take-profit and stop-loss rules were applied.\n\nAll baseline strategies were evaluated under the same market environment with performance metrics to ensure fair and consistent comparison.\n\nDespite careful tuning and consistent evaluation, the predominantly negative baseline returns reflect the inherent difficulty of volatility trading rather than implementation issues:\n\n**1. Model assumption burden**: Traditional methods require explicit volatility modeling choices that introduce misspecification risks, while RL learns optimal actions directly without these constraints.\n\n**2. Path-dependent execution gap**: Building on these modeling limitations, optimal hedging decisions depend on complex interactions between market conditions, position Greeks, and future price paths (Section 2.2, Figure 1). ML models provide point forecasts but cannot capture the sequential, path-dependent nature where the sequence of hedging decisions critically impacts P&L. Converting forecasts into profitable trading sequences requires manually designed rules that our grid search experiments showed are difficult to optimize across market regimes.\n\n**Additional baseline details (DLOT)**: While [1] trades multiple assets' straddles without delta hedging (using only straddle P&L as labels), we focus on single-asset volatility trading requiring delta hedging. Therefore, we use delta-hedged straddle P&L as prediction labels. This baseline closely resembles our Phase 1 Oracle policy - one uses supervised learning loss, the other uses offline RL loss.\n\n**Results analysis**: DLOT significantly outperforms RV prediction-based methods, benefiting from an end-to-end design that avoids model assumptions - similar in spirit to our RL approach. However, lacking adaptive hedger selection and further RL optimization, it cannot match our method's performance.\n\n| Method | TR(%)↑ | ASR↑ | ACR↑ | ASoR↑ | AVOL(%)↓ | MDD(%)↓ | WR(%)↑ | PLR↑ |\n|---------|--------|------|------|-------|----------|---------|--------|------|\n| **BTC Delta-hedged P&L** | 4.91 | 0.52 | 0.55 | 0.66 | 21.40 | 8.92 | 47.97 | 1.11 |\n| **ETH Delta-hedged P&L** | 1.19 | 0.07 | 0.09 | 0.09 | 17.14 | 13.53| 42.80 | 1.34 |\n\nThis confirms that even sophisticated prediction-based approaches struggle with the execution challenges our RL framework addresses.\n\n### **Presentation Improvements**\nThank you for these valuable suggestions. We will comprehensively revise the manuscript to address acronym clarity (defining IV, RV, P&L in the main text), format consistency, logical flow reorganization, and comprehensive proofreading to ensure all technical terms are properly introduced. We appreciate your thorough review and will incorporate these improvements to enhance the paper's clarity and accessibility.\n\n### **Reference:**\n[1] Deep Learning for Options Trading: An End-To-End Approach. ICAIF 2024."}]

Author rebuttal:

Meta-review: The paper proposes a two-agent RL framework for options volatility trading—an OP-Agent times long/short/neutral exposure while an HR-Agent routes among hedgers to manage delta and path-dependent P&L—formulated as a cooperative MDP and trained via oracle-initialized offline pretraining plus joint online learning. Reviewers praise the decomposition, careful engineering (n-step TD, lower-frequency HR switching, relative rewards), and broad empirical gains on BTC/ETH options, especially after the rebuttal added advanced volatility baselines (GARCH, DeepVol, DLOT) and clarified transaction-cost modeling (fees amounting to ~6–9% of P&L). Remaining concerns center on modest RL novelty, reliance on BSM Greeks, lack of formal guarantees, limited multi-seed/stability reporting, and generalization beyond crypto.
Decision: Accept (poster)