TradeMaster: A Holistic Quantitative Trading Platform Empowered by Reinforcement Learning
Authors: Shuo Sun
Molei Qin
Wentao Zhang
Haochong Xia
Chuqiao Zong
Jie Ying
Yonggang Xie
Lingxuan Zhao
Xinrun Wang
Bo An
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: quantitative trading
reinforcement learning

Abstract: The financial markets, which involve over \$90 trillion market capitals, attract the attention of innumerable profit-seeking investors globally. Recent explosion of reinforcement learning in financial trading (RLFT) research has shown stellar performance on many quantitative trading tasks. However, it is still challenging to deploy reinforcement learning (RL) methods into real-world financial markets due to the highly composite nature of this domain, which entails design choices and interactions between components that collect financial data, conduct feature engineering, build market environments, make investment decisions, evaluate model behaviors and offers user interfaces. Despite the availability of abundant financial data and advanced RL techniques, a remarkable gap still exists between the potential and realized utilization of RL in financial trading. In particular, orchestrating an RLFT project lifecycle poses challenges in engineering (i.e. hard to build), benchmarking (i.e. hard to compare) and usability (i.e. hard to optimize, maintain and use). To overcome these challenges, we introduce TradeMaster, a holistic open-source RLFT platform that serves as a i) software toolkit, ii) empirical benchmark, and iii) user interface. Our ultimate goal is to provide infrastructures for transparent and reproducible RLFT research and facilitate their real-world deployment with industry impact. TradeMaster will be updated continuously and welcomes contributions from both RL and finance communities.

TradeMaster: A Holistic Quantitative Trading Platform Empowered by Reinforcement Learning
Shuo Sun'
Molei Qin
Wentao Zhang
Haochong Xia
Chuqiao Zong
Jie Ying
Yonggang Xie Lingxuan Zhao Xinrun Wang
Bo An
Nanyang Technological University, Singapore {shuo003, www.hhonoo00,..... zong0005 , ying0024, xiey0035, zhao0375) wtw.eee..u..eeee wtw.e...uuuuuu.gg.....e
{molei qin, wt. .rtnzh xinrun. wang, boan) } @ @ntu.edu.sg
*Equal contribution
CCorespponding author
Abstract
The financial markets, which involve over $90 trillion market capitals, attract the attention of innumerable profit-seeking investors globally. Recent explosion of reinforcement learning in financial trading (RLFT) research has shown stellar performance on many quantitative trading tasks. However, it is still challenging to deploy reinforcement learning (RL) methods into real-world financial markets due to the highly composite nature of this domain, which entails design choices and interactions between components that collect financial data, conduct feature engincering, build market environments, make investment decisions, evaluate model behaviors and provides user interfaces. Despite the availability of abundant financial data and advanced RI. techniques, a remarkable gap still exists between the potential and realized utilization of RL in financial trading. In particular, orchestrating an RLFT project lifecycle poses challenges in engineering (i.e., hard build), benchmarking (i.e., hard compare) and usability (i.e., hard optimize, maintain and use). To overcome these challenges, we introduce TradeMaster, a holistic open-source RLFT platform that serves a software toolkit, ii) empirical benchmark, and iii) user interface. Our ultimate goal is to provide infrastructures for transparent and reproducible RLFT research and facilitate their real-world deployment with industry impact. TradeMaster will be updated continuously and welcomes contributions from both RL and finance communities.
Software Repository: http::///ggtaaooc//////attatstts
1 Introduction
Quantitative trading, which refers to the process of applying mathematical models and computer algorithms to automatically identify trading opportunities [8], has been a popular research topic for decades [3]. With the increasing accessibility of financial data [1 1] and the development of RI. techniques [63, 75, 12. 18]. RLFT has made great strides in offering profitable financial trading models [68] in recent years. From algorithmic perspective, many RLFT algorithms are proposed for core quantitative trading tasks such as portfolio management [77, 76, 80, 30], algorithmic trading [13, 66, 42]. and order execution [39, 17, 50], respectively. In addition, plethora of research has tried to address other key problems alongside the RLFT pipeline, including preprocessing financial data [40, 71], alpha discovery [10, 82], market simulation [41, 60. 9], systematic evaluation [67] and online deployment [11]. However, the vast majority of existing works are formulated, addressed and implemented separately, resulting in stylized range of methods with limited acknowledgemen the complexities and interdependencies in the RLFT pipeline (as a composite). This leads to an often
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

punishing translational barrier between -tate-of-iee sfate-of-the-ar the-art RLFT research and investors' usage of RL techniques to earn real financial benefits [29, 11, 68].
Three Challenges. To facilitate RLFT research and their industry deployments, we call for a holistic platform for the design, development and evaluation of RLFT applications. Specifically, due the ever-changing nature of financial markets [49] and high performance variability of RL [25], managing real-world RLFT workflows non-trivial with the following three challenges:
First and foremost, the engineering problem is that building sophisticated RLFT pipelines requires significant upfront efforts. For immature domain (e.g., RLFT), it is common to spend >90% of work on implementation details and <10% of work on core technical questions [58]. As an RLFT researcher, it seldom the case that enough resources are available to build high quality code base of the complete RLFT workflow. As result, a holistic platform that encapsulates all major components of RLFT with comprehensive support of mainstream settings and state-of-the-an algorithms is highly desirable for both academic researchers and industry practitioners.
Second, the benchmarking problem is that the performance of any component depends on its context. For instance, the profitability of RLFT algorithm is intimately tied to the features used to train RL agents [10] and the market status during evaluation [51]. However, existing works typically examine the merits of each component individually and arbitrarily configure surrounding settings for only ensuring "all else equal" conditions, which lead to inconsistent reporting of revenues with no general consensus on the relative strength of popular RLFT methods [68]. A reproducible empirical benchmark that honestly reflects interactions of RLFT components is urgently needed.
Third, the usability problem is incurred by the complexities of RLFT, which requires extensive interdisciplinary knowledge to be properly optimized and maintained in practice. Specifically, many knobs (e.g., hyperparameters) need to be tuned for RLFT methods [33] and regular maintenance (e.g., model retraining with up-to-date market data) is unavoidable due to the significant temporal distribution shift financial markets [35]. These difficulties are usually insurmountable obstacles for non-expert users (e.g.. individual investors) and hinder the wide utilization of RLFT methods. A user- friendi platform where the whole process is highly automated with excellent user interfaces is indispensable for this domain.
Contributions. TradeMaster is proposed to tackle the above three challenges in one unified RLFT platform with the following contributions: i) As a software toolkit, it offers open-source imple- mentations of the whole RLFT workflow including 13 real-world financial datasets, high-fidelity RL environments for 6 mainstream quantitative trading tasks. 15 popular RLFT algorithms and dozens of tools for systematic evaluation and visualization. This modular and composable structure enables fast development and decrease the cost of collaboration and code-sharing. ii) As an empirical benchmark, it provides rigorous comparison of state-of-th oftt--of-the-art RLFT algorithms to determine their effectiveness across different tasks, financial markets and evaluation metrics. The standardized pipelines of TradeMaster ensure that the comparison is transparent, fair and reproducible. iii) As user interface, TradeMaster provides multiple options for practical use: an open-source Github repository, a Python package, an online web service with graphical user interface (GUT), multiple Jupyter Notebook tutorials and cloud version using Colab. In addition, we customize advanced AutoML techniques into TrsteMaster to finish many key steps of RLFT (c.g., feature engineering and hyperparameter tuning) in an automatic way. These efforts significantly improve the usability of TradeMaster and make it accessible for users with various background and investment goals.
2 Quantitative Trading as a Markov Decision Process
As shown in Figure 2, we formulate quantitative trading tasks as a Markov Decision Process (MDP) following a standard RL scenario, where an agent (investor) interacts with an environment (the financial markets) in discrete time make actions (investment decision) and get re- ward (profits). Formally, we define the MDP as a 6-tuple: (S, A, P, R, 7, H). Specifically, S is a finite set of states. A is a finite set of actions. P S x A S - [0, 1] state transition function, which consists of a set of conditional transition probabilities between states.
Invessor
Maricet Information
Invesmment Action
Profit
A
Pintrcal Market
p Nassday
Figure 2: MDP formulation in RLFT

---

Eva Evaluation
Financial Metrics
Visualization Toolkit
Interface
GitHub Repo
Profit
Diversity
t-SNE
Comparison
Test
Python
Risk
Robustness
PRIDE-Star Star
Diversity Heatmap
Performance Profile
Tutorial
Colab
Algorithm
RLFT Algorithm
Classic RL Algorithm
AutoML
DeepScalper
ETTO
ElIE
IMIT
DDPG
Optuna
Deep Trader
SARL
OPD
A2C
DON
OpenFE
Simu lator
Macro-level Task
Micro-level Task
Crypto
Portfolia Management
Intraday Trading
Order Execution
High Frequency
Market
Trading
Trading
Making
Data Source
Type
Financial Asset
Frequency
Preprocess
Data Cleaning
Feat. Embedding
K-line Chart
Stock
Crypto
1 Day
Hour
Data Imputation
Alpha Discovery
Limit Order Book
Future
FX
15 Min
1 Min
Normalization
Feature Selection
Figure 1: Overview of the Mrster platform, which includes six key components for RLFT.
R A R the reward function. Y E [0, 1) the discount factor and H a time horizon indicating the length of the trading period. A (stationary) policy Te S x A [0, 1], parameterized by 0, assigns each state s E S distribution over actions, where a E .A has prob- episode corresponds
ability [[ass) During training, time step through one whole trading period. The objective of the agent is to learn an optimal pol- icy (investment strategy) that maximizes the expected sum of discounted reward (overall profit):
making investment decisions each
argmar Br, El= 7"res
We further describe details on the definition of S, A.
R and in existing works as follows:
State Space (S): A state represents RL agents' perception on market status. Specifically, price information (e.g., open, high, low and close price). trading volume, technical indicators and snapshot of limit order book are used in existing works for different trading tasks [76, 66, 17, 78]. In addition, some works feed alternative data, i.e., financial news [56], prediction of future trend [80], experts' investment behaviors [14] and overall market status [77], into states to further improve the performance of RL agents. Furthermore, trader's personal information (e.g., holding position and remaining cash) is also applied private state in [48, 66, 50] to help investment decision making.
Action Space (A): At each time step, the action space represents all possible actions of RI agents' investment decisions that vary under different trading tasks [68]. For single asset trading, the actions are the number of shares to buy/sell long/short the asset used in [13, 42]. For portfolio management, the actions represent the proportions of capitals allocated into each financial asset as in [76, 77]. For order execution and market making, the actions correspond to a limit order that shows traders' desired price and quantity as in [17, 39].
Reward Function (R): Previous works [76, 42, 31] commonly apply the change of capitals (how much money earned/losed) as the reward function, which natural and easy to implement. Many practical constraints such commission fee and slippage further included be more realistic. In addition, there are also several alternative options designed for specific settings including Sharpe ratio in [77], the hindsight reward function in [66] and the binary sparse reward function in [39].
Transition Function (P) is defined by the financial market progression following the investment actions. The progression is defined by building realistic data-driven simulated environment based on real-world financial data following the paradigm in [41, 53].
Examples of the MDP in RLFT. Considering simple trading scenario with only one stock, we obtain of and pf+1. which denote the close price of the stock time and + from historical data. The action at time / is to buy l shares of the stock. Then, the reward at time / is the account profit defined as k (pf+1 pf). For state, we use historical market data to calculate technical indicators external state and investors' private information such as remaining cash and current position is

---

applied as internal state. Similar procedures to build RL environments with historical market data have been applied many RLFT work [40. 77, 80).
Further Clarification, We would like to highlight that the above descriptions on the MDP formulation is general version. As holistic platform, Mrster covers wide range of trading scenarios, which is impossible to offer detailed MDP formulation for each scenario here due to page limit. For readers unfamiliar with RLFT, Appendix A and this survey [66] offers detailed introduction on related terminologies and detailed MDP formulation of each RLFT task.
3 TradeMaster a Software Toolkit
To provide a high-quality implementation of the complete RLFT workflow, we first introduce our 3 design principles [26, 29] based our experience in prototyping and developing RLFT projects. Then, we discuss details on the six component modules of TradeMaster as shown in Figure 1.
Trate Design Principles
Due to the strict "separation of concerns" produced by the high-level API functions of each component, the goal of Trade:asster is to first enable the proper pipeline functionality, while the intricacies and configurations of individual algorithm choices the back burner. Be Minimal and Unintrusive: While workflow development needs to be unified and systematic, we believe learning this manifests the clear code structure and user-friendly interfaces of TradeMaster that enables easy usage and adoption for different users. Encourage Extension: Since RLFT is emerging research direction with many novel methods proposed rapidly, the platform components should be extensible for the incorporation of new methods. Concretely, TradeMaster encapsulates the design of functions within each component module with support of easy extension.
Pipeline First, Algorithms Second:
the platform needs be easy and intuitive. Concretely,
Data Sources. We include 13 long-term real-world financial datasets that are highly diverse in terms of financial assets (e.g., stock, FX, and Crypto), financial markets (e.g., US and China), data granularity (e.g., day-level and minute-level) and dataset sizes (e.g., small, medium and large-scale). We summarize statistics of these datasets in Table 1. We host all raw data and corresponding datasheets [23] accessible for users download both Google Drive and Baidu Cloud. We further offer scripts for calling API data providers, i.c., Yahoo Finance and AKShare, for users who desire to build personalized datasets. We also plan further expand collection of datasets in future updates. See Appendix D for more details on dataset description, collection, maintenance and usage.
Table Dataset statisties of market, frequency, asset amounts, size, chronological period and source
Dataset
Market
Freq # of Assets
Size
From
To
Source
DJ30
US Stock
day
29
72994
12/01/03
21/12/31
Yahoo
SP500
US Stock
day
363
2009568
00/01/01
22/01/01
Yahoo
Russell
US Stock
day
691
2391650
07/09/26
22/06/29
Yahoo
KDD17
US Stock
day
41
149407
07/09/26
22/06/29
Yahoo
ACL18
US Stock
day
74
264954
07/09/26
22/06/29
Yahoo
SSE50
China Stock
hour
26
134680
16/06/01
20/08/30
Yahoo
HSTech
HK Stock
day
30
60120
88/12/30
23/03/27
AKShare
HSI
HK Stock
day
72
206866
07/09/27
22/06/29
AKShare
Future
Future
5 min
20
20370
23/03/07
23/03/28
AKShare
FX
FX
day
22
110330
00/01/01
19/12/31
Kaggle
USDCNY
FX Crypto 1 day
day
5014
00/01/01
19/12/31
Kaggle
Crypto
1
2991
13/04/29
21/07/06
Kaggle
BTC
Crypto 1 min
17113
21/04/07
21/04/19
Binance
Preprocessing. We follow the DataOps paradigm [16, 4, 41] in this component to implement an efficient pipeline for financial data engineering. During data cleaning, we offer scripts to remove duplicated data and merge data from different sources into one unified format. addition, we apply conditional score-based diffusion model [71] to conduct data imputation for missing values. For technical indicators, Master supports both version 13 features [81, 69] and Alpha 158 [79]. which are widely used for research papers [19, 67] and industry scenarios, respectively. Multiple data

---

normalization methods are supported. where z-score serves as the default choice. In addition, users can choose to pick threshold that filters out valuable features with sufficiently large information coefficient on the return for feature selection. After preprocessing, the raw data is converted into high quality ready-to-use data that can be directly used to build realistic data-driven RI. environments.
Environments. We build data-driven market environments with real-world financial data following the de facto standard of OpenAl gym [6, 41] and the paradigm in [41, 53]. Each environment has three functions: reset () function that resets the environment back to the initial state so; step () function that takes an action a, of the agent and updates state from 8t to $1+1; reward () function computes the reward value generated by action incorporate many practical constraints (e.g.., transaction costs, slippage and non non-negative balance) for different scenarios, There are also options for leveraging, short and loss stopping.
To make the environments more realistic, we
Specifically, the current version supports 6 mainstream quantitative trading tasks [68]] For Crypto trading, investors continuously buy/sell cryptocurrencies (e.g., Bitcoin) to make profits. ii) For portfolio management, investors hold a pool of different financial assets and reallocate the proportion of capitals invested in each asset periodically to pursue stable long-term profits. iii) For intraday trading, investors finish the buy/sell long/short actions within the same trading day to capture the fleeting intraday opportunities and all positions are sold out on market price at the end of the day to avoid overnight risk. iv) Order execution is a micro-level trading task that finish the execution goal within a fixed time horizon (e.g., selling 10 thousand shares of Apple stock in 20 minutes) and try to minimize the costs the same time. v) Market making focuses on providing liquidity of the market by trading on both buy/sell sides at the same time. vi) High frequency trading utilizes micro-level information and trades in high-fryyuency to capture the tiny local price fluctuation.
Algorithms. Although there has been extensive works on RL algorithms for different trading scenarios [66, 77, 17], only tiny fraction of authors choose to make their code publicly available due to the intense competitions and zero-sum game essence in financial markets [72]. As far as we know, TradeMaster is the first open-source platform that offers standardized implementations of wide range of state-of cote-offthhh-eee -h-thhrrt RLFT algorithms. For algorithms with official source code, we convert the code into TradeMaster's 's code structures following the same networks. hyperparameters and other details. This condition fits for SARL [80], DeepTrader [77] and OPD [17]. For other algorithms without publicly available implementations, we reimplement them and try our best to maintain consistency based on the original papers. Brief descriptions of RLFT algorithms implemented in TradeMaster are
as follows:
SARL [80] proposes augteetd RL framework, which leverages the price movement prediction as additional states, based on deterministic policy gradient [62] methods.
DeepTrader (DT) [77] is a policy gradient based methods, which leverages both maximum drawdown and return rate as reward functions to balance between risk and profit.
EHIE [31] is considered as the first deep RLFT method with an ensemble of identical independent evaluators topology, a portfolio vector memory, and an online stochastic learning scheme.
Investor-Imitator (IMIT) [14] is an RL approach that imitates behaviors of different types of investors using investor-specific reward functions with a set of logic descriptions.
DeepScalper (DS) [66] is a risk-aware RL framework for intraday trading, which contains both micro-level and macro-level market embedding, hindsight reward function capture long-term trend, and a volatility prediction auxiliary task to model risk.
ETEO [39] is designed based on the classic proximal policy optimization (PPO) algorithm for order execution with a sparse reward function and market encoder using LSTM.
OPD [17] is developed based on ETEO [39] and further adds a policy distillation mechanism to learn from teacher trained based on perfect market information.
Besides advanced RLFT algorithms, TradeMaster also includes 9 efficient customized implementa- tions of classic RL algorithms based the widely used RLib library [37] including PPO [57], A2C [46], Rainbow [27], SAC [24], DDPG [38], DQN [45], DDQN [74], PG [70], TD3 [21] for various quantitative trading tasks.
Evaluations. To provide a systematic evaluation, TradeMaster offers a plethora of evaluation measures and visualization tools based on TRUDEX-Compase [67] to assess RLFT algorithms from
5

---

5 axes: profitability, risk-control, universality, diversity and reliability. Specifically, we implement range of financial metrics including profit metrics (e.g., return rate), risk-adjusted profit metrics (e.g.. Sharpe ratio [59]) and risk metrics (e.g., volatility and maximum drawdown). Besides evaluating on regular point-wise financial metrics, we further provide many robust measures and evaluation tools to pursue comprehensive evaluation. For instance, the performance profile [15] and rank distribution [1] plots are included as two unbiased and robust measures towards reliable RLFT methods. We apply t-SNE [73] and heatmap to show data-level and decision-level diversity in different financial markets. We also provide tools to evaluate RLFT methods under extreme market conditions with black swan events. See Appendix for more details.
Interfaces. TradeMaster provides multiple options for practical use. Specifically, we host an open- source Github repository, build Python package, serve an online web service with graphical user interface (GUT) offer dozens of hands-on Jupyter Notebook tutorials for quick start and develope cloud version using Colab to improve accessibility. More details on the usage of different user interfaces are available in Section
TradeMaster as an Empirical Benchmark
Evaluating any algorithm depends on its context. For instance, how well an RLFT algorithm ultimately performs depends on the choices of many factors such as datasets, preprocessing and evaluation measures. While current RLFT works typically seek to isolate individual gains through "all-else-equal" configurations in experiments, there often limited overlap in pipeline configurations across different studies. To promote transparency and comparability, TradeMaster aims to serve as structured evaluation framework by providing comprchensive RLFT empirical standard. Next, describe how we benchmark RLFT algorithms using TradeMaster with an demonstrative example of portfolio management task on US stock markets. We first introduce experimental setup (Section 4.1) and analyze the performance of RLFT algorithms (Section 4.2). To offer a more comprehensive analysis, we further report the results of an unbiased and robust measure called rank distribution plots (Section 4.3) and performance on extreme market conditions with black swan events (Section 4.4).
4.1 Experimental Setup
We conduct reproducible experiments of the classic portfolio management task stock pool of Dow Jones 30 index using the datasets in Table 1 to benchmark results of state-of-the-an RLFT algorithms. Due limited space, we briefly introduce the experimental setup in the following table with red outlines. More details are available in Appendix and the TradeMaster Github repository.
Table Performance comparison (mean of individual runs) on the US stock market of RLFT algorithms in terms of 8 financial metries. Pink and green indicate best and second best results.
TR(%)f
SR
CR
SoRt
MDD(%%
VOL(5))
ENTT
ENB
A2C
51.92
0.750
1.510
1.468
32.44
1.373
2.161
1.373
DDPG
56.95
0.800
1.614
1.562
32.37
1.372
2.667
1.265
EIE
49.66
0.756
1,526
1.468
30.86
1.358
3.368
1.110
IMIT
-4.95
0.102
0.240
0.234
50.52
2.078
1,460
2.149
SARL
52.13
0.752
1.544
1.490
32.23
1.420
2.758
1.192
TD3
57.15
0.804
1.616
1.564
32.36
1.373
2.608
1,214
PG
51.17
0.742
1.492
1.452
32.49
1.373
2.146
L,385
PPO
50.99
0.742
1.490
1,450
32,49
1.372
2.136
1.393
4.2
Performance Comparison
report the overall performance of 8 methods Table 2. TD3 achieves the best profitability with highest values in TR, SR, CR and SoR, while DDPG is also a good option with slightly lower performance. For RI. algorithms designed for trading (e.g.. EITE, IMIT and SARL), they fail to outperform properly tuned classic RI algorithms such as TD3 in terms of profitability. For risk- control, EIIE shows best performance with the lowest MDD and VOL. For diversity measures, EIIE
'htt://trademmsttr.ai

---

and IMIT perform the best for ENT and ENB, respectively. The overall performance of IMIT is much worse comparing to other methods. One possible reason is the lack of high-quality data of professional investmentors' behaviors [14]. Generally speaking, existing algorithm achieves dominating results on all financial metries and there are still vast opportunities to improve in this filed. In addition, we observe that classic RI. algorithms (e.g., DDPG and TD3) still outperform RLFT trading algorithms with proper hyperparameter tuning in terms of profit-related metrics (e.g.. TR, SR, CR and SoR). At the same time, RLFT algorithms (e.g., EIIE, IMIT and SARL) achieve portfolios with better risk-control and diversity, which demonstrate the effectiveness of the risk-related components. We hope the reproducible benchmark results from TradeMaster can help researchers easily notice the relative strength of existing algorithms and inspire the design of new algorithms.
Experimental Setup of Portfolio Management on US Stock Markets
We collect 10-year (2012-2021) historical prices of 29 influential stocks in the Dow Jones 30 index with top unit price as a strong assessment of the US stock market's overall tendency from Yahoo Finance.
Dataset:
Features: We generate 11 widely used temporal features based on historical price to describe the stock markets following [81, 67, 69]. Detailed formulas are available in Appendix F.
Preprocessing:
We conduct z-score normalization of each feature based on the mean and standard deviation of train, validation and test set, respectively. Input missing values are imputed using CSDI [71].
Training Setup:
We perform all experiments on one RTX 3090 GPU. Each algorithm is trained for 10 epochs for each random seed. We report the average performance of 5 random seeds. It takes about hour to train, validate and test on the US stock market.
Algorithms: We compare the performance of 8 state-of-the-art algorithms including classic RI. algorithms (A2C [46], DDPG [38], TD3 [21], PG [70] and PPO [57]) and 3 RL. algorithms designed for financial trading (EIIE [31], IMIT [14] and SARL [80]).
We apply financial metric including 1 profit metric: total return (TR), 3 risk-adjusted profit metrics: Sharpe ratio (SR) [59], Calmar ratio (CR) and Sortino ratio (SoR), 2 risk metrics: volatility (Vol) [61] and maximum drawdown (MDD) [43] and 2 diversity metries: entropy (ENT) [32] and effect number of bets (ENB) [34]. The mathematical definitions of these metrics are in Appendix E.
Financial Metries:
Hyperparameters: We first select the values of hyperparamters following two conditions: i) if there are authors' official or open-source RLFT library implementations, we apply the same hyperparameters for fair comparison since they applies for A2C, DDPG, TD3, PG, PPO and SARL. ii) there are no publicly available imple- mentations, reimplement the algorithms and try our best to maintain consistency based on the original papers. This applies for EIE and IMIT. Later on, we apply grid search on several key RL hyperparameters based on the Mrste codebase to further improve performance. Specificially, we try batch size in list [256. 512, 1024), hidden size in range [64, 128, 256] and learning rate in [3e [3e-2_5e 4 7,7e 49e "J for both actor and critic. Adam is used as the optimizer. More details on the hyperparamters are available in the TradeMaster GitHub repository.
tuned in financial domain. This condition
Data Split: We follow the rolling data split paradigm in [67] (see figure on the left). Phase three uses the last year for test, penultimate year for validation and the remaining of the dataset for train- ing. For phase one and two, their validation/test sets roll back one and two years.
Demonstration of Data Split:
1 year 1 year year
Phase
Phase 1. Training
vild
Test
Phose 2: Traniing
Phase 3: tnining
Vile
lsst
Statt
Time
4.3 Rank Distribution to Demonstrate the Rank of RLFT methods
In Figure 3, we plot the rank distribution plot [1] of RLFT methods in terms of TR, SR, VOL and Entropy across 3 test periods with results of 5 random seeds in each period. It is more robust to outliers comparing to the widely-used mean performance, where the i-th column in the rank distribution plot shows the probability that given method is assigned rank i in the corresponding
7

---

metrics. For x-axis, rank and 8 indicate the best and worst performance. For y-axis, the bar length of a given method on given metric with rank i corresponds to the fraction of rank it achieves across the 3 test periods
[MIT
and 5 random seeds (3 x 5 = 15 in total). For TR and SR, TD3 slightly outperforms DDPG with 24% and 27% probability to achieve the best performance. For Vol,
Frauctin lin
EIIE gets the overall best performance while TD3 goes through higher volatility. For ENT, EIlE significantly out- performs other methods with 100% probability for rank which demonstrates its ability to construct portfolios with enough diversity.
Figure 3: Rank distribution in terms of financial metrics the US stock market.
4.4
The Impact of Extreme Market Conditions
To further provide an evaluation of risk-control and reliability, we pick one extreme market period with black swan events during testing. In this case, the period is September 1-30 in 2021, when the US stock market is violate and goes through the largest decrease after COVID- 19 due to the concern
interest rate increase and the congressional shutdown. In Figure 4, we plot the bar chart TR and SR during the period of extreme market conditions. The red line indicates the market average. Classic RL methods (e.g., A2C, DDPG, TD3, PG, PPO) achieves similar performance comparing to the market average as they have the tendency to keep conservative during extreme market conditions. In con- trast, radical methods such as EllE and SARL are more suitable options with better performance [44]. This analy-
Awerage
sis on extreme market conditions can shed light on the de-
Total Return
sign of RLFT methods, which in line with economists' efforts understanding the financial markets. For in- stance, researchers may incorporate volatility- aware aux- iliary task [66] into the design of RLFT algorithms to make them be aware of extreme market conditions in advance and behave as profit-secking agents during extreme market conditions.
-1.25
AIC DDPS EIIE SAKL
PPO
Figure 4: Performance of RLFT methods during extreme market conditions.
Demonstrative Code of Master Platform
# Load Conf configuraiion File args parse args() cfg Config. fromfileeargss
Setup Loss and Transition Function criterion build loss(cfg) transition wulld_trransitio((ffg))
# Build Environments for Train/Valid/Test dataset build dataset(cfg)) train_env [cf___ee dataset, taske"train")) valid env build quiild_en((cff) dataset,
Build Reinforcement Learning Agent agent build build_aaeeee agent degeentccoo..efaatttaasssiiiiiie
# Build Trainer Based on Environments trainer bewiduutraaiiieeeeeeee wtw.ieee(cg..deeeeeeeee cooltttarg=dicct train_env=train. env, valid env=valid _nv,,
default_ irffautt_arg=dictte taske"valid"))
test_env = build_envich dataset, default_args=dicttt task="test"))
test_envstest_eny agent=agent))
The Procedure of Training and Validation trainer.traiie traii_and_walldde
# Setup Network and Optimizer net = buuld_net(cfg.eaee optimizer build bwwlddoptiiizerrr..eeeeeeeppeeee optimizer(cfg, default_args=dictt.]
The Procedure of Testing trainer.
TradeMaster as a User Interface
In this section, we introduce our efforts to make TradeMaster a great user interface. First, we show the clean and extendable code structure of TradeMaster with demonstrative example. Second, describe the multiple practical usage options for users from different background TradeMaster. Third, we discuss the AutoML component to further improve usability.
Demonstrative Code. The code snippet above shows the code of the whole pipeline in TradeMaster including 8 main steps (green code comments), which is simple, clean and casy to extend. We first load

---

the configuration file and setup the environment, network, optimizer, loss and transition function based on it. Then, we build RL agents and trainer, respectively. Later on, we call train_ande aasn__nndvvall valid () function for the training and validation of RI. agents. Finally, we call test () function to test the performance of RI. agents on various financial metrics.
Multiple Options for Practical Use. We provide different ways to use TradeMaster for users with different background and investment goals. First, we host Github repository for TradeMaster including source code, update log and supplementary materials. Second, we provide a Colab version of TradeMaster for users who prefer using cloud computation resources that can run directly. Third, we offer an online web GUI interface. Users can simply pick the settings they are interested in. By clicking on the button, they can train and evaluate RLFT algorithms in few minutes. This serve as an accessible interface for users with no need on any expert knowledge. Fourth, we provide dozens of Jupyter Notebook tutorials for educational purposes covering key features of Trater
Improving Usability with AL. To achieve decent performance, many knobs (e.g.. features and hyperparameters) need to be properly tuned for RLFT algorithms [68]. However, this requires in-depth knowledge of RL and coding, which is an insurmountable obstacle for many users without Al background. To make TradeMaster more accessible to these users, we implement an AutoML component [28] to finish the RLFT pipeline in a more automatic way. We aim to provide an interface that enables existing AutoML implementations to be conveniently plugged in. Specifically, we support automatic feature generation and feature selection by customizing the OpenFE [83] library, which firstly generates a set of new features by combining many useful operator (e.g., +, -. min, max) of raw data and then efficiently filters out top-ranked features following a synergistic two-stage evaluation module. In addition, we offer an automatic hyperparameter tuning interface based on Optuna library [2] that enables automatic search for optimal values of learning rate, exploration rate and hidden node amounts. More details are in Appendix G. We note that the current version based on [83, 2] shows the effectiveness of AutoML for the RLFT pipeline and our unified interface design enables easy extension with other advanced AutoML techniques.
Related Work and Uniqueness of TradeMaster
Quantitative Trading with Reinforcement Learning. In the early age, tabular RL is applied for financial trading [47, 48). Jiang et al. [31] make the first attempt of deep RL in financial markets. FDDR [13] and iRDPG [42] are designed to learn financial trading signals and mimic behaviors of professional traders for algorithmic trading, respectively. For portfolio management, deep RL methods are proposed to account for the impact of market risk [77] and commission fee [76]. A PPO-based framework [39] is proposed for order execution and a policy distillation mechanism is added to bridge the gap between imperfect market states and optimal execution actions [17]. There are also efforts on market making [65, 64], hedging [7]. high frequency trading [5] and Crypto trading [52].
Although many works have shown the potential of RL in financial trading, there is still a gap between academic research and real-world deployment. We hope the release of Mrster can move one step further toward deployment and facilitate the design of RLFT pipelines.
There has been many attempts financial trading platforms. In the early stages, many open-source libraries are proposed for traditional finance approaches [20], event-driven backtesting [55] and portfolio risk analysis [54]. With the advent of AI, recent financial trading platforms mostly focus on data-driven machine learning techniques. OLPS [36] presents a toolbox for online portfolio selection that implements collection of strategies powered by machine learning. Qlib [79] proposes an Al-oriented quantitative investment platform with a focus on prediction- based supervised learning settings. FinRL [40] makes the first attempt on developing RL-based quantitative trading platform. FinRL-Meta [41] offers a series of data-driven RL environments for various trading scenarios.
Trading Platforms.
However, due to the existence of alpha decay phenomenon [22], financial practitioners are never keen to share source code of their algorithms and platforms [79] that achieve good performance in real-world settings. Mrster is proposed to serve as a first-of-its-kind best-in-class platform with a focus on RL into the family of ooen-source trading platforms.

---

Uniqueness of TradeMaster. As shown in Table 3, we compare TradeMaster with existing trading platforms [79, 40] to show its uniqueness as follows: i) while existing platforms claims support of various markets, they only offers links to many data providers that users have to write scripts to acquire data flexibly. TradeMaster prepares 13 well-processed datasets and serve them on Google Drive to be downloaded directly. ii) Master covers wide range of trading scenarios with realistic simulation under practical constraints. As far as we know, many scenarios (e.g., intraday trading and market making) are not covered by any existing platforms. iii) TradeMaster provides high-quality implementations of classic RL algorithms and RL for trading algorithms, where about half of them is not covered by existing platforms. iv) Mrster include series of evaluation measures and visualization toolkits to provide a systematic evaluation instead of pure financial metrics in existing platforms.
We build GUI interface the official TradeMaster website for users without in-depth coding skills to explore the potential of RLFT. vi) We include an AutoMI. module for automatic feature selection and hyperparameter tuning to further improve usability.
Table 3: Comparison of TrateMaster and existing trading platforms. indicates "the number of".
# financial
RLFT algorithms
# trading # evaluatton plot toolkits interface AutoML
GUI
Platform
markets
scenarios metrics
Qlib [79]
2
1
4
x
FinRL [40]
3
5
x
TradeMaster
13
16
6
10
6
Discussion
Potential Impacts. We hope that TradeMaster can facilitate the development of the RLFT domain with broad impact on both academic researchers and financial practitioners. For RLFT researchers, they can rapidly implement and compare their own methods with focus real scientific problems instead of engineering details. For RL researchers, TradeMaster introduces many challenging trading financial markets. For finance researchers and
scenarios for the test of novel RL algorithms individual investors, we can have a taste on RL-based trading methods without in-depth knowledge of Al and coding. For professional trading firms, Maser serve the code base enhance their internal trading systems with advanced RL techniques.
Auxiliary Information. Due to space limitations, include some auxiliary yet important infor- mation in the Appendix. We introduce the software documents Appendix C.2, dataset details in Appendix D, official website in Appendix C.1, usage of online GUI in Appendix descriptions evaluation measures in Appendix E, usage of the AutoML component in Appendix G, setup details to reproduce the benchmarking experiments in Appendix F, license and future plan in Appendix H.
Desiderata. Machines will never fully replace investors' understanding the complex economic world, nor researchers' effort designing novel RL algorithms [29]. TradeMaster enables rapid prototyping, benchmarking and deployment of RLFT procedures, so that investors concentrate on discovering the essential laws of financial markets and researchers can spend more time on technical challenges. To help grease the wheels, we hope the release of TradeMaster can facilitate the RLFT research and serve bridge between finance and AI for broader interdisciplinary impact.
8
Acknowledgements
This project is supported by the National Research Foundation, Singapore under its Industry Align- Fund Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
10

---

References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
[2] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generatiog hyperparameter optimization framework. In ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 2223, 2019.
[3] Bo An, Shuo Sun, and Rundong Wang. Deep reinforcement learning for quantitative trading: Challenges and opportunities. IEEE Intelligent Systems, 37(2):23-26, 2022.
[4] Harvinder Atwal. Practical DataOps: Delivering agile data science at scale. Springer, 2019.
[5] Antonio Briola, Jeremy Turiel, Riccardo Marcaccioli, and Tomaso Aste. Deep reinforcement learning for active high frequency trading. arXiv preprint urXiv:2101.07 2021.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:16066 000066015400 5400 2016.
[7] Jay Cao, Jacky Chen, John Hull, and Zissis Poulos. Deep dedging of derivatives using reinforce- ment learning. The Journal of Financial Data Science, 3(1): 2021.
[8] Ernest P Chan. Quantitative trading: how to build your own algorithmic trading business. John Wiley & Sons, 2021.
[9] Andrea Coletta, Aymeric Moulin, Svitlana Vyetrenko, and Tucker Balch. Learning to simulate realistic limit order book markets from data world agent. In ACM International Conference on Al in Finance, pages 428-436 2022.
[10] Can Cui, Wei Wang, Meihui Zhang, Gang Chen, Zhaojing Luo, and Beng Chin Ooi. Al- phaevolve: learning framework discover novel alphas in quantitative investment. In International Conference Management of Data, pages 2208 2208-221,, 2021.
[11] Marcos Lopez De Prado. Advances in financial machine learning. John Wiley Sons, 2018.
[12] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897): 414 419, 2022.
[13] Yue Deng. Feng Bao, Youyong Kong. Zhiquan Ren, and Qionghai Dai. Deep direct reinforce- ment learning for financial signal representation and trading. IEEE Transactions on Neural Networks and Learning Systems, (8(3):653-664 2016.
[14] Yi Ding, Weiqing Liu, Jiang Bian, Daoqiang Zhang, and Tie-Yan Liu. Investor-imitator: A framework for trading knowledge extraction. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), 2018.
[15] Elizabeth D Dolan and Jorge J Moré. Benchmarking optimization software with performance profiles. Mathematical Programming, 91(2):201-213, 2002.
[16] Julian Ereth. Dataops-towards a definition. LWDA, 2191:104-112,, 2018.
[17] Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie- Yan Liu. Universal trading for order execution with oracle policy distillation. In AAAI Conference Artificial Intelligence, pages 107-115, 2021.
[18] Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Parede, Mohammadanin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszez, et al. Discovering faster matrix multiplication algorithms with reinforcement learning, Nature, 610(7930):47-533 2022.

---

[19] Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, and Tat-Seng Chua. Enhancing stock movement prediction with adversarial training. arXiv preprint 17Xi::1110009936, 2018.
[20] N Firth. Why use quantlib. URL http://www. maths. ox. ac. uk/ firth/reseercchquunliii pdf. 2004.
[21] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning, pages 1587-1596, 2018.
[22] Nicolae Garleanu and Lasse Heje Pedersen. Dynamic trading with predictable returns and transaction costs. The Journal of Finance, (8(6):2309-23444 2013.
[23] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86-99) 92, 2021.
[24] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870, 2018.
[25] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI conference on artificial intelligence, 2018.
[26] Mario Hermann, Tobias Pentek, and Boris Otto. Design principles for industrie 4.0 scenarios. In 2016 49th Hawaii International Conference on System Sciences (HICSS), pages 2016.
[27] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning, In AAAI Conference on Arlificial Intelligence, 2018.
[28] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, systems, challenges. Springer Nature, 2019.
[29] Daniel Jarrett, Jinsung Yoon, Ioana Bica, Zhaozhi Qian, Ari Ercole, and Mihaela van der Schaar. Clairvoyance: A pipeline toolkit for medical time series. International Conference on Learning Representations, 2021.
[30] Zhengyao Jiang and Jinjun Liang. Cryptocurrency portfolio management with deep reinforce- ment learning. In 2017 Intelligent Systems Conference (IntelliSys), pages 905-913, 2017.
[31] Zhengyao Jiang, Dixing Xu, and Jinjun Liang. A deep reinforcement learning framework for the financial portfolio management problem. arXiv preprint arXiv: 10059, 2017.
[32] Lou Jost. Entropy and diversity. Oikos, 13(())363-335, 2006.
[33] Mariam Kiran and Melis Ozyildirim. Hyperparameter tuning for deep reinforcement learning applications. arXiv preprint arXiv:2201.11182, 2022.
[34] Ulrich Kirchner and Caroline Zunckel. Measuring portfolio diversification. arXiv preprint 112iv:1102.4722. 2011.
[35] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie. Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark of in-the-wild distribution shifts. International Conference on Machine Learning, pages 5637-5664, 2021.
[36] Bin Li, Doyen Sahoo, and Steven CH Hoi. Olps: a toolbox for on-line portfolio selection. Journal of Machine Learning Research, 17(1):142-11266 2016.
12

---

[37] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning, pages 3053-3062, 2018.
[38] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Brez, Yuval Tassa, David Silver, and Daan Wienstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 02977, 2015.
[39] Siyu Lin and Peter A Beling. An end-to-end optimal trade execution framework based on proximal policy optimization. In International Joint Conference Artificial Intelligence, pages 2020.
[40] Xiao- Yang Liu, Hongyang Yang, Jiechao Gao, and Christina Dan Wang. Finrl: Deep reinforce- ment learning framework to automate trading quantitative finance. In ACM International Conference on Al in Finance, pages 1-9, 2021.
[41] Xiao-Yam Yang Liu, Ziyi Xia, Jingyang Rui, Jiechao Gao, Hongyang Yang, Ming Zhu, Christina Dan Wang, Zhaoran Wang, and Jian Guo. FinRL-Meta: Market environments and benchmarks for data-driven financial reinforcement learning. In Advances in Neural Information Processing Systems (Datasets and Benchmarky Track), 2022.
[42] Yang Liu, Qi Liu, Hongke Zhao, Zhen Pan, and Chuanren Liu. Adaptive quantitative trading: An imitative deep reinforcement learning approach. In AAAI Conference on Artificial Intelligence, pages 2128 283-1135,22 2020.
[43] Malik Magdon-Isaa Ismail and Amir Atiya. Maximum drawdown. Risk Magazine, 17(10)999-102. 2004.
[44] Velayoudoum Marimoutou, Bechir Raggad, and Abdelwahed Trabelsi. Extreme value theory and value at risk: application to oil market. Energy Economics, 31(4):519 -30- 2009.
[45] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5666 2013.
[46] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep rein- forcement learning. In International Conference on Machine Learning, pages 1928-1933 2016.
[47] Ralph Neuneier. Optimal asset allocation using adaptive dynamic programming. In Advances in Neural Information Processing Systems, pages 952-958, 1996.
[48] Yuriy Nevmyvaka, Yi Feng, and Michael Kearns. Reinforcement learning for optimized trade execution. In International Conference on Machine Learning (ICML), pages 673-680, 2006.
[49] Marco Pagano. Financial markets and growth: an overview. European Economic Review, 37 (2-3):613-622, 1993.
[50] Feiyang Pan, Tongzhe Zhang, Ling Luo, Jia Hc, and Shuoling Liu. Learn continuously, act discretely: Hybrid action-space reinforcement learning for optimal execution. In International Joint Conference on Artificial Intelligence, pages 3912-3918, 2022.
[51] Robert Pardo. The evaluation and optimization of trading strategies, volume 314. John Wiley & Sons, 2011.
[52] Molei Qin, Shuo Sun, Wentao Zhang, Haochong Xia, Xinrun Wang, and Bo An. Earn- hft: Efficient hierarchical reinforcement learning for high frequency trading. arXiv preprint arXiv:2309. /289/ 2023.
[53] Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: near real-world benchmark for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:24753-24766 2022.
13

---

[54] Quantopian. http://ggthab.commuuuuuunppppppp
Github, 2015.
[55] Quantopian. http://gghhbbbccomqqquuppppppppp Github, 2016.
[56]
Ramit Sawhney, Arnv Wadhwa, Shivam Agarwal, and Rajiv Shah. Quantitative day trading from natural language using reinforcement learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4018 4030,, 2021.
[57] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06333 2017.
[58] David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips,, Dietmar Ebner, Vinay Chaudhary, Michael Young. -rancoiac Crespo, and Dan Dennison. Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems, 28, 2015.
[59] William Sharpe. The sharpe ratio. Journal of Portfolio Management, pages 169-185, 1998.
[60] Zijian Shi and John Cartlidge. State dependent parallel neural hawkes process for limit order book event stream prediction and simulation. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1607-1615, 2022.
[61] Robert Shiller. Market volatility. MIT press, 1992.
[62] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning. pages 387-395, 2014.
[63] David Silver, Julian Schrittiisser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354 359, 2017.
[64] Thomas Spooner and Rahul Savani. Robust market making via adversarial reinforcement learning. In International Joint Conference Artificial Intelligence, pages 4590-4500 4596, 2020.
[65] Thomas Spooner, John Fearnley, Rahul Savani, and Andreas Koukorinis. Market making via reinforcement learning. In International Conference on Autonomous Agents and MultiAgent Systems, page 442, 2018
[66] Shuo Sun, Wanqi Xue, Rundong Wang, Xu He, Junlei Zhu, Jian Li, and Bo An. Deepscalper: A risk-aware reinforcement learning framework capture fleeting intraday trading opportunities. In ACM International Conference on Information & Knowledge Management, pages 1858-1867, 2022.
[67] Shuo Sun, Molei Qin, Xinrun Wang, and Bo An. PRUDEX compas:: Towards systematic evaluation of reinforcement learning in financial markets. Transactions on Machine Learning Research, 2023.
[68] Shuo Sun, Rundong Wang, and Bo An. Reinforcement learning for quantitative trading. ACM Transactions on Intelligent Systems and Technology. 2023.
[69] Shuo Sun, Xinrun Wang, Wanqi Xue, Xiaoxuan Lou, and Bo An. Mastering stock markets with efficient mixture of diversified trading experts. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, page 2109-2119, 2023.
[70] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999.
[71] Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804-2481 2021.
[72] Stuart M Tumbull. Swaps: A zero sum game? Financial Management, pages 15-21, 1987.
14

---

[73] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(1 2008.
[74] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-leamnng, In AAAI Conference on Arlificial Intelligence (AAAI), volume 30, 2016.
[75] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in stareraft using multi-agent reinforcement learning. Nature, 575(7782)3350-354 2019.
[76] Rundong Wang, Hongxin Wei, Bo An, Zhouyan Feng, and Jun Yao. Commission fee is not enough: A hierarchical reinforced framework for portfolio management. In AAAI Conference on Artificial Intelligence (AAAT), 2021.
[77] Zhicheng Wang, Biwei Huang, Shikui Tu, Kun Zhang, and Lei Xu. Deeptrader: A deep reinforcement learning approach to risk-return balanced portfolio management with market conditions embedding. In AAAI Conference on Artificial Intelligence, 2021.
[78] Hongyang Yang, Xiao Yang Liu, Shan Zhong, and Anwar Walid. Deep reinforcement learning for automated stock trading: An ensemble strategy. In ACM International Conference Al in Finance, pages 1-8, 2020.
[79] Xiao Yang, Weiqing Liu, Dong Zhou, Jiang Bian, and Tie- Yan Liu. Qlib: An ai-oriented quantitative investment platform. arXiv preprint arXiv:2009.1 11 1899 2020.
[80] Yunan Ye, Hengzhi Pei, Boxin Wang, Pin- Yu Chen, Yada Zhu, Ju Xiao, and Bo Li. Reinforcement-eerming based portfolio management with augmented asset movement pre- diction states. AAAI Conference on Artificial Intelligence, pages 1112-1119, 2020.
[81] Jaemin Yoo, Yejun Soun, Yong-chan Park, and Kang. Accurate multivariate stock movement prediction via data-axis transformer with multi-level contexts. In ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 2037-2045, 2021.
[82] Tianping Zhang, Yuanqi Li, Yifei Jin, and Jian Li. Autoalpha: an efficient hierarchical evolutionary algorithm for mining alpha factors in quantitative investment. arXiv preprint arXiv:2002.08225,, 2020.
[83] Tianping Zhang, Zheyu Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Wei Cao, and Jian Li. Openfe: Automated feature generation beyond expert-level performance. arXiv preprint arXiv:2211.12500 2022.
15


Reviews:
[{"review_id": "T8gdjR9Aot", "reviewer": "Reviewer_Vtxm", "summary": "", "questions": "", "limitations": "I think the paper can do a lot better as an open-sourced project. There are lots of RL for QT projects out there. The authors didn't clearly state why RLFT is better than the other.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "This project is open-sourced with functional API,a Python package, an online web service with a graphical user interface (GUI), multiple Jupyter Notebook tutorials and a cloud version using Colab. The setup is good for the initial stage of QT", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "SiLokLxj9J", "reviewer": "Reviewer_DHLR", "summary": "", "questions": "", "limitations": "Please see in above. ", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is clearly written in terms of the challenges and contributions mentioned. It seems that the authors provide a well-constructed platform to implement reinforcement learning in financial data. To demonstrate the platform, the authors provide a clear review about the basic concepts in RL, which helps readers to better understand the scope of the paper. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "sp3dFDNj3z", "reviewer": "Reviewer_k4uA", "summary": "", "questions": "", "limitations": "Platforms like TradeMaster possess significant potential to advance research in the relevant community. However, a notable challenge is attracting a broader pool of researchers to leverage this platform effectively. It would be insightful to delve into the authors' perspectives and analyses regarding strategies to overcome this bottleneck and enhance the platform's impact and reach.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Significance of the contribution: The aim of creating an all-encompassing platform is notable, which tries to provid a one-stop solution for academic researchers and industry practitioners alike. The developed platform is a valuable toolset for RLFT applications.\n\nQuality of the research: Quite impressive, demonstrating a substantial amount of work. This reflects the significant effort and dedication invested by the author team into the project.\n\nRelevance: Yes, relevant to the Datasets and Benchmarks Track\n\nClarity of paper: The paper is well-structured and articulate, with clear exposition of the TradeMaster platform and its functionalities.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "A1JDbo5En3", "reviewer": "Reviewer_fVic", "summary": "", "questions": "", "limitations": "1. As a software, TradeMaster is similar to [4,5].\n2. As a benchmark, although there are many baseline RL algorithms evaluated, this paper does not provide enough experiments, including hyper-parameters, training epochs, and standard deviations.\n\n[4] Xiao Yang, Weiqing Liu, Dong Zhou, Jiang Bian, and Tie-Yan Liu. Qlib: An ai-oriented quantitative investment platform. arXiv preprint arXiv:2009.11189, 2020.\n[5] Xiao-Yang Liu, Hongyang Yang, Jiechao Gao, and Christina Dan Wang. Finrl: Deep reinforcement learning framework to automate trading in quantitative finance. In ACM International Conference on AI in Finance, pages 1–9, 2021.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. As pointed out in the challenges, the work of implementating details occupies >90% in the whole pipeline, and <10% of work on core technical questions. This RLFT platform will fill the gap between laboratory financial algorithms and real-world financial markets, providing users a useful toolkit to practically and conveniently evaluate their new RL techniques and models.\n2. This platform provides a convenient tool to fairly compare different RLFT techniques with the same evaluation conditions, including 13 real-world financial datasets and 6 mainstream quantitative trading tasks.\n3. As a benchmark, this paper provides comprehensive comparison of 15 popular RLFT algorithms.\n4. Considering the impact of extreme market conditions makes the benchmark more comprehensive.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors present an open-source platform that aims to tackle the challenges of reinforcement learning in financial trading (RLFT).

The reviewers generally agreed that the contribution is significant, and that the work will be a valuable tool for RLFT applications.
The reviewers especially noted the strong execution of this work, and the clarity of the paper.

While there were initial concerns, the authors put in significant effort during the author response period, and in the end all reviewers were positive about the paper. I recommend acceptance.
Decision: Accept (Poster)