SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We report the overall performance of 8 methods in Table 2. TD3 achieves the best profitability with highest values in TR, SR, CR and SoR, while DDPG is also a good option with slightly lower performance. For RL algorithms designed for trading (e.g., EIIE, IMIT and SARL), they fail to outperform properly tuned classic RL algorithms such as TD3 in terms of profitability. For risk-control, EIIE shows best performance with the lowest MDD and VOL. For diversity measures, EIIE
<sup>1</sup> http://trademaster.ai
and IMIT perform the best for ENT and ENB, respectively. The overall performance of IMIT is much worse comparing to other methods. One possible reason is the lack of high-quality data of professional investors' behaviors [14]. Generally speaking, no existing algorithm achieves dominating results on all financial metrics and there are still vast opportunities to improve in this filed. In addition, we observe that classic RL algorithms (e.g., DDPG and TD3) still outperform RLFT trading algorithms with proper hyperparameter tuning in terms of profit-related metrics (e.g., TR, SR, CR and SoR). At the same time, RLFT algorithms (e.g., EIIE, IMIT and SARL) achieve portfolios with better risk-control and diversity, which demonstrate the effectiveness of the risk-related components. We hope the reproducible benchmark results from TradeMaster can help researchers easily notice the relative strength of existing algorithms and inspire the design of new algorithms.
Experimental Setup of Portfolio Management on US Stock Markets
Dataset: We collect 10-year (2012-2021) historical prices of 29 influential stocks in the Dow Jones 30 index with top unit price as a strong assessment of the US stock market's overall tendency from Yahoo Finance.
Features: We generate 11 widely used temporal features based on historical price to describe the stock markets following [81, 67, 69]. Detailed formulas are available in Appendix F.
Preprocessing: We conduct z-score normalization of each feature based on the mean and standard deviation of train, validation and test set, respectively. Input missing values are imputed using CSDI [71].
Training Setup: We perform all experiments on one RTX 3090 GPU. Each algorithm is trained for 10 epochs for each random seed. We report the average performance of 5 random seeds. It takes about 1 hour to train, validate and test on the US stock market.
Algorithms: We compare the performance of 8 state-of-the-art algorithms including 5 classic RL algorithms (A2C [46], DDPG [38], TD3 [21], PG [70] and PPO [57]) and 3 RL algorithms designed for financial trading (EIIE [31], IMIT [14] and SARL [80]).
Financial Metrics: We apply 8 financial metric including 1 profit metric: total return (TR), 3 risk-adjusted profit metrics: Sharpe ratio (SR) [59], Calmar ratio (CR) and Sortino ratio (SoR), 2 risk metrics: volatility (Vol) [61] and maximum drawdown (MDD) [43] and 2 diversity metrics: entropy (ENT) [32] and effect number of bets (ENB) [34]. The mathematical definitions of these metrics are in Appendix E.
Hyperparameters: We first select the values of hyperparameters following two conditions: i) if there are authors' official or open-source RLFT library implementations, we apply the same hyperparameters for a fair comparison since they are tuned in financial domain. This condition applies for A2C, DDPG, TD3, PG, PPO and SARL. ii) if there are no publicly available implementations, we reimplement the algorithms and try our best to maintain consistency based on the original papers. This applies for EIIE and IMIT. Later on, we apply grid search on several key RL hyperparameters based on the TradeMaster codebase to further improve performance. Specifically, we try batch size in list [256, 512, 1024], hidden size in range [64, 128, 256] and learning rate in  $[3e^{-4}, 5e^{-4}, 7e^{-4}, 9e^{-4}]$  for both actor and critic. Adam is used as the optimizer. More details on the hyperparameters are available in the TradeMaster GitHub repository.
Data Split: We follow the rolling data split paradigm in [67] (see figure on the left). Phase three uses the last year for test, penultimate year for validation and the remaining of the dataset for training. For phase one and two, their validation/test sets roll back one and two years.
Demonstration of Data Split:
4.3 Rank Distribution to Demonstrate the Rank of RLFT methods
In Figure 3, we plot the rank distribution plot [1] of 8 RLFT methods in terms of TR, SR, VOL and Entropy across 3 test periods with results of 5 random seeds in each period. It is more robust to outliers comparing to the widely-used mean performance, where the  $i$ -th column in the rank distribution plot shows the probability that a given method is assigned rank  $i$  in the corresponding
metrics. For x-axis, rank 1 and 8 indicate the best and worst performance. For y-axis, the bar length of a given method on a given metric with rank  $i$  corresponds to the % fraction of rank  $i$  it achieves across the 3 test periods and 5 random seeds ( $3 \times 5 = 15$  in total). For TR and SR, TD3 slightly outperforms DDPG with  $24\%$  and  $27\%$  probability to achieve the best performance. For Vol, EIIE gets the overall best performance while TD3 goes through higher volatility. For ENT, EIIE significantly outperforms other methods with  $100\%$  probability for rank 1, which demonstrates its ability to construct portfolios with enough diversity.
Figure 3: Rank distribution in terms of 4 financial metrics on the US stock market.
4.4 The Impact of Extreme Market Conditions
To further provide an evaluation of risk-control and reliability, we pick one extreme market period with black swan events during testing. In this case, the period is September 1-30 in 2021, when the US stock market is violate and goes through the largest decrease after COVID-19 due to the concern on interest rate increase and the congressional shutdown. In Figure 4, we plot the bar chart of TR and SR during the period of extreme market conditions. The red line indicates the market average. Classic RL methods (e.g., A2C, DDPG, TD3, PG, PPO) achieves similar performance comparing to the market average as they have the tendency to keep
conservative during extreme market conditions. In contrast, radical methods such as EIIE and SARL are more suitable options with better performance [44]. This analysis on extreme market conditions can shed light on the design of RLFT methods, which is in line with economists' efforts on understanding the financial markets. For instance, researchers may incorporate volatility-aware auxiliary task [66] into the design of RLFT algorithms to make them be aware of extreme market conditions in advance and behave as a profit-seeking agents during extreme market conditions.
Figure 4: Performance of RLFT methods during extreme market conditions.
Demonstrative Code of TradeMaster Platform
# Load Configuration File
args = parse_args()
cfg = Config.fromfile(args.config) # Setup Loss and Transition Function
criterion = build_losscfg)
transition = build_transitioncfg)
# Build Environments for Train/Valid/Test
dataset = build_datasetcfg)
train_env = build_envcfg, dataset,
default_args=dict,..., task="train")
valid_env = build_envcfg, dataset,
default_args=dict,..., task="valid")
test_env = build_envcfg, dataset,
default_args=dict,..., task="test")
# Setup Network and Optimizer
net = build_netcfg.act)
optimizer = build_OPTimizer.cfg,
default_args=dict(...)) # The Procedure of Testing and Validation
trainer.train_and_valid()
# The Procedure of Testing
trainer.test()
5 TradeMaster as a User Interface
In this section, we introduce our efforts to make TradeMaster a great user interface. First, we show the clean and extendable code structure of TradeMaster with a demonstrative example. Second, we describe the multiple practical usage options for users from different background in TradeMaster. Third, we discuss on the AutoML component to further improve usability.
**Demonstrative Code.** The code snippet above shows the code of the whole pipeline in TradeMaster including 8 main steps (green code comments), which is simple, clean and easy to extend. We first load
the configuration file and setup the environment, network, optimizer, loss and transition function based on it. Then, we build RL agents and trainer, respectively. Later on, we call train_and_valid() function for the training and validation of RL agents. Finally, we call test() function to test the performance of RL agents on various financial metrics.
Multiple Options for Practical Use. We provide different ways to use TradeMaster for users with different background and investment goals. First, we host a Github repository for TradeMaster including source code, update log and supplementary materials. Second, we provide a Colab version of TradeMaster for users who prefer using cloud computation resources that can run directly. Third, we offer an online web GUI interface. Users can simply pick the settings they are interested in. By clicking on the button, they can train and evaluate RLFT algorithms in a few minutes. This serve as an accessible interface for users with no need on any expert knowledge. Fourth, we provide dozens of Jupyter Notebook tutorials for educational purposes covering key features of TradeMaster.
Improving Usability with AutoML. To achieve decent performance, many knobs (e.g., features and hyperparameters) need to be properly tuned for RLFT algorithms [68]. However, this requires in-depth knowledge of RL and coding, which is an insurmountable obstacle for many users without AI background. To make TradeMaster more accessible to these users, we implement an AutoML component [28] to finish the RLFT pipeline in a more automatic way. We aim to provide an interface that enables existing AutoML implementations to be conveniently plugged in. Specifically, we support automatic feature generation and feature selection by customizing the OpenFE [83] library, which firstly generates a set of new features by combining many useful operator (e.g., +, -, min, max) of raw data and then efficiently filters out top-ranked features following a synergistic two-stage evaluation module. In addition, we offer an automatic hyperparameter tuning interface based on Optuna library [2] that enables automatic search for optimal values of learning rate, exploration rate and hidden node amounts. More details are in Appendix G. We note that the current version based on [83, 2] shows the effectiveness of AutoML for the RLFT pipeline and our unified interface design enables easy extension with other advanced AutoML techniques.
6 Related Work and Uniqueness of TradeMaster
Quantitative Trading with Reinforcement Learning. In the early age, tabular RL is applied for financial trading [47, 48]. Jiang et al. [31] make the first attempt of deep RL in financial markets. FDDR [13] and iRDPG [42] are designed to learn financial trading signals and mimic behaviors of professional traders for algorithmic trading, respectively. For portfolio management, deep RL methods are proposed to account for the impact of market risk [77] and commission fee [76]. A PPO-based framework [39] is proposed for order execution and a policy distillation mechanism is added to bridge the gap between imperfect market states and optimal execution actions [17]. There are also efforts on market making [65, 64], hedging [7], high frequency trading [5] and Crypto trading [52].
Although many works have shown the potential of RL in financial trading, there is still a gap between academic research and real-world deployment. We hope the release of TradeMaster can move one step further toward deployment and facilitate the design of RLFT pipelines.
Trading Platforms. There has been many attempts on financial trading platforms. In the early stages, many open-source libraries are proposed for traditional finance approaches [20], event-driven backtesting [55] and portfolio risk analysis [54]. With the advent of AI, recent financial trading platforms mostly focus on data-driven machine learning techniques. OLPS [36] presents a toolbox for online portfolio selection that implements a collection of strategies powered by machine learning. Qlib [79] proposes an AI-oriented quantitative investment platform with a focus on prediction-based supervised learning settings. FinRL [40] makes the first attempt on developing an RL-based quantitative trading platform. FinRL-Meta [41] offers a series of data-driven RL environments for various trading scenarios.
However, due to the existence of alpha decay phenomenon [22], financial practitioners are never keen to share source code of their algorithms and platforms [79] that achieve good performance in real-world settings. TradeMaster is proposed to serve as a first-of-its-kind best-in-class platform with a focus on RL into the family of open-source trading platforms.
Uniqueness of TradeMaster. As shown in Table 3, we compare TradeMaster with existing trading platforms [79, 40] to show its uniqueness as follows: i) while existing platforms claims support of various markets, they only offers links to many data providers that users have to write scripts to acquire data flexibly. TradeMaster prepares 13 well-processed datasets and serve them on Google Drive to be downloaded directly. ii) TradeMaster covers a wide range of trading scenarios with realistic simulation under practical constraints. As far as we know, many scenarios (e.g., intraday trading and market making) are not covered by any existing platforms. iii) TradeMaster provides high-quality implementations of 9 classic RL algorithms and 7 RL for trading algorithms, where about half of them is not covered by existing platforms. iv) TradeMaster include a series of evaluation measures and visualization toolkits to provide a systematic evaluation instead of pure financial metrics in existing platforms. v) We build a GUI interface on the official TradeMaster website for users without in-depth coding skills to explore the potential of RLFT. vi) We include an AutoML module for automatic feature selection and hyperparameter tuning to further improve usability.
Table 3: Comparison of TradeMaster and existing trading platforms. # indicates "the number of".
Platform	# financial markets	# RLFT algorithms	# trading scenarios	# evaluation metrics	# plot toolkits	GUI interface	AutoML
Qlib [79]	2	1	1	4	-	×	×
FinRL [40]	2	8	3	5	-	×	×
TradeMaster	13	16	6	10	6	√	√
Potential Impacts. We hope that TradeMaster can facilitate the development of the RLFT domain with a broad impact on both academic researchers and financial practitioners. For RLFT researchers, they can rapidly implement and compare their own methods with a focus on real scientific problems instead of engineering details. For RL researchers, TradeMaster introduces many challenging trading scenarios for the test of novel RL algorithms in financial markets. For finance researchers and individual investors, we can have a taste on RL-based trading methods without in-depth knowledge of AI and coding. For professional trading firms, TradeMaster can serve as the code base to enhance their internal trading systems with advanced RL techniques.
Auxiliary Information. Due to space limitations, we include some auxiliary yet important information in the Appendix. We introduce the software documents in Appendix C.2, dataset details in Appendix D, official website in Appendix C.1, usage of online GUI in Appendix B, descriptions on evaluation measures in Appendix E, usage of the AutoML component in Appendix G, setup details to reproduce the benchmarking experiments in Appendix F, license and future plan in Appendix H.
Desiderata. Machines will never fully replace investors' understanding of the complex economic world, nor researchers' effort on designing novel RL algorithms [29]. TradeMaster enables rapid prototyping, benchmarking and deployment of RLFT procedures, so that investors concentrate on discovering the essential laws of financial markets and researchers can spend more time on core technical challenges. To help grease the wheels, we hope the release of TradeMaster can facilitate the RLFT research and serve as a bridge between finance and AI for broader interdisciplinary impact.
This project is supported by the National Research Foundation, Singapore under its Industry Alignment Fund – Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
[73] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008.
[74] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In AAAI Conference on Artificial Intelligence (AAAI), volume 30, 2016.
[75] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
[76] Rundong Wang, Hongxin Wei, Bo An, Zhouyan Feng, and Jun Yao. Commission fee is not enough: A hierarchical reinforced framework for portfolio management. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
[77] Zhicheng Wang, Biwei Huang, Shikui Tu, Kun Zhang, and Lei Xu. Deeptrader: A deep reinforcement learning approach to risk-return balanced portfolio management with market conditions embedding. In AAAI Conference on Artificial Intelligence, 2021.
[78] Hongyang Yang, Xiao-Yang Liu, Shan Zhong, and Anwar Walid. Deep reinforcement learning for automated stock trading: An ensemble strategy. In ACM International Conference on AI in Finance, pages 1-8, 2020.
[79] Xiao Yang, Weiqing Liu, Dong Zhou, Jiang Bian, and Tie-Yan Liu. Qlib: An ai-oriented quantitative investment platform. arXiv preprint arXiv:2009.11189, 2020.
[80] Yunan Ye, Hengzhi Pei, Boxin Wang, Pin-Yu Chen, Yada Zhu, Ju Xiao, and Bo Li. Reinforcement-learning based portfolio management with augmented asset movement prediction states. In AAAI Conference on Artificial Intelligence, pages 1112-1119, 2020.
[81] Jaemin Yoo, Yejun Soun, Yong-chan Park, and U Kang. Accurate multivariate stock movement prediction via data-axis transformer with multi-level contexts. In ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2037-2045, 2021.
[82] Tianping Zhang, Yuanqi Li, Yifei Jin, and Jian Li. Autoalpha: an efficient hierarchical evolutionary algorithm for mining alpha factors in quantitative investment. arXiv preprint arXiv:2002.08245, 2020.
[83] Tianping Zhang, Zheyu Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Wei Cao, and Jian Li. Openfe: Automated feature generation beyond expert-level performance. arXiv preprint arXiv:2211.12507, 2022.