PRUDEX-Compass: Towards Systematic Evaluation of Reinforcement Learning in Financial Markets
Authors: Shuo Sun
Molei Qin
Xinrun Wang
Bo An
Venue: Accepted by TMLR ()

Abstract: The financial markets, which involve more than $90 trillion market capitals, attract the attention of innumerable investors around the world. Recently, reinforcement learning in financial markets (FinRL) has emerged as a promising direction to train agents for making profitable investment decisions. However, the evaluation of most FinRL methods only focuses on profit-related measures and ignores many critical axes, which are far from satisfactory for financial practitioners to deploy these methods into real-world financial markets. Therefore, we introduce PRUDEX-Compass, which has 6 axes, i.e., Profitability, Risk-control, Universality, Diversity, rEliability, and eXplainability, with a total of 17 measures for a systematic evaluation. Specifically, i) we propose AlphaMix+ as a strong FinRL baseline, which leverages mixture-of-experts (MoE) and risk-sensitive approaches to make diversified risk-aware investment decisions, ii) we evaluate 8 FinRL methods in 4 long-term real-world datasets of influential financial markets to demonstrate the usage of our PRUDEX-Compass, iii) PRUDEX-Compass together with 4 real-world datasets, standard implementation of 8 FinRL methods and a portfolio management environment is released as public resources to facilitate the design and comparison of new FinRL methods. We hope that PRUDEX-Compass can not only shed light on future FinRL research to prevent untrustworthy results from stagnating FinRL into successful industry deployment but also provide a new challenging algorithm evaluation scenario for the reinforcement learning (RL) community.

Published in Transactions on Machine Learning Research (03/2023)
PRUDEX-Compas: Towards Systematic Evaluation of Reinforcement Learning in Financial Markets
Shuo Sun
shuo003 Ge. ntt.tnu..u....
School of Computer Science and Engineering Nanyang Technological University
Molei Qin
molci. qin wwt.ee..eeddu.....mmm.mm
School Computer Science and Engineering
Nanyang Technological University
Xinrun Wang"
zinrun, wowggn.wnn@@nuuu.c edu.sg
School of Computer Science and Engineering Nanyang Technological University
Bo An"
boan Ontu. edu.s9
School of Computer Science and Engineering Nangang Technological University
Reviewed on OpenReview: https: // openreview. forum? id= Jjba
Abstract
The financial markets, which involve more than $90 trillion market capitals, attract the attention of innumerable investors around the world. Recently, reinforcement learning in financial markets (FinRL) has emerged as promising direction to train agents for making profitable investment decisions. However, the evaluation of most FinRL methods only focuses on profit-related measures and ignores many critical axes, which are far from satisfactory for financial practitioners to deploy these methods into real-world financial markots. Therefore, we introduce PRUDEX-Compass which has 6 axes, i.c., Profitability, risk-control, Universality, Diversity, rEliability, and eX eXplainability, with a total of 17 measures for systematic evaluation. Specifically, i) since most existing FinRL algorithms designed to maximize profit with poor performance under systematic evaluation, we introduce AlphaMix+, which leverages mixture-of-experts and risk-sensitive approaches, to serve as one strong FinRL baseline that outperforms market average on all axes PRUDEX-Compas. ii) we evaluate AlphaMix+ and 7 other FinRL methods in 4 long-term real-world datasets of influential financial markets to demonstrate the usage of our PRUDEX- Comppsss and the superiority of AlphaMix++, iii) PRUDEX-Compase together with 4 real-world datasets, standard implementation of 8 FinRL methods, portfolio management environment and related visualization toolkits is released as public resources to facilitate the design and comparison of new FinRL methods. We hope that PRUDEX-Compass can not only shed light on future FinRL research to prevent untrustworthy results from stagnating FinRI into successful industry deployment but also provide a new challenging algorithm evaluation scenario for the reinforcement learning (RL) community.
only
"Corresponding Authors
Inttps://github. http://ihhhub.ccoaaaaaeeaa com/Traadeaatter Compass

---

Published in Transactions on Machine Learning Research (03/2023)
1 Introduction
Quantitative trading (QT) refers to trading strategics, which applics mathematical models to automatically identify profitable trading opportunities (Chan, 2021). With the development of the artificial intelligence, the trading volume of QT continuousyy increases and accounts for more than 70% and 40% trading volumes, in developed markets (e.g., US) and developing markets (e.g,, China), respectively (Karpoff, 1987). How to make profitable investment decisions against the various uncertainties in quantitative trading becomes one of the main challenges for financial practitioners (An et al., 2022). Among the various machine learning methods, such as deep learning (Xu & Cohen, 2018; Sawhney et al., 2021) and boosting decision trees (Ke et al., 2017), deep reinforcement learning (DRL) is attracting increasing attention from both academia and financial industries (Sun et al., 2023) due to its stellar performance on solving complex sequential problems such Go (Silver et al., 2017), StarCraftIII (Vinyals et al., 2019), muclcar fusion (Degrave et al., 2022) and malrix mulplication (Fawzi et al., 2022).
Deep RL has achieved significant success in various quantitative trading tasks. Specifically, FDDR (Deng et al., 2016) and iRDPG (Liu et al., 2020b) are designed to learn financial trading signals and micmic behaviors of professional traders for algorithmic trading, respectively. For portfolio management, deep RL methods are proposed to account for the impact of market risk (Wang et al., 2021b) and the commission fee (Wang et al., 2021a). A PPO-based framework (Lin & Beling, 2020) is proposed for order execution and policy distillation mechanism is added to bridge the gap between imperfect market states and optimal execution actions (Fang et al., 2021). For market making, deep RI. methods are introduced from both game-theoretic (Spooner et al., 2018) and adversarial learning (Spooner & Savani, 2020) perspectives as an adaptation of traditional mathematical models.
However, the evaluation of existing FinRI, methods (Sun et al., 2022; Wang et al., 2021a; Fang et al., 2021; Liu et al., 2020a) only focuses on profit-rclated measurcs, which ignores several critical axes, such as risk-control and reliability. In addition to profitability, financial practitioners care about many other aspects of FinRL methods, ie., how much risk need to take for per unit of profit; how FinRL algorithms behave when the market status changes. In preliminary experiments, we find many examples that indicate the weakness of existing profit-seeking FinRL algorithms. For instance, IMIT (Ding et al., 2018) may lead to catastrophic capital loss when black swan events happen (Section 6.7) ) and SAC (Haarnoja et al., 2018) shows poor risk-control ability, which is not an ideal option for conservative traders (Section 6.3). In practice, with the existence of low signal-to-noise ratio and distribution shift in financial markets (Malkiel, 2003), FinRL methods with only great profitability performance and become risky to be deployed in real-world scenarios (De Prado, 2018). As Robert Pardo (CEO of hedge fund) said, he will never trade with a method that does not prove itself through a systematic evaluation (Pardo, 2011). Therefore, a benchmark for the systematic evaluation of FinRL methods is urgently needed.
backtesting are very likely to overfit on historical data
In this paper, we first introduce PRUDEX-Compase which has 6 axes with a total of 17 measures for systematic evaluation of FinRL methods. we then propose AlphaMix+, a deep RL method composed of diversified exxperss-ofeeppets and risk-aware Bellman backup, as a strong FinRL baseline that significantly outperforms existing FinRL methods under systematic evaluation by mimicking the bottom-up hierarchical trading strategy design workflow in real-world companies (Khorana et al., 2007). In addition, 7 widely used FinRL methods ter with AlphaMix+ on 4 long-term real-world datasets spanning over 15 years on popular trading tasks to demonstrate the usage of -LCDEX-CCopass and the superiority of M. Accompaniod with an open-source library' of datasets, bascline implementation, RL environment and evaluation toolkits, we call for a change in how we evaluate FinRL methods to facilitate the industry deployment of FIn methods. Moreover, PRUDEX-Compant also provides the RL community new algorithm evaluation scenarios to test the effectiveness of novel RL algorithms in the ever-changing financial markets.
evaluate
2 PRUDEX-Compase. Systematic Evaluation of FinRL
To provide a clear exposition of FinRL evaluation, we introduce SOUDEX-Compass to provide an intuitive visual means to give financial practitioners sense of comparability and positioning of FinRL methods. PRUDEX-Compass is composed of two central elements: i) the axis-level (inner), which specifies the different.
2

---

Published in Transactions on Machine Learning Research (03/2023)
Figure 1: Illustration of our FinRL methods evaluation benchmark PRUDEX-Compass. The inner star plot provides visual indication of the relative strength of different FinRL methods in terms of six axes, A mark on the star plot's inner circle suggests the market average*. The outer level of the PRUDEX PRUDEX-Compass specifies which particular measures have been evaluated in practice to show the status of evaluation. We fill the compass with AlphaMix+ and 7 widely-used FinRL methods to provide an intuitive example. In general, AlphaMix gets the best performance (largest score out of axes) with a comprehensive evaluation (much more markers in the outer level).
Peoft
Control
Photiaalllyy
Diseuity
ralling
Reliablltty
sutrlation
LSNE
AZC
PRO
SAC
SARL
DeadTraderr
EIE
IMIT
AlghaMixa
axes considered for FinRL evaluation and ii) measure-level (outer), which specifies the measures used for benchmarking FinRL methods. Figuratively speaking, the axis-level maps out the relative strength of FinRL methods in terms of each axis, whereas the measure-level provides a compact way to visually assess which setup and evaluation measures are practically reported to point out how comprehensive the evaluation for FinRL algorithms. To provide practical basis, we have directly filled the exemplary compass visualization in Figure 1. The contributions of PURDEX-Compass are three-fold: i) carefully collecting 17 measures from the literature of multiple disciplines (e.g., finance, artificial intelligence, statistics and engineering) and properly categorizing them into 6 axes; ii) proposing customized version of the mcasures suitable for FinRL together with a set of easy-to-use visualization tools (Section 6.2 6.7); iii) introducing PRUDEX-Compase unified visual interface composed of two core elements to indicate the relative performance strength of FinRL methods (inner part) and their evaluation completeness (outer part). We remark that the visualization design of PRUDEX- E-DDX-s is inspired and built top of the template code provided in CLEVA-Compass (Mundt et al., 2021).
2.1 Axes of PRUDEX-Compant
We choose the design of the axis-level compass as a star diagram following the idea of (Wang et al., 2022; Mundt et al., 2021). Specifically, the axis-level element contains two hexagons, marks on all vertices of the two hexagons", and lines connecting the central point and vertices of hexagons. We add marks in the middle of central point and outer circle of the axis-level to indicate the performance of market average for convenient check. To indicate the relative strength of FinRL methods in terms of 6 critical evaluation perspectives, we first calculate the normalized score of each FinRL algorithm in terms of each axis by normalizing the mumeric values of original experimental results into an integer score t E [0, 100]. Inspired by the widely used performance ratio in Arcade Learning Environment (Mnih al., 2015; Machado et al., 2018), we choose normalize FinRL methods performance based on their relative performance to market average strategy, which is considered the golden standard for many financial practitioners. Specifically, we assign score t= 50 to market average and introduce a parameter 6%, where k% better worse relative to the market average is assigned 100 and 0, respectively. We further clip score values, which are larger than 100 to 100 and lower than 0 to 0, to alleviate the impact of ext values (e.g., extremely high return rate under markets with sudden
Phtups:/ http:/gttthhhbtmmom.g/ hoom/mmlreeeeee /ww/://lllrcc//ee////// /C///CLEVC-CCCi Market average indicatos the trading strategy, which invests equal amount of money into all financial assets in the pool, to reflect the average market conditions.
4The edges and vertices of the inner hexagon become slightly blurry but still distinguishable after filling in the results of FInRL algorithms.
3

---

Published in Transactions on Machine Learning Research (03/2023)
increase of some Crypto assets). For the value choice of k we find it is an empirically robust option to set k = 20 for our experiments on 8 FinRT, methods across 4 financial markets. Moreover, we consult with our industry collaborators and they agree that it is reasonable to consider 20% better than the market average as a very succossful trading strategy (c.g.. t= 100). Users can directly follow our scttings and compare the score (larger is better). Detailed descriptions of normalization equations are available in Appendix A.3. We then decide the position of vertices based on the score and connect them to provide a visual impression on the performance of each FinRL method. The advantages of this design are three-fold: i) it provides an ideal visual representation when comparing plot elements (Fuchs et al. 2014), fi) it allows human perceivers to quickly learn more facts by fast visual search (Elder & Zucker, 1993), and îïi) the geometric region boundaries in the star plot have high priority in perception (Elder & Zucker, 1998). We introduce the 6 axis-level elements of SOUDEX-Compase as follows:
Profitability, Aligned with the key objective of QT, profitability focuses on the evaluation of FinRL methods' ability to gain market capital. Besides pure return, it also measures how stable (Franco & Leah, 1997) and persistent (Gârleanu & Pedersen, 2013) FinRL methods are to achieve high profit.
Risk- Control Due to the well-known tradeoff between profit and risk in finance (Brink & McCarl, 1978), financial practitioners take great efforts on the assessment and control of both systematic risk and idiosyneratic risk (Goyal & Santa, 2003), which is also of vital importance in FinRI, evaluation.
Universality. The financial market is a complex ecosystem that involves innumerable assets, countries, time-scale and trading styles. Universality tries to evaluate FinRL's ability to achieve satisfied performance (e-g., better than market average) in various quantitative trading scenarios. Designing FinRI. methods with better universality (Fang et al., 2021) is in line with popular machine learning topics such transfer learning (Pan & Yang, 2009) and meta learning (Hospedales et al., 2021).
Diversity. In finance, diversification refers to the process of allocating capitals in a way that reduces the exposure to any one particular asset or risk. As Markowitz (Nobel Laurcate in Economics) said (Tu & Zhou, 2011), diversity is the only free lunch in investing that plays an indispensable role on enhancing profitability and risk-control. In RL community, diversity is widely used to encourage exploration (Parker et al., 2020). This axis of PRUDEX-Compann tends to address the lack of diversity evaluation of FinRL methods.
Reliability. RL methods have the lendency to be highly variable in performance and sensitive to many factors such as random seeds (Henderson et al., 2018) and market stationarity shift across time (Lee et al., 2010). This variability issue hinders a reliable method and can be costly or even dangerous for high-stake applications such as quantitative trading. This axis introduces techniques on RL reliability evaluation (Chan et al., 2019; Agarwal et al., 2021) with a focus on quantitative trading.
Explainability. Psychologically speaking, if the users do not trust model, they will not use it (Ribeiro et al., 2016). Explainability generally refers to any technique that helps users developers of models understand why models behave the way they do. In FinRL, it can come in the form that tells traders which model is effective under what market conditions or why one trading action is mistaken and how to fix it. Rigorous regulatory requirements in financial markets further enhance its imporlance for model debugging (Bhatt et al., 2020), monitoring (Pinto et al., 2019) and audit (Bhatt et al., 2020).
2.2 Measures of PRUDEX-Compan
As the inner star plot contextualizes macroscopic axes of FinRL evaluation, the outer measure-level places emphasis on detailed evaluation setup and metrics. In essence, a mark on the measure-level indicates that a method practically reports corresponding measures in its empirical investigation, where more marks indicate more comprehensive evaluation. We list the 17 measures on the outer level of the PRUDEX-Compp Compass in Table with brief descriptions. In addition, we Ieave measures of FinRL explainability as future work due to the lack of FinRL algorithms with solid design of explainability. We conduct literature review on RL explainability and point their potential application in FinRL as follows. DSP (Landajuela et al., 2021) is proposed to discover symbolic policy with expert knowledge. Differentiable decision trees are incorporated into RL for better explainability (Silva et al., 2020). Another line of works tries to discover interpretable features with techniques such as self-supervised learning (Shi et al., 2020) and adversarial learning (Gupta
4

---

Published in Transactions on Machine Learning Research (03/2023)
et al., 2020). Open-XAI (Agarwal et al., 2022) offers comprehensive open-source framework for evaluating and benchmarking post hoc explanation methods. We plan to incorporate suitable evaluation methods, i.e., LIME (Ribeiro et al., 2016) and SHAP (Lundberg & Lee, 2017), from Open-XAI into TRUDEX-Compase with customized adoption on decision-basse FinRL methods.
Table I: Brief summary of evaluation measures in outer level of PRUDEX-Compase Profitability, Risk-control, Universality, Diversity, rEliability, and eXplainability.
Axes
Measures
Descriptions
Profit
A class of metrics to assess FinRL's ability to gain market capital.
P
Loss in the investment decision making ability of FinRL methods over time due to distribution shift in financial markets (Pénasse, 2022).
Alpha Decay
Equity Curve
A graphical representation of the value changes
time.
Risk
A class of metrics to assess the risk level of FinRL methods (Shiller, 1992).
Risk-adjusted
A class of metrics that calculate the normalized profit with regards to different kinds of risks, i... volatility and downside risk (Magdon & Atiya, 2004).
R
Profit
Extreme Market
The relative performance of FinRL methods on extreme market condition during black swan events (Aven, 2013) such
and covid-19.
Financial market across both developed countrics (e.g., US and Europe) and developing countries (e.g., China and India).
Country
Asset Type
Various financial asset types, i..,, stock, future, FX and Crypto
U
Time-Scale
Both coarse-grained (e.g., day level) and fine-grained (e.g., second level) financial data match different trading styles.
A statistical visualization tool to map high-dimensional time-series data points into 2-D dimension (Vander & Hinton, 2008) to assess the data-level diversity.
t-SNE
Entropy-based metrics from information theory (Reza, 1994) to show the diver- sity of FinRL methods' trading behaviors.
D
Entropy
Metrics that account the correlation (Kirchner & Zunckel, 2011) between finan- cial assets to assess the diversity of FinRL methods.
Correlation
Diversity
A visualization tool to demonstrate the diversity of investment decisions among different financial assets with heatmap (Harris et al., 2020)
Heatmap
Performance Profile
A visualization of FinRL methods' empirical score distribution (Dolan & Moré, 2002), which casy to read with qualitative comparisons.
E
The performance standard deviation across different random seeds and hyper- parameters (Henderson et al., 2018).
Variability
Rolling
Using rolling time window to retrain or fine-tune FinRL methods and evaluate the performance on multiple test periods (De Prado, 2018).
Window
Rank
A visualization toolkit to show the rank of FinRL, methods across different metrics, which will not be dominated by extreme values (Agarwal et al., 2021).
Comparison
X
We discuss current status and highlight promising further directions.
3
FinRL Preliminaries and Problem Formulation
3.1
Portfolio Management
Portfolio management is a fundamental quantitative trading task (Sun et al., 2023), where investors hold a pool of different financial assets, i.c., stocks, bonds, as well as cash, and reallocate the proportion of capitals invested in each asset. periodically to maximize future profit.
5

---

Published in Transactions on Machine Learning Research (03/2023)
OHLCV refers to the raw information of bar chart acquired from the financial markets. OHLCV vector at time &. is denoted price, pf is the close price and 0t is the volume.
(Pi: 1/. Pt. Pf: vt), where pi is the open price, pf is the high price, p is the low
Technical Indicator indicates high-order features derived based on a formulaic combination of the original OHLCCV with financial insights. We define the vector of technical indicator at time t: yr Us v where of = (n, 0²,, fh and o are the formula function and the parameter of technical indicator k, respectively.
Portfolio is the proportion of capitals allocated to each asset that can be represented as vector:
Wt = [ur. wy .. w E RM and Σ wy =
(1)
where M + 1 is the number of portfolio's constituents, including cash and M financial assets. wi represents the ratio of the total portfolio value invested at time t on asset i and w/ represents cash.
Asset Price refers to the vector of close price for each financial asset defined as pt. = =br -v..rr Pt................ p& where p is the close price of asset i at time t. Note that the price of cash pt is a constant.
Portfolio Value 1441 at time + defined based on the asset price change and portfolio weight as:
U++1 u. - W P
Pi
The objective of portfolio management is to maximize the final portfolio value given long-term time horizon by dynamically tuning the portfolio weight at each time step. As unified benchmark, evaluation metrics proposed in PRUDEX-Compass can be easily adopted to all quantitative trading tasks. We focus portfolio management in this work as a demonstrative example.
3.2 MDP Formulation
We consider a standard RL scenario in which an agent (investor) interacts with an environment (the financial market) in discrete time. Formally, we introduce MDP, which is defined by the tuple: MDP = (S, A, P.P. y, H). Specifically, S a finite set of states. A is a finite set of actions. P: x Axs - [] is a state transaction function, which consists of a set of conditional transition probabilities between states. R:S x immediate reward of taking an action in state. 7 E [0, 1) is the discount factor and H is a time horizon indicating the length of the trading period. A (stationary) policy ιa S x A - [0, 1], parameterized by 0, assigns each state E S a distribution over actions, where a E A has probability Taass). A Q-value function gives expected accumulated reward when executing action ae in state and following policy π in the future, which is Q(st, dz) = (....).....)) Etrnnrrss, a.). During training, one episode corresponds to adjusting the portfolio at cach time step through the whole trading periods, i.c., time scope of training set, with time horizon H. The objective of the agent is to learn an optimal policy: = argmar,, 1E, Ei= ( St = $]
R the reward function, where is a continuous set of possible rewards and R indicates the
State st E S at time involves the concatenation of technical indicator vectors of M financial assets (yt, (t) as the external markets state, where y! represent the technical indicator vector of asset at time t. In addition, a 2-dimension tuple of investors' position and remaining cash is added as internal state. Action space at time t is an M +1 dimension vector [w] w!...,, w[M] as portfolio w, to represent. the proportion of capitals invested at each asset. Reward Ft at time is the change of portfolio value: r = -1+11 Uy where positive/negativv values indicate carning/losin money, respectively.
3.3 Soft Actor-Crite (SAC) and Other Popular FinRL Methods
We introduce SAC (Haarnoja et al., 2018), which is the base model of many popular FinRL methods (Yuan et al., 2020). SAC is a popular ooolpo actor-critic RL method based on the maximum entropy RL framework (Ziebart, 2010), which maximizes a weight objective of the reward and the policy entropy, encourage robustness to noise and exploration. For parameter updating, SAC alternates between a soft policy evaluation and soft policy improvement. At the soft policy evaluation step, soft Q-function Qn(s), at);
6

---

Published in Transactions on Machine Learning Research (03/2023)
which is modeled neural network with parameters 0, is updated by minimizing the following soft Bellman residual:
Caritic (6) = BL.wDLLQ(TTT.. 0)],
(3)
(T 0) (Qo(st, (4) r1 2VV (8t-)))
(4)
where V(s)) Bapnt Q0((s, at) a log m st)].
where 7 (st; oe, Pt: St+1) represents transition, D indicates replay buffer, 0 are the delayed parameters, and a is used as temperature parameter. To conduct steps of soft policy improvement, the policy with its parameter is updated through minimizing the following objective:
actoor (0) = E2 Φ)], I (8t, Φ) = Baiws, (ar log To(a 8x) Qo(se, ax)]-
(6)
(7)
To handle continuous action spaces, the policy is modeled as a Gaussian with mean and covariance given by neural networks. In addition to SAC, A2C (Mnih et al., 2016), a popular actor-critic RL method, shows stellar performance in algorithmic trading (Zhang et al., 2020). The simple and efficient policy gradient method PPO (Schulman et al., 2017) performs well capturing trading opportunities for order execution (Lin & Beling, 2020). EIIE (Jiang al., 2017) and Investor-Imitator (LMIT) (Ding et al., 2018) are two pioneering works that apply deep RL for quantitative trading. Furthermore, SARL (Ye et al., 2020) and Deeptrader (Wang et al., 2021b) are proposed with augmented market embedding to take market risk into account for portfolio management.
4 AlphaMix+: A Strong Baseline
Before diving into systematic evaluation bench- mark PRUDEX-Compas: AlphaMix+, an FinRL algorithm based on ensemble learning, is proposed to fill the gap due to the poor performance of existing FinRL methods under systematic evaluation. Consid- cring the limitation of existing FinRL methods, the major one that investment decisions are made by single agent with high potential risk. The success of real-world trading firms relies on efficient bottom- up hicrarchical workflow with risk management as illustrated in Figure 2). First, multiple experts con- duct data analysis and build models independently based on personal trading style and risk tolerance. Later on, a senior portfolio manager summarizes their results, manage risk and makes final investment decisions.
Expert
Up Down
Portfolio Manager
Stock
Stock
Market Status
Up Down
Risk Control
Stock
Investment Strategy
Stock n
Expert
Figure Workflow of real-world trading firms.
Inspired by it, we propose AlphaMix+; universal deep RI. framework with diversified risk-aware mixture-of- experts following the idea of (Lee et al., 2021b), to mimic this efficient workflow. In principle, AlphaMix- can be combined with most modern off-policy RL algorithms for any quantitative trading task. As SAC (Haarnoja et al., 2018) is a sample-effciinncy algorithm in quantitative trading (Yuan et al., 2020), we pick it as the base model of AlphaMix here for exposition. An overview of AlphaMix+ is shown in Figure 3.
Risk-aware Bellman backup. We consider a trading firm with N trading experts, i.e,, (Qa, Ta, L where 6, and Φ; denote the parameters of the i-th experting expert's soft Q-function and policy. To apply classic Q-learning based on the Bellman backup in Eq. (3) in FinRL, one major issue is the severe negative impact of error propagation that induces the market noise to the learning trading signals (true Q-value) of
Readers whose main interest the ovaluaion benchmark may skip this Section and take AlphaMix- strong FInRI. baseline.
7

---

Published in Transactions on Machine Learning Research (03/2023)
the current Q-function (Kumar al., 2020) which can cause unstable convergence. In order to overcome this issue, for trading expert i, we apply risk-aware Bellman backup (Tee et al., 2021b) as follows:
Lwo(Te 0.) w(s:+1, (st, ae) - (s(+1)))
(8)
where 7r (Se, de, Te, 8141) transition, d++1 ~ To. (a 8t), and w(ss a) is a confidence weight based on the ensemble of target Q- Q-functions:
w(s.a) a) o(-
T) +
(9)
where Qsta(s, a) indicates the empirical standard deviation of all trading experts' target Q-functions (Qo, H and T' is temperature parameter (Hinton al., 2015) to adapt the scale of Quta(s a). is the sigmoid function and k > 0 is used to control the value range of confidence weight. The confidence weight is bounded in A k + 0.5] as (sta(s, a) is always positive number. Intuitively, the objective LwQ down weights the sample transitions with inconsistent trading suggestions from different experts (high variance across target. Q-functions), resulting in a loss function for the Q-updates with better risk management.
Diversified Experts. We encourage the diversity between trading experts with two simple yet efficient tricks based on (Lee et al., 2021b). First, initialize the model parameters of all trading ex- perts with random parameter values for inducing an initial diversity in the models following (Lakshminarayanau et al., 2017; Wenzel al., 2020). Socond, we apply dif- ferent samples to train each agent based similar idea in Ensemble (Wen et al., 2019). For each trading expert at each time step t, construct the binary masks me, by sampling from the Bernoulli distribution with parameter 33 E (0, 1]. Later on, we multiply the bootstrap mask to each objective function (e.g., (8t, (i) and mr,;Cwol (Te, 0,) in Eq-(7) and (8)) while updating parameters of trading experts. This encourages each expert to think individually with diversified strategies. We find it sufficient for AlphaMix+ to have desired diversity (experiments in Section 6.6) with the two simple tricks. Other tricks such as a KL divergence (Yu et al., 2013) loss term is not further incorporated to keep simplicity. We remark that although similar diversity encouragement techniques have been shown effective in classic RL tasks (Osband et al., 2016; Lee et al., 2021b), this work is the first to explore their potential in financial markets. We conduct ablation studies on the effectiveness of each component in AlphaMix+ and parameter analysis to probe sensitivity. We put related experimental results in Appendix C.2 and C.3, respectively.
Actor
Risk-aware Bellman backup
Bootstrap
Critic
Expert
Actor
Market History
Critic
Investment Action
Expert
Figure 3: An illustration of AlphaMix+, universal deep RL framework with diversified risk-aware mixturee-ffexperts.
5 Experimental Setup
5.1 Datasets
We collect real-world financial datasets spanning over 15 years of US stock, China stock, Cryptocurenncy (Crypto) and For- eign Exchange (FX) from Yahoo Finance and Kaggle. All raw data and related processing scripts publicly available. We summarize statistics of the 4 datasets Figure 4: Train/valid/test split procedure with rolling windows.
1 year a/e-
year *le year * 1 year >
Phase
Phase 1: Training
Valid
Test
Phase 2: Training
Valid
Test
Phase Training
Valid
Test
Start
Time
End
with further elaboration in Table 2. US Stock dataset contains 10-year historical prices of 29 influential stocks with top unit price as strong assessment of the market's 's overall health and tendencies. China Stock dataset contains 4-ycar historical prices of 47 influential stocks with top capitalization from the Shanghai exchange. Crypto? dataset contains G-year historical
Both US and China slock data is collected from
"nttpe://githan con/ conn/aabo--i finance 7 httpps
kaggle. cone.co///ae con/datasats//nteee suddal wtw.ta/ttte/eddaaaaaaaaaee/aar/e htd.aaaiwt........gggggeeoggrgga
8

---

Published in Transactions on Machine Learning Research (03/2023)
prices of 9 infiuential virtual currency with top unit price and trading volume collected. FX dataset contains 20-year historical prices of 22 most popular currency with top foreign exchange reserves for US dollars. For each dataset, we filter out financial assets with missing values. For data split, we apply the similar split procodure in (Sawhney et al., 2020) with rolling window for all four datasets. As shown in Figure 4, phase 3 uses the last. year for lest, penultimate year for validation and the remaining of the dataset for training. For phase one and two, their validation/tess sets roll back one and two years, respectively.
Table 2: Dataset statistics
Dataset
Market
Freq
Number
Days
From
To
Source
US Stock
US
1d
29
2517
12/01/03
21/12/31
Yahoo
China Stock
China
1h
47
1036
16/06/01
20/09/01
Yahoo
Crypto
1d
9
2014
16/01/01
21/07/06
Kaggle
FX
1d
22
5015
00/01/03
19/12/31
Kaggle
5.2 Features
We generate 11 temporal features as shown in Ta- ble 3 to describe the stock markets following (Feng et al., 2018; Yoo et al., 2021). Zopen, Zhigh and Soo represent the relative values the open, high, low prices relative the close price current time step, respectively. Zelose and Zadj_ close represent the rela- tive values of the closing and adjusted closing prices compared with time step & 1. 2dR represents long-term moving average of the adjusted close prices during the last time steps relative to the current close price. We apply z-score normalization on each fcature.
Table 3: Features to describe the financial markets
Features
Calculation Formula
Zopen opens/closer
Zopen, Thigh How Zelose
Zelase = closes/ ccloset-1
Zd 5. 10, 215
Ad 2i closet-i/ close;
2d 20+ 2d
25+ Zd 30
5.3 Baselines
We conduct experiments with 7 representative FinRL methods including 3 classic RL methods: A2C (Mnih et al., 2016), PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018), 4 RI-based trading methods: EIIE (Jiang et al., 2017), livestor-mmitttor (IMIT) (Ding et al., 2018), SARL (Ye et al., 2020) and DeepTrrader (DT) (Wang et al., 2021b) and our AlphaMix+. Descriptions basclines are as follows:
A2C (Mnih et al., 2016) is a classic actor-critic RL algorithms that introduce an advantage function to enhance policy gradient update by reducing variance.
PPO (Schulman et al., 2017) is a proximal policy optimization that constrain the difference between current policy and updated policy with simplified clipping term in the objective function.
SAC (Haarnoja et al., 2018) a widely-used off-policy actor-critic method based on the maximum entropy RL framework to encourage algorithm robustness.
SARL (Ye et al., 2020) proposes a slate-augmented RL framework, which leverages the price movement prediction additional states, based on deterministic policy gradient (Silver et al., 2014) methods.
DeepTrader (DT) (Wang et al., 2021b) is policy gradient based method. To tackle the risk-return balancing issue, it simultaneously uses negative maximum drawdown and price rising rate as reward functions to balance between profit and risk with an asset scoring unit.
EIIE (Jiang et al., 2017) is a deterministic policy gradient based RL framework, which contains: 1) an ensemble of identical independent evaluators topology; 2) a portfolio vector memory; 3) an online stochastic batch learning scheme.
8 htttps: //wwz/ kaggle. www.ggl...oo//aaaoo http//ttasatts/uunnoll////// hag.rrrroo..oo---aaaaaaaaarrrrrr

---

Published in Transactions on Machine Learning Research (03/2023)
Investor-Imitator (IMIT) (Ding et al,, 2018) imitates behaviors of different investors (e.g., ora- obe/collaborate/publii investor) using investor-specific reward functions with a set of logic descriptors.
Non-RL methods not included as baselines for two reasons: i) In general, RL methods outperform different types of non-RL methods (Moskowitz et al., 2012; Ke et al., 2017; Xu & Cohen, 2018) in different trading tasks. ii) PRUDEX-Compase focuses on the evaluation of RL methods in financial markets. We leave comparison with non-RL methods as future directions and discuss our plan in Section 7.
5.4 Training Setup
We perform all experiments on an RTX 3090 GPU with 5 fixed random seeds. We apply grid search for AlphaMix*+ on Crypto and FX datasets and apply the same hyperparameters on China and US stock datasets. We try scale parameter & in list [0.3, 0.5, 0.7, 0.9), binomial sample parameter 3 in list [0.3, 0.4, 0.5, 0.6, 0.7] and lemperature T in list [18, 19, 20, 21, 22). We explore batch size in list. [256, 512, 1024] and hidden size in range [64, 128], We apply learning rate 7e - for both actor and critic. Adam used as the optimizer. One full list of hyperparamcters is available in Appendix B.1. We train AlphaMix+ for 10 epochs on all datasets. It takes about 60 minutes to train and test on China stock, US stock, FX and Crypto datasets, respectively,
For other FinRL methods, there are two conditions: i) if there are authors' official or open-source FinRL library (Liu et al., 2020a) implementations we apply the same hyperparameters for a fair comparison. This condition applies for A2C, PPO, SAC, SARL and DeepTrader. ii) if there are no publicly available implementations, we reimplement the algorithms and try our best to maintain consistency based on the original papers. This applies for EIIE and IMIT.
5.5 RL Environment Implementation
In this work, we apply the popular portfolio management environment (Liu et al., 2020a) implemented basod on OpenAI Gym (Brockman et al., 2016), which simulates live financial markets with realistic historical market data, according to the principle of time-driven simulations. During training, of technical indicators input of RL agents. RL agents generate portfolio (action) and the environment returns the net value change at each time step as reward. By interacting with the environment, the trading agents will try to derive a trading strategy with high profits. In addition, the environment assumes the trading volume of agents is not very large and has little impact on the market. Then, it is reasonable to use the price fluctuation of oflline historical financial data to build a model for reward calculation during online sinulation. We provide a concrete example here to further clarify how FinRL environment is built. Considering a simple trading scenario with only one stock, we obtain pf and pf+1, which denote the close price of the stock at time t and t ++ from historical data. The action at time t is to buy & shares of the stock. Then, the reward ra at time t. is the account profit. defined as k* (p841 rf). For state, we use historical market data to calculate technical indicators in Table 3 as external state and investors' private information such as remaining cash and current position is applied as internal state. Similar procedures to build RL environment with historical market data have been applied many FinRL work (Liu et al., 2020a; Wang et al., 2021b; Ye el al., 2020). Readers may check our open source code- for more details of the RL environment.
feed observations
6 Demonstrative Usages of PRUDEX-Comper and Related Evaluation Toolkits
In this scction, we conduct experiments on portfolio management with real-world datasets of 4 influential finnniil markets to demonstrate the usage of BRUDEX-Compann and related evaluation toolkits In Section 6.1, we show how different investors can get general impression FinRL algorithms' performance with SOUDEX-Compase Moreover, provide example usage of other evaluation toolkits with focus one particular perspective including: (1) t-SNE plot to show data-level diversity, (2) PRIDE-Star to report the performance of 8 point-wise financial metrics for evaluating profitability, risk-contrro and diversity, (3) performance profile and rank distribution plot as unbiased and robust measures towards reliable FinRL
methods, (4) portfolio diversity heatmap to evaluate the decision-level diversity, (5) extreme market sccnarios
Both authors' official or open-source FInRI. library (Liu et al., 2020a) implementations are taned FInRI domains.
10

---

Published in Transactions on Machine Learning Research (03/2023)
with black swan events to evaluate the risk-control and generalization ability of FinRL algorithms. In particular, investors can either use these evaluation toolkits together with PRUDEX-Compass to pursue a systematic evaluation or as an independent measure with a focus on the perspective they care about.
6.1 A General Impression with PRUDEX-Compass
As shown in Figure 1, we fill the PRUDEX- PRUDEX-Compass based on the experimental results of the 8 FinRL methods. For axis-level, it directly illustrates the relative performance of each method in terms of 6 axes to provide a general impression. We normalized the score into 100 with 50 as the market average (dctails in Appendix A.2). For explainability, all methods are scored 50 as we leave it as future direction. A performs best in all 5 remaining axes. Specifically, it outperforms other FinRL methods 53% and 43% in universality and diversity, respectively, which demonstrates the effectiveness of the weighted Bellman backup and diversified bootstrap initialization.
For measure-level, we give a mark if one measure is used in the evaluation of the FinRL methods, the goal here is to show how comprehensive the methods are evaluated. Together with all measures we proposed, AlphaMix+ clearly has a more rigorous evaluation, which makes the results more trustworthy. Arguably, PRUDEX-Compass provides compact visualization to evaluate FinRL methods that is much better than only looking at a result table of different metrics especially when lots of FinRL methods are involved. In other words, the compass highlights the required subtleties, that may otherwise be challenging to extract from text descriptions, potentially be under-specified and prevent readers to misinterpret results based on result table display. With PRUDEX-Compant interests. Conservative traders may prefer methods with a relative stable profit rate and low risk. Aggressive traders may pay more attention on profitability, as they are willing to take high risk to pursue extremely high profit. For international trading firms, they may have high expectation on universality and diversity.
users can flexibly pick suitable methods with regards to their personal
6.2 Visualizing Financial Markets with t-SNE
Even though it is wide consensus that different financial markets share different trading patterns (Campbell et al., 1998), there is lack of visualization tool to demonstrate how different are these markets. To show
data-level diversity of evaluation, we use t-SNE (Vander & Hinton, 2008) here to map all 4 datasets into a
2-D dimension plot with the 11 features described in Table 3 as the imput. To avoid overlapping in financial data, we pick a data point every 30 time step for each asset across 4 financial markets. Each data point corresponds each time step, where necessary temporal information is maintained In Figure 5, the US stock and FX datasets lie in the lower left and upper right corner, respectively, as a whole cluster, which is consistent with their status as relative mature and stable markets (Emenyonu & Gray, 1996). For China stock and the Crypto, data points are scattered with more outliers that demonstrate their essence as emerging and violate markets (De Santis et al., 1997). The t-SNE plot is useful to provide intuitive expression on the data-level diversity of different markets while evaluating FinRL methods.
China
US
FX
Crypto
the value vector of 11 features
Figure t-SNE market visualization.
6.3 Star for Evaluating Profitability, Risk-Control and Diversity
As the evaluation measures for Profitability, Risk and DivErsity (PRIDE) are point-wise metrics with real number values, we use the PRIDE-star, which is a star plot to show the relative strength of 8 metrics including profit motrics: total return (TR), risk metrics: volatility (Vol) (Shiller, 1992) and maximum drawdown (MDD) (Magdon & Aliya, 2004), risk-adjusted profit metries: Sharpe ratio (SR) (Sharpe, 1998), Calmar ratio (CR) and Sortino ratio (SoR), and 2 diversity metrics: entropy (ENT) (Jost, 2006) and effect number of
10 common incorporate temporal information into features in Fintech. For instance, zd_s s uses the close price time
step
11

---

Published in Transactions on Machine Learning Research (03/2023)
bets (ENB) (Kirchner & Zunckel, 2011). The mathematical definitions of these metrics in Appendix A.1. We report the overall performance across the 4 financial markets of the 6 FinRL methods in Figure 6, where the inner circle represents market average. In general, AlphaMix+ performs best in the PRIDE-Star plot. In addition, A outperforms the second best by 25% in terms of ENT that shows the effectiveness of the boostrap with random initialization component AlphaMix+.
CR
SR
CR
SR
CR
SR
CR
SR
SoR
TR
Sei
IR
Sukk
TR.
12
BN3
END
Vi
adp
ENTT
CCD
ENT
MDD
.NV
A2C
PPO
SAC
SARL
55
Solt
TR.
SON
TR
AL.
74
AM
PM
P9
SNN
Mdp
INE
MCD
UND
Mdk
ENTT
Mdp
AN3
DeepTrader
EIIE
IMIT
AlphaMix-
Figure 6: Overall performance across 4 financial markets on PRIDE-Star to evaluate profitability, risk-control and diversity, where AlphaMix+ achieves the best performance in out metrics.
DeepTrader
AlphaMix+
Deaperader
EIIE
IMIT
AlphaMix-
1.00
100
TR
100
100
VOL
100
Entropy
Fracton of runs with score
0.75
Fraction (In %)
0.50
60
40
40
40
40
0.00
20-
20-
20-
20
20
40
60
80
100
total
12345670
12345678
12345678
(b)
Figure 7: (a) Performance profile of total return score distributions across 4 financial markets. Shaded regions show pointwise 95% confidence bands based on percentile bootstrap with stratified sampling. (b) Rank distribution lerms TR, SR, Vol and ENT across financial markets.
6.4 Performance Profile: An Unbiased Approach to Report Performance
The performance profile (Agarwal et al., 2021) reports the score distribution of all runs across the financial markcts that are statistically unbiased and more robust to outliers compared to the widely-used mean performance. Performance profiles proposed herein visualize the empirical tail distribution function of a random score (higher curve is better), with point-wise confidence bands based stratified bootstrap (Efron,
12

---

Published in Transactions on Machine Learning Research (03/2023)
1979). A score distribution shows the fraction of runs above certain normalized score that is an unbiased estimator of the underlying performance distribution. As shown in Figure 7a, AlphaMix+ is generally a robust but conservative FinRL methods that shows the least bad runs, which makes it an attractive option for conservative investors that care more about risk. However, radical investors may pick SAC as it has the largest probability of achieving score 100, which indicates a return rate higher than twice the market average.
6.5 Rank Distribution to Demonstrate the Rank of FinRL methods
In Figure 7b, we plot the rank distribution (Agarwal et al., 2021) of 8 FinRL methods in terms of TR, SR, VOL and Entropy across 4 financial markets with results of 5 random seeds in each market. The i-th column in the rank distribution plot shows the probability that a given method is assigned rank in the corresponding metries. For x-axis, rank and 8 indicate the best and worst performance." 11 For y-axis, the bar length of a given method on a given metric with rank i corresponds to the % fraction of rank i it achieves across the 4 financial markets, 3 lest periods and 5 random seeds (4 x 3 of TR is purely red, it indicates A achieves the highest TR in all random seeds across 4 financial markets.
60 in total), L..,, if the rank column
For TR and SR, AlphaMix+ slightly outperforms other methods with 27% and 35% probability to achieve top performance. For Vol, SAC gets the overall best performance while AlphaMix+ goes through higher volatility. For ENT, AlphaMix+ significantly outperforms other FinRL methods with over 56% probability for rank 1, which demonstrates its ability to train mixture of diversified trading experts.
6.6 Visualizing Strategy Diversity with Heatmap
To demonstrate the overall investment diversity of FinRL, methods, we show the average portfolio across the test period as a heatmap in Figure 8, Formally, we define the average portfolio as w:
Ej Lj=t +h mi
1ith Ej=t W
w = h h+1
(10)
h+1
h+1
where M + is the number of portfolio's 's constituents, including cash and M financial assets. wo represents the ratio of the total portfolio value invested at time on asset i, represents the length of the evaluation period and wf represents cash.
For IMIT, it puts all capital in
Relative weight of different stocks in a portfolio
one or two assets. The portfo-
A2C
0.2
lio of SARL and EIE is not
PPO
that diversified with near 0 weight on many assets more (red). For A2C, DT, PPO and SAC, the portfolio is closed to uniform, which is not desirable due to poor profitability. Our + achieves an ideal investment portfolio, which is generally diversified and allocate more weights on a few bullish stocks.
SAC
SARL
0.1
DT
EllE
IMIT
Ours
0.0
Figure 8: Heatmap of average portfolio on China stock market.
6.7 The Impact of Extreme Market Conditions
To further evaluate the risk-control and reliability, we pick three extreme market periods with black swan events. For China stock market, the period is from February 1st to March 31st 2021 when strict segregation policy is implemented in China to fight against the COVID-19 pandemic (Lee ct al., 2021a). For US stock market, the period is from March 1st to April 30th 2020, when the global financial markets are violate due the global pandemic of COVID-19 (Mazur et al., 2021). For Crypto, the period is from April 1st to May 31st 2021 when many countries posed regulation against Crypto oligarch. To report the results of different
For TR, SR and entropy, higher values indicate hetter performance. For VOL. lower values indicate better performance
13

---

Published in Transactions on Machine Learning Research (03/2023)
metrics with different numerical value scale, normalize them into score Macore follows:
mecoee (Mave/ Mave (aaalaaaeemmam/aammmmmmm (m,x/mav - k +
(11)
We define the metrics value for FinRL methods and market average as my! and Mave; respectively, & is a scale parameter.
In Figure 9, we plot the bar chart of TR and SR during the period of extreme market conditions. As a conservative method, AlphaMix+'s performance is unsatisfactory in extreme market conditions, which proves the general consensus that radical methods such as DeepTrader (DT) and SARL are more suitable for extreme markets (Marimoutou et al., 2009). Analyzing the performance on extreme market conditions can shed light on the design of FinRL methods, which is in line with economists" efforts on underslanding the financial markets. For instance, incorporating volatility-aware auxiliary task (Sun et al.; 2022) and multi-objective RL (Hayes al., 2022) + may further make be aware of extreme market conditions in advance and behave as profit-seeking agents to achieve better performance during extreme market conditions.
average
average
average
TR
SR
Score
Score
DT
ENIE
(a) US
(b)
China
(c) Crypto
Figure 9: Performance of FinRL methods during extreme market conditions.
7 Discussion
Complementary Related Efforts. Apart from the above aspects described in PRUDEX-Compass, there also exist several orthogonal perspectives for FinRL evaluation, which encompass a check-list (Pineau et al., 2021) for quantitative experiments, the construction of elaborate dataset sheets (Gebru et al., 2021), and the creation of model cards (Mitchell et al., 2019). We stress that these perspectives remain indispensable, regularly suggested in FinRL and the PRUDEX- PRUDEX-Compass does not
novel datasets and their variants disclose intended use with respect to hunan-centered application domains, ethical consideration or their caveats. We believe that it is best to report both the prior works and PRUDEX- PRUDEX-Compass Compass together to further improve the evaluation quality.
Potential Impact. For FinRL community, PRUDEX- -CEBX-ss can serve as i) a systematic evaluation toolkit; ii) a benchmark for comprehensive comparison of FinRL methods and iii) a standard code base for future design and development of novel FinRL algorithms. We hope that PRUDEX-Compass could encourage both researchers and financial practitioners to avoid fooling themselves by evaluating FinRL methods in systematic way and facilitate the design of stronger FinRI. medhods. While accounting for all elements in TRUDEX-Compar is not a panacea, we believe it is a good start with more trustworthy results for the community and further increase the confidence for real-world industry deployment. For RL community, PRUDEX -omppsss introduces a new challenging scenario with different evaluation axes for the test. of novel RL algorithms in financial markct. For FenTechh community, this work enables industry practitioners with limited RL background to easily explore the potential of RL in different FinTeec scenarios. In addition, the usage of PRUDEX- Compass is not limited in RL settings, most elements of PRUDEX- Compass can be easily PRUDEX-Compass
generalized to supervised learning settings with broader impact.
Future Plans. We plan to improve PRUDEX- CODDp-Ssopass from the following perspectives: i) For axis-level, we plan to explore the evaluation FinRL explainability with measures and plots; ii) For measure- we plan to include metrics to evaluate alpha decay and more metries, c.g., optimality gaps, for profits and risks; iii) To further improve the accuracy and reliability of TRUDEX-Compase we plan customize existing scientific and non-exploitative RL evaluation procedures (Jordan et al., 2020) into FinRL domains
14

---

Published in Transactions on Machine Learning Research (03/2023)
for the normalization and summarization of experimental results. iv) For comprehensive evaluation under different market stationarity assumptions, we plan to add one toolkit to automatically categorize market. into different styles (e.g,, bull/bear) and evaluate the performance of FinRL methods under different styles. Furthermore, there can be a data-driven simulator to generate unseen stylized data for further evaluation; v) For visualization, we plan further develop a GUT software version accompanied with a website to further lower the barrier for dissemination and use; vi) As most axes and measures in PRUDEX- PRUDEX-Compast are also key points for non-RL trading scenarios, we plan to bring PRUDEX-Compen learning settings with implementations and results of non- RL methods for broader impact.
into more general machine
Auxiliary Experiments. Due to space limitations, we have included some auxiliary yet important experiments in Appendix C. Specifically, the result tables with mean and standard deviation of the 8 metrics in PRIDE-Ster Star are reported in Appendix C.1. We plot the PRIDE-Stax performance profile and rank comparison of each financial market individually in Appendix C.4, C.5, C.6, respectively respoctivll.Fuurthermorr, Furthermore, we include ablation studies on the effoctiveness of cach component in A in Appendix C.2 and hyperparameter sensitivity experiments in Appendix C.3.
Hosting, Maintenance, Licensing. The PRUDEX-Compese datasets are hosted Google Drive. The The authors
source code are publicly available at https:/ http://gttttb..c. h TradeMaster-NTU/PRRRII /PRPDEER-RRoppa will provide important bug fixes to the community as commits to the GitHub repository. There will be summary of changes to the code and the datasets in the README web page of the GitHub repository. the unlikely case that the Google Drive link stops operating, we will migrate the dataset to another hosting and announce the new links in the GitHub repository. The provided source code and dataset are copyrighted by us and under the MIT 1icense Users have the permission to reuse the codes for any purpose.
8 Acknowledgments
This project supported by the National Research Foundation, Singapore under its Industry Alignment Fund Prre-posttioiin (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not, reflect the views of National Research Foundation, Singapore.
15

---

Published in Transactions on Machine Learning Research (03/2023)
References
Chirag Agarwal, Eshika Saxena, Satyapriya Krishna, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxal: Towards transparent evaluation of model explanations. arXiv preprinl arXiv:2206.110 11104, 2022.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, pp. 29304 29320, 2021.
Bo An, Shno Sun, and Rundong Wang. Deep reinforcement learning for quantitative trading: Challenges and opportunitics. IEEE Intelligent Systems, 37(2):23 26, 2022.
Terje Aven. On the meaning black swan in a risk context. Safety Science 57, 2013.
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. Explainable machine learning in deployment. In ACM Conference on Fairness, Accountability, and Transparency, pp. 648-657, 2020.
Lars Brink and Bruce McCarl. The tradeoff between expected return and risk among cornbelt farmers. American Journal of Agricultural Economics, 60(2), 1978.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. Xrviv preprint erXiu:16066 540, 2016.
John Y Campbell, Andrew W Lo, Craig MacKinlay, and Robert Whitelaw. The econometrics of financial markets. Macrocconomic Dynamics, 2(4), 1998.
Ernest P Chan. Quantitative trading: how to build your own algorithmic trading business. Wiley, 2021.
Stephanie CY Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama. Measuring the reliability of reinforcement learning algorithms. In International Conference Learning Representations, 2019.
Marcos Lopez De Prado. Advances in financial machine learning. John Wiley & Sons, 2018.
Giorgio De Santis et al. Stock returns and volatility in emerging financial markets. Journal of International Money and Finance, 16(4), 1997.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement. learning. Nature, 602(7897):414-444 414--199, 2022.
Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai. Deep direct reinforcement learning for financial signal representation and trading. IEFE Trunsactions Neural Nelworks and Learning Systems, 28(3), 2016.
Yi Ding, Weiqing Liu, Jiang Bian, Daoqiang Zhang, and Tie-Yan Liu. Investor-imitator: A framework for trading knowlodge extraction. In ACM SIGKDD International Conference Knowledge Discovery & Dala Mining, 2018
Elizabeth D Dolan and Jorge Moré. Benchmarking optimization software with performance profiles. Mathemalical Programminng, (1(2):201 213, 2002.
B Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1979.
James Elder and Steven Zucker. The effect of contour closure on the rapid discrimination of two-dimensional shapes. Vision Research, 33(7):981-99)) 1993.
James H Elder and Steven W Zucker. Evidence for boundary-speeific grouping. Vision Research, 38(1): 452, 1998.
16

---

Published in Transactions on Machine Learning Research (03/2023)
Emmanuel N Emenyonu and Sidney J Gray. International accounting harmonization and the major developed stock market countries: an empirical study. The Internalional Journal of Accounting, 31), 1996.
Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tic- Yan Liu. Universal trading for order execution with oracle policy distillation. In AAAT Conference Arlificial Intelligence, pp. 107-115, 2021.
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredess Mohammadamin Barckatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszez, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-63, 2022.
Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, and Tat-Seng Chua. Enhancing stock movement prediction with adversarial training. arXiv preprint orXiv: 00o:1810.09930 2018.
Franco and Leah. Risk-adjusted performance. Journal of Portfolio Management, 23(2):45 54, 1997.
Johannes Fuchs, Petra Isenberg, Anastasia Bezerianos, Fabian Fischer, and Enrico Bertini. The influence of contour similarity perception of star glyphs. IEEE Transactions Visualization and Computer Gruphics, 20(12):2251-22 22260, 2014.
Nicolae Gârleanu and Lasse Heje Pedersen. Dynamic trading with predictable returns and transaction costs. The Journal of Finance, 68(6):2309-238) 2340, 2013.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86 92, 2021.
Amit Goyal and Pedro Santa, Clara. Idiosyncratic risk matters! The Journal of Finance, 58(3), 2003.
Piyush Gupta, Nikaash Puri, Sukriti Verma, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh. Explain your move: understanding agent actions using specific and relevant feature attribution. In International Conference on Learning Representations, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1861-1870, 2018.
Charles Harris, K Jarrod Millman, Stéfan Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array programming with numpy. Nature, 585(7825), 2020.
Conor F Hayes, Roxana Rädulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathicu Reymond, Timothy Verstraeten, Luisa. M Zintgraf, Richard Dazeley, Fredrik Heintz, al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agena Systems, 36 (1), 2022.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI conference on artificial intelligence, 2018.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint SrXiv:1508.02533 2015.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(9):5149-5100 2021.
Zhengyao Jiang, Dixing Xu, and Jinjun Liang. A deep reinforcement learning framework for the financial portfolio management problem. arXio preprint arXiw:1706.10059. 2017.
Scott Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip Thomas. Evaluating the performance of reinforcement learning algorithms. In International Conference on Machine Learning, pp. -972-4973, 2020.
17

---

Published in Transactions on Machine Learning Research (03/2023)
Lou Jost. Entropy and diversity. Oikos, 113(2), 2006.
Jonathan M Karpoff. The relation between price changes and trading volume: A survey. Journal of Financial and Quantitative Analysis, 22(1):109-122 -200 1987.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yar Liu. Lightgbm:: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, 2017.
Ajay Khorana, Henri Servaes, and Lei Wedge. Portfolio manager ownership and fund performance. Journal of sinancial economics, 85(1):179 204, 2007.
Ulrich Kirchner and Caroline Zunckel. Measuring portfolio diversification. arXiv preprint arXiv: 1rXiu:1102.4722 2011.
Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. Aduances in Neural Information Processing Systems, pp. 18560-18572, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances Neural Information Processing Systems, 2017.
Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep reinforcement learning. In International Conference on Machine Learning, pp. 5979 5989, 2021.
Chi-Chuan Lee, Chien-Chiang Lee, and Yizhong Wu. The impact of covid-19 pandemic on hospitality stock returns in china. International Journal of Finance & Economics, 2021a.
Chien-Chiang Lee, Jun-De Lee, and Chi-Chuan Lee. Stock prices and the efficient market hypothesis: Evidence from panel stationary test with structural breaks. Japan and the world economy, 22(1):49-55, 2010.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, 2021b.
Siyu Lin and Peter A Beling. An end-to-end optimal trade execution framework based on proximal policy optimization. In International Joint Conference Artificial Intelligence, pp. 4548-4554, 2020.
Xiao- Liu, Hongvang Yang. Qian Chen, Runjia Zhang, Liuqing Yang, Bowen Xiao, and Christina Dan Wang. Finrl: A deep reinforcement learning library for automated stock trading in quantitative finance. arXiv preprint arXiv:2011.0960) 2020a.
Yang Liu, Qi Liu, Hongke Zhao, Zhen Pan, and Chuanren Liu. Adaptive quantitative trading: An imitative deep reinforcement learning approach. In AAAT Conference on Artificial Intelligence, pp. 1128-2135, 2020b.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Marlos Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523 562, 2018
Malik Magdon and Amir F Atiya. Maximum drawdown. Risk Magazine, 17(10):99 102, 2004.
Burton G Malkiel. The efficient market hypothesis and its critics. Journal of Economic Perspectives, 17(1): 59-82, 2003.
Velayoudoum Marimoutou, Bechir Raggad, and Abdelwahed Trabelsi. Extreme value theory and value risk: Application to oil market. Energy Economics, (1), 2009.
18

---

Published in Transactions on Machine Learning Research (03/2023)
Mieszko Mazur, Man Dang, and Miguel Vega. Covid-19 and the march 2020 stock market crash. evidence from s&p1500. Finance research letlers, 38, 2021.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timmit Gebru. Model cards for model reporting. In ACM Conference on Fairness, Accountability, and Transparency, pp. 220-229, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nalure, 518(7540):529 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukeuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pp. 1928 1937, 2016.
Tobias J Moskowitz, Yao Hua Ooi, and Lasse Heje Pedersen. Time series momentum. Journal of Financial Economics, 104(2):228 250, 2012.
Martin Mundt, Steven Lang, Quentin Delfosse, and Kristian Kersting. Cleva-compass: continual learning evaluation assessment compass promote research transparency and comparability. In International Conference on Learning Represenlations, 2021.
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy, Deep exploration via bootstrapped dqn. Advances in Neural Information Processing Systems, 2016.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 2009.
Robert Pardo. The evaluation and optimization of trading strategies, volume 314. John Wiley & Sons, 2011.
Jack Parker, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning, Advances in Newral Information Processing Systems, pp. 18050 18062, 2020.
Julien Pénasse.
Understanding alpha decay. Monagement Science, 2022.
Joelle Pineau, Philippe Vincent- Vincent-Lamarrre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research: A report from the neurips 2019 reproducibility program. Journal of Machine Learning Research, 22(1):
7459 7478, 2021.
Fábio Pinto, Marco OP Sampaio, and Pedro Bizarro. Automatic model monitoring for data streams. arXiv preprint Xiv::11908. 04240, 2019.
Fazlollah M Reza. An introduction to information theory. Courier Corporation, 1994.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should i trust you?" explaining the predictions of any classifier. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135 1144, 2016.
Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, and Rajiv Ratn Shah. Spatiotemporal hypergraph convolution network for stock movement forecasting. In 2020 IEEE International Conference on Dala Mining (ICDM), pp. 482-491, 2020.
Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, and Rajiv Shah. Exploring the scale-free nature of stock markets: Hyperbolic graph learning for algorithmic trading. In Acm Web Conference, pp. 11 11-22, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint or N07007077.03477 2017.
William F Sharpe. The sharpe ratio. Journal of Portfolio Management, 1998.
19

---

Published in Transactions on Machine Learning Research (03/2023)
Wenjie Shi, Gao Huang, Shiji Song, Zhuoyuan Wang, Tingyu Lin, and Cheng Wu. Self-supervised discovering of interpretable features for reinforcement learning. TEFE Transactions on Pallern Analysis and Machine Intelligence, 44(5):2712 2724, 2020.
Robert J Shiller. Market volatility. MIT press, 1992.
Andrew Silva, Matthew Gombolay, Taylor Killian, Tvan Jimenez, and Sung-Hyun Son. Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 1855-1865, 2020.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithuns. In International Conference on Machine Learning, pp- 387-395, 2014.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676), 2017.
Thomas Spooner and Rahul Savani. Robust market making via adversarial reinforcement learning. In International Joint Conference Artificial Intelligence, 2020.
Thomas Spooner, John Fearnley, Rahul Savani, and Andreas Koukorinis. Market making via reinforcement
learning. In International Conference on Autonomous Agents and MulliAgent Systems, 2018
Shuo Sun, Wanqi Xue, Rundong Wang, Xu He, Junlei Zhu, Jian Li, and Bo An. Deepscalper: A risk-aware reinforcement learning framework to capture fleeting intraday trading opportunitiss. In ACM International Conference on Information & Knowledge Management, pp. 1858-18666 2022.
Shuo Sun, Rundong Wang, and Bo An. Reinforcement learning for quantitative trading. ACM Transactions on Intelligent Systems and Technology, 2023.
Jun Tu and Guofu Zhou. Markowitz meets talmud: A combination of sophisticated and naive diversification strategies. Journal of Financial Economics, 99(1):204 215, 2011.
Laurens Vander and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9 (11), 2008
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782), 2019.
Rundong Wang, Hongxin Wei, Bo An, Zhouyan Feng, and Jun Yao. Commission fee is not enough: A hierarchical reinforeed framework for portfolio management. In AAAI Conference on Artificial Intelligence, pp. 626-633, 2021a.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image foreign language: Beit pretraining for all vision and vision- languag tasks. arXiv preprint arXiv: 10422, 2022.
Zhicheng Wang, Biwei Huang, Shikui Tu, Kun Zhang, and Lei Xu. Deeptrader: A deep reinforcement learning approach to risk-return balanced portfolio management with market conditions embedding. In AAAT Conference on Artificial Intelligence, pp. 643 650, 2021b.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2019.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. Advances in Neural Information Processing Systems, pp. 6514-6527, 2020.
Yumo Xu and Shay B Cohen. Stock movement prediction from tweets and historical prices. In Annual Meeting of the Association for Compulational Lánguistics, pp. 1970-1979, 2018.
20

---

Published in Transactions on Machine Learning Research (03/2023)
Yunan Ye, Hengzhi Pei, Boxin Wang, Pun-Yu Chen, Yada Zhu, Ju Xiao, and Bo Li. Reinfoceement-eamnigg based portfolio management. with augmented asset movement prediction states. In AAAT Conference Artificial Intelligence, pp. 1112 1119, 2020.
Jaemin Yoo, Yejun Soun, Yong-chan Park, and U Kang- Accurate multivariate stock movement prediction via data-axis transformer with multi-level contexts. In ACM SIGKDD Conference Knowledge Discovery & Dala Mining, pp. 2037 2045, 2021.
Dong Yu, Kaisheng Yao, Hang Su, Gang Li, and Frank Seide. Kl-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition. In TEEE International Conference Acoustics, Speech and Signal Processing, 2013.
Yuyu Yuan, Wen Wen, and Jincui Yang. Using data augmentation based reinforcement learning for daily stock trading. Electronics, 9(9), 2020.
Zihao Zhang, Stefan Zohren, and Stephen Roberts. Deep reinforcement learning for trading, The Journal of Financial Data Science, 2(2), 2020.
Brian D Ziebart. Modeling purposeful adaptive behawior with the principle of marimum causal entropy. Carnegie Mellon University, 2010.
21

---

Published in Transactions on Machine Learning Research (03/2023)
Appendix
A PRUDEX-Compass
A.1 Definition Financial Metrics
Profit measure contains metrics to evaluate FinRL methods' ability gain market capital. Total return (TR) is the percent change of net value time horizon h. The formal definition is TR (Neth -4)/ where ny is the corresponding value at time t.
Risk includes a class of metrics to assess the risk level of FinRL methods.
Volatility (Vol) is the variance of the return vector r. It is widely used measure the uncertainty of return rate and reflects the risk level of strategies. The definition is Vol ir].
Maximum drawdown (MDD) measures the largest decline from the peak in the whole trading period to show the worst case. The formal definition MDD max+e((,0) [max)e [ n==Re] nr
Downside deviation (DD) refers to the standard deviation of trade returns that are negative.
Risk-adjusted Profit calculates the potential normalized profit by taking one share of the risk. We define three metrics with different types of risk:
Sharpe ratio (SR) refers the return per unit of deviation: SR Fir
a
Sortino ratio (SoR) is another risk-adjusted profit, which applics DD risk mcasure: SoR
E(rl DD
Calmar ratio (CR) another risk-adjusted profit, which applies MDD as risk measure: CR MDD
E[r]
A.2 Creating a PRUDEX-Compant
To make the PRUDEX-Compen accessible possible and disseminate in convenient way, provide two options for practical use based the template provided in CLEVA-Compes. (Mundt et al., 2021).
We provide LaTeX template for PRUDEX- Compass, making use of the TikZ library to draw the compass with. We envision that such template makes it easy for readers include a compass into their future research, where they adapt the naming and values of the entries respectively.
We further provide a Python script to generate the PRUDEX- PRUDEX-Compass. In fact, because the use of drawing LaTeX with TikZ may be unintuitive for some, we have written Python script that automatically fills above LaTeX template, that it can later simply be included into LaTeX document. The Python script takes path to JSON file that needs to be filled by the user. There is default JSON file that is easy to adopt.
A.3 Computing Scores for PRUDEX- Coupaxs Axes
We introduce how the value of each axe for each FinRL method is computed in this subsection. The basic principle is to propose distinguishable and robust way show the performance difference of FinRL methods across multiple evaluation measures in terms of each of different measures on the 6 axes to score from 0 to 100. We mark market average strategy (evenly invest on all assets) as 50. All scores are calculated with the average of financial markets. We define the metrics value for FinRL methods and market average as Myt and moves respectively. For the 4 profit-related metrics (TR, SR, CR, SoR), we normalized them into score Syro with range [0, 100): Spro = (myz/Mave /Mqve 0.8) 250, where 20% higher profit than market average is scored 100. We clip values lower than and higher than 100 and 100, respectively. For the 2 risk metrics (Vol, MDD), normalized them into score Srisk with
Generally speaking, we normalize the performance
22

---

Published in Transactions on Machine Learning Research (03/2023)
range [0, 100]: Srisk (1.2 - myt/mave 250, where 20% lower risk than market average is scored 100. We clip values lower than 0 and higher than 100 as 0 and 100, respectively.
For universality, we directly use the raw return rate and calcualte the 4 indicators of profitability, we then plot a rank graph and for each measures of profitability, we multiply the probability obtained from the rank matrix and the rankscore(if it's 1st, the score 100 and 6th, the score 0) to get a score for that measure. Then we average the 4 measures together get the university score for that algorithm. For the 2 diversity metrics (ENT, ENB), normalized them into score diw
mas/Mace X 100, for ENT
Saic
myt/ myt/ Mave x 50, for ENB
where the diversity of uniform policy is scored 100 for ENT and 50 for ENB. For explainability, we set the explainability score 50 for all FinRI. methods. For reliability, we use the total return rate score we just normalized using average policy as a indicator and draw a performance profile graph, then we use the area under the curve for each algorithm calculate the reliability. The constants in the equations (e.g., 20%) are applied make the plot distinguishable and easy to follow, which does influence the robustness of axes in TRUDEX-COmpase
B Experiment Setup
B.1 Training Setup
Table 4: Hyperparameters of AlphaMix+
Hyperparameter
Value
Hyperparameter
Value
Replay buffer size
10000
Initial step
10000
Laycr(MLP)
1128,128)
Stacked frame
3
Evaluation episodes
10
Optimizer
Adam
Temperature
20
Uncertainty
0.5
Actor learning rate
0.0007
Critic learning rate
0.0007
Batch size
256
Action numbers
29(US) 47(hiina) 22(FX) 9(Crypto)
Discount 7
0.99
Ber mean
0.5
Non-linear
Sigmoid
Observation
Number of assets X 11
C Experimental Results
C.1 Result Table
In this subsoction, we report detailed results of metrics in the four financial markets. Since we apply year rolling window during training, each financial market has tables for consecutive years.
Table US Stock 2021
Metrics
A2C
PPO
SAC
SARL
DeepTrrader
ELIE
IMIT
AlphaMix+
TR(%)
12.416.3
13.114.4
20.0110.3
17.312..
17.217.4
16.9=1.2
4.0.10.11
20.741.8
1.03+0.50
0.9410.26
1.30+0.57
1.37+0.12
1.22+0.50
1.39+0.08
0.35+40.64
1.64+0.11
CR
0.8140.34
0.8310013
0.9190.19
1.06-:0.33
0.8410.23
1.06-:0.02
0.47400.1
1.0910.02
SoR
1.52+0.70
1.38+0.44
1.89+0.87
2.024:0211
18110.775
2.03+0.11
0.52+0.911
2.36:0.17
MDD(%)
15.04:.9
15.342.5
19.7444.6
15.64112
19.143.9
15.2=0.6
14.643.7
17.9+14
VOL(%%
0.78-10.06
0.8740.03
0.9120.02
0.76110.2
0.86400.3
0.73+0.03
1.0:0.112
0.74
ENT
2.26:10.26
1.8210.02
1.6740.02
2.7910.11
1.8210.01
2.90+1.26
1.8940.1
3.25
ENB
1.34+0.10
1.49+0.01
1.61+0.02
1.14=0.06
1.51+0.02
1.1940.10
1.7340.10
1.11+0.03
23

---

Published in Transactions on Machine Learning Research (03/2023)
Table US Stock 2020
Metrics
A2C
PPO
SAC
SARL
Deep'rrader
ELIE
IMIT
AlphaMix+
TR(%)
7.4624.86
18.3+8.4
7.77+21.1
9.23=8.79
10.2+10.5
15.8#16.1
-20.699.5
12.7
0.3611.12
0.6211019
0.3710.52
0.4110.21
0.4210.25
0.55.10.30
-0.2810.21
0.52.10.04
0.3611.12
0.54100.1
0.30+0.48
0.38=0.19
0.42100222
0.49:10.22
-0.2810022
0.47:10.03
SoR
0.4442..14
0.76#00233
0.50+0.63
0.50+0.26
0.56:0.33
0.68
-0.36++.227
0.62++0.05
(DD(%)
36.61224
42,2424
42.8:55.3
37.8=3.19
34.913.91
38.926.69
43.526.42
38.240.96
(OL(%%
2.25=+0.99
2.32+0003
2.4510.16
2.2610.17
2.31
0.21
2.9 0.39
2.16+0004
ENT
1.96=0.20
1.82=0.02
1.61=0.03
2.07=0.67
1.82+0.02
2.41=0.67
1.85 #0.21
3.22+0.03
ENB
1.05#0001
1.00+0.004
1.140.003
1.05=0.03
1.05#0.002
1.05=0.05
1.13# 0.04
1.0 0.006
Table 7: US Stock 2019
Metrics
A2C
SAC
SARL
DeepTrader
EII
IMIT
AlphaMix +
TR(%)
20.519.27
21.5110.1
27.319.79
27.711..3
22.913.51
30.2:1842
20.619.52
25.111..22
SR
1.47:00.88
1.53+:0.22
1.72+0.54
1.00
1.60+0.24
2.12+0.27
1.28+0.21
1.98+0.006
1.00=20.06
1.0140.06
1.05+00.09
1.0240.05
1.01
1.06 20.05
0,28:0.22
1.03:0.000
SoR
1.96:0.78
2.0440.83
2.1710.78
2.52:1..4
2.16:0.43
2.82:10422
0.36+10.27
2.47-10.09
MDD(%%
18.8:45.95
19.446.26
23.4+4.70
24.742..3
21.242.41
25.2+44.83
43.622.42
22.3=20.99
VOL(%%
0.82110011
0.8310.02
0.9110.05
0.7910.11
0.84
0.79
1.9.0.39
0.73:10.02
ENT
1.83=0.01
1.82+0.01
1.63+0.02
2.08=0.67
1.81#0.01
2.32+0048
1.89+0.116
3.27=0.03
ENB
1.28:+0.01
1.27+0.01
1.38+0.01
1.2940.21
1.26::0.005
1.15#0.06
1.73++0.10
1.05# 0.01
Table China Stock 2020
Metries
A2C
PPO
SAC
SARL
DeepTrader
EIFE
IMIT
AlphaMix+
TR(%)
5.524-4.82
6.2027.57
13.4::20.1
11.428.84
12.3+14.2
13.0=3.8
52.6420.8
14.843.30
SR
0.3510.21
0.4010.37
0.59:1.68
0.6311.41
0.62:10.0
0.731 0.17
2.6810.83
0.7940.12
0.25+00.15
0.29=0.27
0.4140.48
0.42-0.27
0.36+0.39
0.51+00.8
1.1840.24
0.53+0.06
SoR
0.40+0.24
0.45=0.42
0.7140.81
0.71=0.46
0.71+0.69
0.80++0.18
4.04+1.2
0.89+0.15
MDD(%)
28.113368
25.311.59
29.9.17.88
26.5.15.09
31.5:16.22
27.012241
34.219.16
29.111..5
VOL(%)
0.74+0.04
0.68=0.004
0.81+0.06
0.68+00.2
0.76+0001
0.67+00.02
0.66+00.09
0.69+0.01
ENT
1.85+00.42
2.82=0.01
1.10+0.02
2.60
1.53++0.001
2.39-+..10
1.30 +0.85
3.12+0.02
ENB
1.1240.005
1.04=0.01
1.2740.01
1.04
1.1640.004
1.06: 0.01
2.82# 0.85
1.0240.01
Table 9: China Stock 2019
Metries
A2C
PPO
SAC
SARL
DeepTrader
EIE
IMIT
AlphaMix
TR(%)
31.9+10.6
29.7#8.42
25.4+10.4
32.6=5.78
22.1+13.6
36.2= 7.09
-7.1440.82
32.242.20
SR.
1.59+0.46
1.65:t0.39
1.13:t0.88
1.80+0.24
1.19+0.59
1.77= 0.05
-0.34=0.1
1.79+0.10
CR
0.90-10.16
0.94+:0.13
0.79
1.01+:0.7
0.74+00.1
1.04= 0.05
-0.29+10007
1.01
SoR
2.32:10.75
2.411:0.63
1.70:10.64
2.63+:0.34
1.72110.91
2.57= 0.15
-0.37 0.08
2.6210.14
MDD(%)
31.7.12.85
28.6.12.82
29.8.12.88
28.912.38
27.514.92
30.5.13.74
20.4. 0.63
28.910..0
VOL(%)
0.65.10.02
0.58.10.004
0.74.1002
0.5710001
0.63110.02
0.64.10.11
0.77.L 0.06
0.57.0.01
ENT
1.54-10.0
2.85110000
1.02-0.02
2.47-0.17
1.531100009
1.97-10.00
1.30-10.85
3.15.0..07
ENB
1.18:10.009
1.05:+0.002
1.3240.009
1.06=0.01
1.18:0.004
1.21 10.18
1.19:10.85
1.0400.01
Table 10: China Stock 2018
Metries
A2C
PPO
SAC
SARL
DeepTrader
EHE
IMIT
AlphaMix+
TR(%)
1.86++11.30
6.56+33.6
1.05++10.77
-5.77444.04
5.67+55.5
2..7711.99
-7.14 0.82
-0.70+1.38
SR
-0.2841061
-0.3110.22
-0.1240.50
-0.2540.21
0.3740.30
-0.05- 0.12
-0.3410.1
0.03400.88
CR
-0.1440.50
-0.2540.16
-0.06+1.42
-0.2040.15
0.40+0.32
-0.0440.11
-0.29110007
0.0440.10
SoR
-0.33310.78
-041100.2
-0.15.10.75
-0.35-10.99
0.54.0..5
-0.07:1
-0.3710.08
0.04.00.11
MDD(%)
22.3+5.52
20.3+1.55
25.9+4.95
20.243..9
19.1
17.5=1.52
20.4+0.63
16.1+:1.4
VOL(%)
0.67:t0.88
0.59
0.75+10.08
0.611:0.88
0.66+00002
0.59t 0.03
0.77100.06
0.57:t0.02
ENT
1.5410.01
2.85:00.007
100100.03
24110040
L.53=0.007
2.55=0.15
1.304 0.85
3.12+:0.02
ENB
1.3140.01
1.08+00005
1.3740.009
1.11+0.10
L.30=0.099
1.05 0.01
1.19 #0.08
1.03++0.03
24

---

Published in Transactions on Machine Learning Research (03/2023)
Table 11: Crypto 2021
Metrics
A2C
РРО
SAC
SARL
DoepTrader
ETIE
IMIT
AlphaMix
TR(%)
2231:446
146=122
3771-882
199-1195
3981344
14641337
1404 155
291171
SR
1.38+0.74
1.22=0.56
1.13+0667
1.41=
1.71
1.46# 0.42
1.24+0.34
1.87::0009
CR
1.77
1.48=0.53
2.1852005
1.75+0.774
2.18:00.4
1.44:10.49
1.39:1058
2.02+:0.6
SoR
2.15#11.23
2.13=1.11
3.1643.31
2.42=1.244
2.97+1.35
2,42 1.22
2.032 0.80
3.142:..56
MDD(%)
77.1115.7
74.9210.4
80.1115.6
79.4
85.118.42
72.5:110.2
85.71:4.1
VOL(%)
7.3011191
6.9710.94
10.9418.13
7.26112.31
7.85:12.05
5.22.11.06
5.561
6.8010.84
ENT
1.0240.31
0.72=0.05
0.47+0.00
1.42=0.25
0.56+0.04
1.53+0.23
0.58+0.23
2.09+0.02
ENB
1.7910.15
1.94:0.04
1.9910.05
1.67:00.0
1.97:0.03
1.12:0.05
1.6610.38
1.47.0.15
Table 12: Crypto 2020
Metrics
A2C
PPO
SAC
SARL
DeepTrader
EIE
IMIT
AlphaMix+
TR(%)
61.4158.1
59.1158.4
16.01138.4
31.4112.8
37.9117.1
30.71 11.7
55.31
17.6
33.816.16
SR
1.1240.66
1.12=0.67
0.51=0.50
0.8140..2
0.85
0.79+0.11
1.3#
0.86+0.08
CR
1.06:0.47
1.02=0.45
0.59+0.58
0.86+00.33
0.97::0.2
0.82+0.19
0.73
0.87#0.09
SoR
1.58411.00
1.62-1.22
0.5910063
0.89:40.29
1.0440.54
0.9240018
1.49 0.32
0.94::0.10
MDD(%)
52.9447.88
52.2-8.03
55.712211
46.6418.8
50.3:+8.04
44.6:13.22
56.61 4.43
46.54:2.43
VOL(%)
3.86:10.22
3.75-10.17
4.46:10.35
3.66:10.76
4.05:10.32
3.37: 0.36
2.86 0.53
3.46:0.22
ENT
0.59:0.004
0.09
0.45=20.02
1.3240.45
0.60+0.01
1.24:0.552
0.58+0.52
2.17#00007
ENB
1.05:10.01
1.06.10.03
1.0710.003
1.011100005
1.05.10.01
1.01100011
1.0110.01
1.00:1.0011
Table 13: Crypto 2019
Metrics
A2C
PPO
SAC
SARL.
DeepTrader
EIIE
IMIT
AlphaMix
TR(%)
82.8
85.8+57.1
73.94:33.9
61.0=19.3
39.145545
50.8+13.2
110.64 17.6
68.3=17.8
SR.
1.37=0.50
1.39+0.55
1.27+0.33
1.26=0.26
0.83+0.59
L15+0.20
2.06 0.08
1.43=0.22
1.14.10.30
1.14.10.3
1.06.10.24
1.06.10.11
0.73.10.39
0.9810.10
1.45.L 0.07
1.08
SoR
2.19-10.98
2.2211.02
2.10410.5
1.80-0.30
1.3041.12
L.7740.28
2,991 0.32
2.16-0.35
MDD(%)
59.6
50.9110.5
59.147.74
52.4-6.49
51.3411.3
49.945.80
56.61 4.43
55.5-3.58
VOI((%)
3.69:10.36
3.69:10.35
3.63:10.27
3.29-10.5
3.46:10.24
3.15:00.30
2.86:t 0.53
3.09-10.18
ENT
0.60:10.02
0.62.10.03
0.43:10.03
1.2110.40
0.5910.02
1.24.1..22
0.58.10.22
2.15.10.10
ENB
1.15.10.006
1.14.0..01
1.1510.008
1.06.10.02
1.14.0.01
LOL10.01
1.0110.01
1.0110.01
Table 14: Foreign Exchange 2019
Metries
A2C
РРO
SAC
SARL
Deep'Irader
ETE
IMIT
AlphaMix+
TR(%)
-0.9443.14
-1.4942.68
-1.0243.11 11
0.96:40.3
-0.19+2.88
144200.24
-5.534 0.05
1.22+0.66
SR.
0.2220073
-0.34+0.64
-0.2140.64
0.29+40.99
-0.03++0.66
0.39=0.05
-0.97 0.01
0.41+0.16
CR
-0.0740.42
-0.16++.33
-0.02++0.3
0.22+00.8
0.04+0.46
0.32=0.07
-0.49 0.01
0.36+0.17
SoR
-..32+1.05
+..51
-0.26+0.86
0.50+0.17
0.06+:1.11
0.65=0.07
-1.5t 0.02
0.74+0.31
MDD(%)
7.90=1.85
6.28111.7
6.941.2.7
4.60:t0.47
6.84
4.65.t 0.43
11.17 0.05
3.7740.56
VOL(%)
0.27
0.2811.01
0.3110.01
0.22:10.11
0.26:10.01
0.23
0.25 10.01
0.19:0.01
ENT
1.44=0.08
1.3740.01
0.93+0.04
2.55+ 0.10
1.35+0.02
2.23=0.08
3.08+0.01
2.9740.04
ENB
1.65=0.06
1.73+0.01
2.02++0.04
1.18+0.10
1.71+00.33
1.16=0.06
1.08+0.01
1.18+0.06
25

---

Published in Transactions on Machine Learning Research (03/2023)
Table 15: Foreign Exchange 2018
Metrics
A2C
PPO
SAC
SARL
Deep'Trader
BIIE
IMIT
AlphaMix+
TR(%)
-..4312.29
-5.25.12.45
-3.61
-..9211.91
-..99.2.82
-6.15.0.28
-5.331 0.05
-4.9210.32
SR
-1.01110.4
-0.9710.49
-0,9110.43
-0.9310.41
-0.9110.55
-1.15.10.14
-0.971
-1.15.10.10
CR
-0.5010.14
-0.4811.11
-0.50-0.18
-0.48-1.199
-0.4910.21
-0.5910.04
-0.491 0.01
-0.5710.03
SoR
-1.43=0.64
-1.36.1068
-1.23=20.44
-1.42=0.62
-1.2940.74
-1.6610.26
-1.54 0.02
-1.7710.16
MDD(%)
10.421.67
10.4+1.85
10.7+1.80
9.08=0.96
9.41+2.20
10.6::.91
11.2# 0.05
8.66:0.14
VOL(%%
0.34110.02
0.34:10.22
0.37.10.03
0.31
0.3410001
0.34110.33
0.25.1
0.27:0.01
ENT
1.38.10.03
1.34.0.03
2.23.10.77
1.3710.00
1.5310.72
3.08.
2.98.10.03
ENB
1.4510.01
1.46:10.33
1.65.10.05
1.16:10.00
1.4510.02
1.3910.27
1.0810.01
11110.01
Table 16: Foreign Exchange 2017
Metrics
A2C
SAC
SARL
DeepTrraee
EIIE
IMIT
AlphaMix+
TR(%)
6.9311.71
6.8511.44
8 8.25113.62
3.81122.6
5.9412.50
6.7112.72
6.42 0.16
7.16:10.36
SR
1.34=00229
1.32+0.24
1.41+0.48
1.34+0.57
1.13+0.51
1.26:0.47
0.81# 0.02
1.72=0.13
CR
0.82:10112
0.8110.12
0.8810.17
0.79110.2
0.77:10.1
0.8810.11
0.73. 0.01
0.93-1..2
SoR
2.15::0.45
2.14+0.42
2.17#1.01
2.28+0.97
1.74+0.89
1.89+0.80
1.21 0.03
3.04
MDD(%)
8.21+1.28
8,20+1.12
9.014:2.45
7.1241.27
7.46+1.87
7.3342208
8,98 0.12
7.61-0.43
VOL(%%
0.32:10.01
0.32+00.11
0.35:10.03
0.28+00.4
0.33+10.2
0.34400.8
0.36:t 0.01
0.25+10001
ENT
1.3420002
1.35+00.02
0.90::0.03
2.17=0.44
1.37+0.01
14140.57
3.08
2.98=0.10
ENB
1.58:0.011
1.59+0.04
1.82+0.02
1.29-0.15
1.87+0.02
1.55+0.188
1.05=0.04
1.10=0.04
C.2 Ablation Studies
Models
Ensemble
Weight BB
Diveristy
TR(%))
SR7
CR1
SoRt
MDD(%)
VOI.(%)
ENTT
ENB1
SAC
7.77
0.37
0.30
0.50
42.8
2.45
1.61
1.10
10.2
0.46
0.50
0.55
31.6
2.15
3.25
1.01
-ix
10.1
0.45
0.48
0.54
32.3
2.15
3.28
1.01
10.9
0.47
0.52
0.57
32.0
2.19
3.26
1.02
12.7
0.52
0.47
0.62
38.2
2.16
3.22
1.00
Table 17: US Stock 2020 Ablation
Models
Ensemble
Weight BB Diveristy
TR(%)
SR²
CRf
SoRt
MDD(%)
VOL(%)
ENI1
ENB1
SAC
13.4
0.59
0.41
29.9
0.81
1.10
1.27
13.6
0.75
0.77
0.83
19.0
0.69
3.15
1.01
AlphaMix-
11.6
0.66
0.68
0.73
18.7
0.68
3.09
1.02
12.0
0.68
0.70
0.76
18.7
0.68
3.20
1.02
14.8
0.79
0.69
1.02
Table 18: China Stock 2020 Ablation
Models SAC
Ensemble
Weight
Diveristy
TR(%)T
SR
CRT
SoRt
MDD(%)
VOL(
ENTT
ENB1
16.0
0.51
0.59
0.59
55.7
4.46
0.45
1.07
24.4
0.73
0.72
0.76
44.8
3.26
2.08
1.00
Mixh
v
27.7
0.77
0.78
0.82
46.5
3.42
2.14
1.00
32.2
0.83
0.86
48.0
3.62
2.19
1.00
V
33.8
0.86
0.87
0.94
46.5
3.46
2.17
1.00
Table 19: Crypto 2020 Ablation
Models
Ensemble
Weight BB Diveristy
TR(%)
SR1
CR
SoRt
MDD(%)
VOL(%)
ENTY
ENB1
SAC
-1.02
-0.21
-0.02
-0.26
6.94
0.31
0.93
2.02
1.06
0.33
0.26
0.58
4.24
0.21
2.98
1.22
1.11
0.34
0.27
0.60
4.47
0.22
2.91
1.19
1.04
0.33
0.25
0.57
4.34
0.21
3.01
1.19
1.22
0.41
0.36
0.74
3.77
0.19
2.97
1.18
Table 20: Foreign Exchange 2019 Ablation
26

---

Published in Transactions on Machine Learning Research (03/2023)
C.3
Parameter Analysis: Probing Sensitivity
Total Return 196)
Total Return (56)
300
300
022
200
22
100
150
230
Binomial Sample B
Temperature
Scale Factor k
25
Crypto
Crypin
Crypto
Sharpe RaDo
2.1
Sharpe Rati
22-
2.2
2.3
0.0
3A5
6.55
2.00
1.0
18.1
20.0
21.
222
20
Binomial Sample B
Temperature
Scale Factor k
Figure 10: Hyperparameter Sensitivity Experiment Results on Crypto
FX
Total Return (90)
Total Return (%)
3.-
3.7
1.34
2,5
0.35
20.0
210
Binomial Sample B
Temperature T
Scale Factor k
0.43
Sharpe Rabo
Sharpe Ratio
0.48
2.
Sharpe
0.2
0.14
22
0.35
Binomial Sample B
Temperature T
Scale Factor k
Figure 11: Hyperparameter Sensitivity Experiment Results on FX
27

---

Published in Transactions on Machine Learning Research (03/2023)
C.4 PRIDE-Star Star
C8
5R
3R
SR
SoR
Sou
TR
-9
MNC
60
ADDD
TNT
aday
1N3
AIDD
1NT
AIDD
JN3
(a)
(b) PPO
(O)
SAC
(d) SARL
16
9R
Solf
Soltt
Saik.
TR
49
ENTT
OCDP
183
1N4
OCUV
1N3
(e) DeepTrader Trader
(f) EITE
(g) IMIT
(h) AlphaMix
Figure 12: PRIDE-Star on US Stock
SR
512
Solt
SoRt
Soutt
Stitt
SINEE
ONEE
A9A
KENGE
OCPP
1N3
1NT
OdTY
(a)
A2C
(b) PPO
(c) SAC
(d) SARL
.SR
SR
3R
SR
Solt
Solt
Sak.
adp
JA3
adly
143
CON
AN3
OCW
1A3
(c) DeepTraaer
(I) EHE
(s) IMIT
(h) AlphaMix+
Figure 13: PRIDE-Star on FX
28

---

Published in Transactions on Machine Learning Research (03/2023)
Solt
Sult
Selt
TL
14
AN3
MUD
ENT
(8)
A2C
(b) PPO
(c) SAC
(d) SARL
SR
9R
Solt
SAR
SeR
JAR
ddDy
1831
GOW
AN3
183
(e) Deep Trader
(I) EITE
(g) 1MIT
(h) AlphaMix+
Figure 14: PRIDE-Star on China Stock
Soke
Solt
Solt
TR
SeRt
100
KENEE
M00
1N3
OCDV
1NE1
EXT
OCW
1N.1
(a) A2C
(b) PPO
(c) SAC
(d) SARL
5R
5R
5R
SeRt.
Seitt
Seat
BAAC
OOFY
GGN
184
OGPY
(e) Deep'Tree Trader
(I) ETIE
(g) IMIT
(h) AlphaMix+
Figure 15: PRIDE-Star on Crypto
29

---

Published in Transactions on Machine Learning Research (03/2023)
C.5 Performance Profile
A2C
PPO EIIE
SAC
SARL
A2C
PPO
SAC
SARL
DeapTrader
AlphaMix+
DeepTrader
EIIE
IMIT
AlphaMii+
1.00
1:00
Fraction runs with SCOrO
with score
0.75
0.75
0.25
0.00
0.00
60
40
100
total return score (rl
score
(a) Crypto
(b) US Stock
A2C
PPO
SAC
SARL
A2C
PPO
SAC
SARL
DeepTrader
Alpha
DeepTrader
AlphaMi++
1.00
1.00
Fraction of runs with score
0.75
0.75
Fraction of
0.25
0.25
0.00
0.00
20
40
100
60 total return score (v)
100
total return score (r)
(c) FX
(d) China Stock
Figure 16: Performance profile on influential financial markets
C.6 Rank Distribution
DeepTrader
EIIE
IMIT
AlphaMix
Dee DeepTrader
EllE
IMIT
AlphaMix
TR
SR
VOL
Entrapy
TR
5R
VOL
Entropy
100
100
200
200
100
Fraction (nn
60
50
60
50
60
60
60
60
Fraction
60-
40
40
60
40
60
12345676
12345670
12345678
12345670
0-
12.3-5670
323A5578
32345678
12345670
(a) Crypto
(b) US Stock
Trader
Trader
EIEE
IMIT
AlphaMix-
Deep Trader
EII
IMIT
AlphaMixc
Tk
SK
VOL
Enkrogy
TR
SR
VOL
Entropy
100
100
100
100
100
100
lin
60
60
60
60
Fraction (in
60-
60-
60-
60
40
40
40
40
40
32245478
12345679
12345678
12245678
12245678
12345678
12245678
12245678
(c) FX
(d) China Stock
Figure 17: Rank distribution on influential financial markets
30


Reviews:
[{"review_id": "ugSIs888BQ", "reviewer": "Reviewer_95pS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.The paper is clearly written.\n2.The paper propose an evaluation framework for FinRL algorithms, which has great value for RL and FinRL community.\n3.The visualization of algorithms is good.\n\nWeaknesses:\n1.The proposed AlphaMix+ algorithm lacks the ablation study, which part is the most critical for the performance?\n2. The variance of AlphaMix+ is larger than baselines.\n3. There seems to be 2 parameters to be tuned, which may cost much computation costs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to provide a systematic evaluation of FinRL algorithms in Financial Markets. The paper propose PRUDEX-Compass, which includes 6 dimensional metrics,  Profitability, Risk-control, Universality, Diversity, rEliability, and eXplainability. AlphaMix+ is proposed as a strong baseline, which includes Risk-aware Bellman Backup and diversified experts. 8 FinRL methods are evaluated on 4 real-world financial datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.Provides detailed ablation study of AlphaMix+.\n2.Specify more details on the RL environments in the experimental parts.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YY9t6YQg4M", "reviewer": "Reviewer_NTff", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Holistic evaluation is critical for high-stake settings, and this paper takes a step toward that for the financial market.\n\n2. Extensive open-source, practical, library for evaluating and visualizing the performance of different methods.\n\nWeakness:\n1. The problem setup is unclear (see below)\n\n2. The evaluation metric is unclear and needs more discussion (but this might be because I am not familiar with the finance aspects). It is not clear what is the actual contribution here? Are these metrics not known in the literature? Are there any understudied nuances about the metrics that people need to be careful about?\n\n3. Baselines are not thoroughly investigated. How is sensitivity analysis done for the parameters? How were the hyper-parameters for all the baselines chosen? How robust are the metrics to the chosen constants in the metric formulation (for example the choice of 20% in Line 712)? The proposed work would be much stronger if these were discussed in detail. \n\n4. Visualization is ok, but I am not sure about the audience for that at TMLR. Perhaps that might be more useful at data visualization conferences? (I would defer to the AE for evaluation of interest at TMLR for this aspect of the paper)\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper looks at providing a holistic evaluation for RL methods on financial markets. Several RL baselines are considered and their performance along different aspects (reliability, diversity,  risk control, etc, ) are provided. The core contribution is identification of relevant metrics and providing a variant of the star plot to visualise these results. EMpirical results are provided on data from 4 financial markets.\n\nOverall, there are three key aspects in the paper, (a) what are the metrics to be considered, (b) what are the performances of the baselines on those metrics, and (c) how to report them in a visually appealing manner. The paper tries to tackle everything, but unfortunately, this makes the paper unclear/incomplete on all of those aspects. Isee Lines, It is not clear what the \n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Since this is a domain-specific paper, can the instantiation of the RL setup be better discussed? For instance, what are the actions here? What is the time horizon? What corresponds to an episode? Is there a discounting factor? What is the reward function (even after I finished reading the paper it was not clear to me what is the task for the RL agent here)? Is it purely offline RL or there was a model built using the past data on which online RL was done? What part of the data was used for training and on what part was the performance evaluated?\n\n\n2. Explain the weighting in Eq 7. My understanding is that the weight should be smaller when the Q functions disagree, i.e., the ensemble has a higher standard deviation, and because of this Eqn 7 uses negative $\\bar Q_{std}$. However, I am not sure about Line 145. Since $\\bar Q_{std}$ is always positive, wouldn’t $\\sigma( - \\bar Q_{std})$ be between [0, 0.5], and therefore the range would be [k, 0.5 +k], instead of [0.5, 0.5+k]? Maybe I am misunderstanding something?\n\n\n3. How to select $T$ and $k$ in Eqn 7 during experiments? How to choose $\\beta$? \n\n\n4. Line 156, $L_\\pi$ is not defined. I am assuming it is similar to $L_{actor}$, but I am not sure which Q value is being used here from the ensemble. Could the authors clarify this?\n\n\n5, Can authors convert Lines 165 to 169 into equations for better clarity about what is the exact procedure? I think I get what the procedure is trying to do, but again many words have been used very loosely. For instance, Line 167 says that the UCB is being optimized, but my guess is that only a _heuristic_ for the UCB is being optimized, and not necessarily the actual UCB.\n\n\n\nPRUDEX compass\n\n6. Figure 1 suggests that AlphaMix+ gets the best performance because of the *largest area in the inner level*. My understanding is that the area of the inner level is *not invariant to the ordering of the axes,* i.e., if one plots the axes in a different order, one might get a different area enclosed. If this is true, then authors might need to really rethink such claims.\n\n\n7. Normalization discussed in lines 191-192 needs a lot more justification. Appendix A.2 is largely focused on the procedure done to obtain the plot, but not much is discussed why that procedure is meaningful. Given that this is one of the core contributions of the paper, and that different normalization schemes can potentially result in vastly different plots, the paper would benefit a lot by having a thorough discussion on this. Also, it would be helpful if the authors could elaborate on what is meant by ‘market average’ here.\n\n\n8. I do not have a finance background, so I am not sure how are the values for the axes being computed. Can authors provide or point me to the exact equations for computing these? Further, can authors also comment on other axes that might be of interest to be added in the future? My first thought was that there would be something about assumptions as well, for instance, the RL setup considered in the work has a very strong assumption of stationarity, which is likely violated in financial applications. One could argue that ‘reliability’ could capture that but it looks like right now it is simply aimed at capturing sensitivity to seed.\n\n9. Line 273: I do not think that there is a compelling reason to not compare against non-RL method. Saying that RL methods outperform non-RL based methods, and citing past works, is not sufficient. The good part about the proposed work is the holistic evaluation, which was not done by prior works. And I can imagine all of the proposed axes equally useful for non-RL methods. I understand that incorporating these non-RL methods might not be easy, but given that the core contribution of the paper is evaluating different algorithms and presenting the results, I think the paper would be much stronger with non-RL baselines.\n\nOther plots:\n10. I am not sure if I understood what is the t-sne plot showing. Does each point on the t-sne plot correspond to a single time-step feature? If yes, how is the time-series aspect captured and how should a reader infer that from these plots?\n \n11. Line 340, how is the ‘rank probability’ computed?\n\n12. For the strategy diversity heatmap, the portfolio is for which time? I am guessing this plot should keep changing over time?\n\n13. Figure 8, I do not understand these plots. What is ‘score’? How is it computed? Is a larger score better or lower?\n", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Aq1V9opzgK", "reviewer": "Reviewer_kMKe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths and Weaknesses]\n\n[+] From what I am aware, prior to this work there was no standard for evaluating RL methods for financial markets. So this paper is making a first step toward a framework for standardized reporting of metrics that matter for financial market evaluation algorithms. \n\n[+] I can tell that the authors made significant efforts to think about measures of interest for the evaluation framework. \n\n[+] I really appreciated the authors' efforts to make the visualization materials available through the latex code and the python package. One minor suggestion here is to have colors that are friendly for color-blind folks. \n\n\n[-] The story of the paper is fundamentally about the PRUDEX-compass framework. So I don’t understand why the authors chose to include a baseline algorithm (AlphaMix+) too. It takes away from the rest of the paper’s story. Also related to the paper’s story, I didn’t understand the difference between PRUDEX-compass and PRIDE-star. It looks to me (but I may be wrong) that the PRIDE-star is part of the PRUDEX-compass but I may be wrong. It should have been made much clearer to the reader how the two are connected. \n\n[-] I’m sorry to say this, but I don’t see why the PRUDEX-compass is the best visualization tool for the purposes that the paper describes; I find it quite confusing and unnecessarily complicated at times. Why didn’t the authors choose to report the relevant metrics (and their respective measurements) in just a google doc? Or something similar to the “Datasheets for Datasets” paper? Here is an example of something I find confusing about the current presentation: For the inner circle, why do the points on the different axes need to be connected and form a convex hull? Do these lines connecting the different points represent anything? From what I understand (although I may be wrong and please do correct me if so) the measurements on the axes are just the dots.\n\n[-] In various parts, it is not clear what exactly is one of the paper’s contributions and what is already known in the literature. For example, for the metrics introduced in Sec 4.1 which of these have already appeared in the literature? If all of them previously have appeared then the paper’s contribution is to create an aggregate visualizer for them. How easy/hard is it to estimate each of these metrics in real-life datasets? Another example: is the t-SNE visualization something that the paper introduced or was this already known? (Tangentially related here: I didn’t understand what the point of talking about the t-SNE visualization was at this point. Were the authors trying to say that t-SNE is a data visualization but that PRUDEX-compass also includes algorithmic considerations?)\n\n[Additional Comments]\n\n* The formalism in the paper needs improvement at various points. For example, in Sec 2.2 T is used to denote both the time horizon and the transaction function. At equation (3), there is a different notation used (Q_\\theta) as opposed to the one introduced before (Q^\\pi). In the “Universality” definition (Sec. 4) how is “decent” performance defined formally? \n\n* In Sec. 4.2 I don’t understand what the point of having the paragraph “FinRL Explainability” is since above you have talked about “explainability” again. Is there a difference between general explainability and FinRL explainability? \n\n* In the definition of OHLCV what is function f_k? \n\n[Typos]\n\n* There are many acronyms that are not helpful for the reader to keep in mind. I would suggest that the authors spell out most of the words.\n\n* Pg 2: “also provide the RL community” → “also provides the RL community” \n* Pg 2: “to all QT tasks,” → “to all QT tasks.”\n* Pg 3: “indicate features” → “indicates features”\n* Pg 6: “Crpyto” → “Crypto”\n* Pg 8: “bootrap initialization” → “bootstrap initialization” \n* Pg 20: “here is the “ → this sentence abruptly ends there.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces an evaluation framework for reinforcement learning (RL) methods for financial markets (FinRL), which are increasingly gaining prominence in the research and applied world. The evaluation framework put forth (called “PRUDEX-Compass”) tries to measure evaluation metrics of interest beyond profitability and returns, as standard measures do. Specifically, the evaluation axes of interest in PRUDEX are Profitability, Risk-control, Universality, Diversity, rEliability, and eXplainability. Subsequently, the paper proposes another RL baseline method for FinRL called AlphaMix+ which leverages mixture-of-experts (MoE) and risk-sensitive approaches to make diversified risk-aware investment decisions. The paper evaluates AlphaMix+ and 7 other FinRL approaches on 4 real-world financial market datasets and reports the results on the PRUDEX-compass. The paper also provides a latex (tikzpictures) implementation for PRUDEX and a python package for creating the latex code from data. The paper finally discusses other, orthogonal approaches from the literature on standardizing evaluation of RL methods and the potential impacts of PRUDEX-compass.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) I am not convinced that the “compass” architecture of the visualization is required. I agree that there needs to be a standardized reporting system, I’m just not convinced that the one put forth by the paper is the correct/best one.\n\n2) Make sure the “story” is complete, based on the guidelines and questions I posed above. \n\n3) Strengthen the formalisms of the paper.\n", "broader_impact_concerns": "I think the broader impact of the paper is adequately addressed ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript introduces a tool, dubbed PRUDEX-Compass, for evaluating reinforcement learning in financial markets (FinRL), which is most systematic and comprehensive for FinRL evaluation to the best of AE’s knowledge. The authors also proposed a strong FinRL baseline named AlphaMix+. Per the insightful reviews, the authors did a good rebuttal such that most of the problems and concerns have been well addressed. There remain several minor issues pointed out by the reviewers, for which the AE recommends Minor Revision. The authors are suggested to fix all the issues and emphasize the impact of this research on the FinTech community in the upcoming revision. 
Decision: Accept (poster)