AlphaQCM: Alpha Discovery in Finance with Distributional Reinforcement Learning
Authors: Zhoufan Zhu
Ke Zhu
Venue: ICML 2025 poster (2025)
Keywords: Computational Finance
Formulaic Alpha
Distributional Reinforcement Learning
Quantiled Conditional Moments
Stock Trend Forecasting
Primary area: applications->everything_else

Abstract: For researchers and practitioners in finance, finding synergistic formulaic alphas is very important but challenging. In this paper, we reconsider the discovery of synergistic formulaic alphas from the viewpoint of sequential decision-making, and conceptualize the entire alpha discovery process as a non-stationary and reward-sparse Markov decision process. To overcome the challenges of non-stationarity and reward-sparsity, we propose the AlphaQCM method, a novel distributional reinforcement learning method designed to search for synergistic formulaic alphas efficiently. The AlphaQCM method first learns the Q function and quantiles via a Q network and a quantile network, respectively. Then, the AlphaQCM method applies the quantiled conditional moment method to learn unbiased variance from the potentially biased quantiles. Guided by the learned Q function and variance, the AlphaQCM method navigates the non-stationarity and reward-sparsity to explore the vast search space of formulaic alphas with high efficacy. Empirical applications to real-world datasets demonstrate that our AlphaQCM method significantly outperforms its competitors, particularly when dealing with large datasets comprising numerous stocks.

AlphaQCM: Alpha Discovery in Finance with
Distributional Reinforcement Learning
Zhoufan Zhu" 2 Ke Zhu
Abstract
gadeesh, 1990). Here, each alpha a function that trans- forms noisy historical stock data into signals for predicting future stock returns. However, recent studies examining the influence of investor behavior and psychology on price dy- namics have uncovered the existence of subtle and intricate alphas that are difficult to formalize using standard financial methods (Barberis, 2018).
For researchers and practitioners in finance, find- ing synergistic formulaic alphas is very important but challenging. In this paper, we reconsider the discovery of synergistic formulaic alphas from the viewpoint of sequential decision-making and conceptualize the entire alpha discovery process as a non-stationary and reward-sparse -spare Markov de- cision process. To overcome the challenges of non-stationarity and reward-sparsity, we propose the AlphaQCM method, novel distributional re- inforcement learning method designed to search for synergistic formulaic alphas efficiently. The AIphaQCM method first learns the Q function and quantiles via Q network and a quantile network, respectively. Then, the AlphaQCM method ap- plies the quantiled conditional moment method to learn unbiased variance from the potentially biased quantiles. Guided by the learned Q func- tion and variance, the AlphaQCM method navi- gates the -totssationaritt and seward-sparsite to explore the vast search space of formulaic alphas with high efficacy. Empirical applications to real- world datasets demonstrate that our AlphaQCM method significantly outperforms its competitors, particularly when dealing with large datasets com- prising numerous stocks.
Most existing Al approaches surpass traditional methods in this field by designing alphas through the use of sophis- ticated machine learning (ML) models in end-to-end manner (Feng et al., 2019; Ding et al., 2020; Koa et al., 2023). Intuitively, the well-traines ML models are effective alphas, since they can transform stock data into predictive signals. However, they are inherently complex and lack sim- ple mathematical representations, leading to the so-called non-formulaic alphas. Yet, the non-formulaic alphas have trust issues due their black-box nature, so they not widely adopted in the industry.
For dealing with the above trust issue, emerging literature focuses on how to automatically discover a set of synergis- tic formulaic alphas. The formulaic nature of these alphas implies that they can be expressed by a simple formula, usu- ally making them compact, interpretable, and generalizable; meanwhile, their synergistic nature allows them be com- bined into a meta-alpha via some interpretable models (e.g.. linear models). Finding these formulaic alphas typically is based on the genetic programming (GP) method (Lin et al., 2019a;b; Zhang et al., 2020; Cui et al., 2021). However, the GP method has a vast search space, which scales exponen- tially with the number of input features and operators.
1. Introduction
Over the past decades, extensive research has investigated the predictive power of historical stock information for forecasting future returns, resulting in the development of several well-known alphas, such as long-term momen- tum (Jegadeesh& & Titman, 1993) and short-term reversal (Je-
To overcome the challenge of vast search space in the GP method, the AlphaGin method (Yu et al., 2023) reformu- lates the alpha discovery problem into the task of finding an optimal policy for a specialized Markov decision pro- cess (MDP), and then achieves this task via a reinforcement learning (RL.) algorithm. Although the AlphaGen method shows state-of-the-at performance, has two substantial theory-practice gaps, resulting in the inefficient and unstable alpha discovery procedure in practice. The first gap is an un- addressed non-stationary issue (Lecarpentier & Rachelson, 2019), where the reward function in the MDP of interest changes dynamically across decision episodes. This issue
'Wang Yanan Institute for Studies in Economics, Xiamen Uni- versity, Xiamen, China DDepartment of Finance, Xiamen Univer- sity, Xiamen, China Department of Statistics & Actuarial Science, School Computing Data Science, University of Hong Kong, Hong Kong. Conrespondence to: Ke Zhu <mazhuke hk..
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
arises from the goal of collecting a set of synergistic formu- laic alphas rather than a single formulaic alpha. The second gap is how to accommodate the reward-sparse nature of the considered MDP, as most discovered alphas are weak and leads to zero rewards.
the AutoML-Zero framework (Real al., 2020), with the IC employed the fitness measure. However, the recent lit- erature highlights the suboptimal performance of GP-based methods in scenarios involving large populations (Petersen et al., 2021). which are essential for alpha discovery due to the complexity of considered financial market.
This paper contributes to the literature by introducing a novel method, AlphaQCM, which addresses these theory- practice gaps and presents new RL solution to the alpha dis- covery problem. Specifically, the AlphaQCM method first leverages the IQN algorithm (Dabney et al., 2018a), to learn the quantiles of cumulative discounted rewards, whereas it studies the mean of cumulative discounted rewards via the DQN algorithm (Mnih et al., 2015). Then, based on the learned quantiles, the AlPhaQCM method adopts the quantiled conditional moments (QCM) method (Zhang & Zhu, 2023) to estimate variance of cumulative discounted rewards, which serves as natural exploration bonus for the mining agent's s action selection to relieve the issue of reward- sparsity. Remarkably, the estimated variance from the QCM method remains unbiased even if the estimated quantiles are biased due to non-stationarity. Hence, by employing the AIphaQCM method, we can alleviate the negative impacts of non-stationarity and reward-sparsity, thereby achieving significantly better empirical performance in discovering formulaic alphas. Our work clearly generalizes and ex- tends the AlphaGen method and can be applied to other non non-stationary and/or -wward-ppars environments.
Conversely, as a RL-bbasdd method, the AlphaGen method (Yu et al., 2023) firstly conceptualizes the alpha discovery process as MDP and employs RL algorithm, specifically the proximal policy optimization (PPO) algo- rithm (Schulman et al., 2017), to discover a set of synergistic alphas with high returns. Although the AlphaGen method has significantly outperformed the previous cp-bbaed meth- ods, has three notable shortcomings. First, due to the reward-sparse nature of the alpha discovery MDP, the Alpha- Gen method struggles to explore the search space efficiently. Second, the AlphaGen method suffers from the issues re- lated to sample efficiency and convergence performance, as the alpha discovery MDP is clearly non-stationary. Third, the AlphaGen method completely ignores the intricate dis- tributional information within the observed expressions and subsequent alpha construction, resulting in an inefficient and unstable alpha discovery process.
2.2. Distributional Reinforcement Learning
Following the standard RL setting, the agent-environment in- teractions are modeled as an MDP, AA A,,,,, P, R), where X' and A are finite sets of states and actions. respec- tively. P X x A + (X)) is the transition kernel, 7 E [0, 1) is the discount factor, and the reward function R : X x A + (R) defines the distribution of rewards, with (A) being a random variable with support A.
We apply our AlphaQCM method to three real-world market datasets to assess its empirical performance, together with baseline methods such as the AlphaGen method and GP- based methods. Extensive experimental results demonstrate that the AlphaQCM method consistently achieves the best performance, with Information Coefficient (IC) values of 8.49%, 9.55%, and 9.16% across the three datasets. Its su- perior performance is particularly evident when the dataset originates from a complex financial system. Finally, we con- duct several ablation studies to investigate the contribution of each component in the AlphaQCM method.
At the 1-th agent-environment interaction, the agent observes state X, 3 (X),, selects action A, (AA), and sub- sequently receives feedback from the environment in the form of the next state X1+1 ~ P (- X,, At) and reward RG ~ R (Xe, At). For a given policy क : X - (AA) and A ~ T(- the discounted cumulative reward can be represented by a random variable 2 (ax, a):
2. Background and Related Work
2.1. Formulaic Alpha
2" (x, a) Σ
M(((X, Ar) Xo x,, Ao = a)
The formulaic alpha has an extensive search space of poten- tial expressions due to the enormous operators and features that are available for selection. Generally speaking, most existing methods for discovering formulaic alphas can be categorized into two classes: GP-based methods and RL- based methods. In the past decade, the GP-based methods have predominantly served as the mainstream to generate formulaic alphas (Lin et al., 2019a;b; Zhang et al., 2020; Cui et al., 2021). For example, the AlphaEvolve method (Cui et al., 2021) evolves new alphas from existing ones using
(R, Xu = zz Au = a),
t=
where (s, E X A is state-action pair'.
The ultimate goal of RL algorithms is to ascertain an op- timal policy ,,, which maximizes the expectation of dis-
In this paper, random variables are represented by uppercase letters (e.g., X₂) and observations are represented by lowercase letters (e.g.. x.).
2

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
counted cumulative rewards, also known as the Q function Q (2r a) (x, A common way to obtain x' is to find the unique fixed point Q' = Q of the Bellman optimality operator T (Bellman, 1966), satisfying
Although the empirical evidence suggests that DRL algo- rithms have the good performance with the desirable robust- ness to variations in hyperparameters, several challenges remain. The most critical challenge is the validity of the estimated quantiles, raising a concern about the consistency of Z (a, a) and its moments. Specifically, the traditional DRL algorithms show the unsatisfactory performance in -un-staiionayy and reward-sparse MDPs, which are com- mon in practical scenarios (e.g., the alpha discovery MDP discussed in this paper).
Q" (2x a) = =TQ*(r,))
= = [R, + qQ" (X(+1, Xx, = 2, A a],
where a² argina, Q" (X++1, In practice, Q" is typically approximated by a parametric functtin, such as the deep Q network (Mnih et al., 2015). However, the ma- jority of RL algorithms only focus on the scalar expectation Q" a). thereby overlooking the valuable distributional in- formation arising from the potential randomness of optimal policy and the stochasticity of considered environment.
3. Methodology
In this paper, we consider a scenario involving N distinct stocks with their prices and volume information. Our goal is find an optimal alpha pool F. (i.e., a set of synergis- tic formulaic alphas), which is effective for constructing predictive linear meta-alpha for future stock returns.
To address this issue, the distributional RL (DRL) algo- rithms concentrate on directly learning the distribution of Z" (2) Let Z = denote the discounted cumulative rewards with optimal policy x². To find n², the distribu- tional Bellman optimality operator TD (Bellemare et al., 2017) is defined as:
To be more specific, we assume that the alpha pool F comprises at most P different formulaic alphas, labeled as
For p = I...., maps market data into alpha values, defined as follows:
., P. each fp is function that
(² a) TT
L RR + 7Z" (X++1, a") X2 = =x, xx = aa
(I)
ap.. fp (H, E R..
where H,-1 includes historical information for N stocks up to time s 1, and px,, the vector of cross-sectional alpha values. Then, the linear meta- meta-alpha alpha & is defined as:
where a" = argmax,' E.A 1l (X1+1, d')] and 2 denotes the equality in probability laws.
In practice, it common to parameterize 7 via the quantile representation (z. a), which is mixture of K Dirac distributions. Specifically,
p
ā pml
ERR E
K-1 Σ (7x11-Th (T) km
where 3, E E for po L...., the linear coefficient (or weight) for each alpha, and ā, are expected to effectively capture subsequent stock returns y. E BR-,, thereby exhibit- ing high IC value?. To achieve this goal, we conceptualize the alpha discovery process as a non-stationary and reward- sparse MDP and propose the AlphaQCM method, a novel and efficient DRL algorithm, to find the most effective F.
Zo, (2, a)
where 5, is Dirac distribution centered at 2 E I, T = (TI)...TKK)) is a vector of quantile levels satisfy- ing 0- < TI < < TK-1 < TK = 1, and = (o...... . ,0K] is a set of functions. Moreover, we denote a) = (z, a)......... (x(x, a))' E RK for notation simplicity, where 02 (2., a) is the T -th quantile of (x, with o (Tx-1 + /22 and To- 0.
3.1. Representation of Formulaic Alpha
Before going further, we begin by introducing how to re- formulate the alpha discovery problem into a sequential decision- making task. Recall that each formulaic alpha has a mathematical expression comprising operators and features. These operators are broadly categorized into time- series operators and cross-sectional operators (Kakushadze, 2016; Lin et al., 2019abb). The time-series operators neces- sitate data spanning multiple days, such as TsRank( Close, d). which calculates the sequential ranking of the most re- cent d closing prices for each stock. In contrast, the cross- sectional operators manipulate single-day data, exemplified
Needless to say, the formulation of Ze. + (a, a) depends on T and 0. As a pioneer work, the QRDQN algorithm (Dabney et al., 2018b) adopts a fixed т, and applies a multi-head deep neural network to learn corresponding 0g (a. a). Fur- thermore, the IQN algorithm (Dabney et al., 2018a) con- siders a random т, while the FQF algorithm (Yang al., 2019) incorporates an additional neural network to learn the optimal T for each a).
Accompanied by the quantile representation, the exist- ing DRL algorithms first learn 0(x, a), and then estimate Q" (x, a) by directly taking the expectation of Z (2, a).
"See Appendix A for more explanations.
3

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
by Rank(Low), which computes the cross-sectional rank- ing of low prices among N stocks. By combining these two types of operators, the formulaic alphas exhibit high nonlinearity, but are still interpretable for humans.
72
X--1
BEG
Law
Environment
Te-1
BEG
Stook Data
Agent
After specifying the operators used in the mathematical expression, we further employ the Reverse Polish Notation (RPN) method to represent the expression (i.e., the form of fp). Figure gives a specific illustrating example on the RPN method. In this figure, the formulaic alpha "Alpha#44" is encoded into a token sequence, where BEG and SEP tokens indicate the beginning and ending of the expression, respectively, and each feature or operator denoted as a token. The details of the available features and operators are listed in Table B.5.
Evaluste
Alpha Pool 2
92
Low
RPN
BEG
Low
Figure 2. Agent-environment interaction diagram.
is allowed to be taken for a specific x since not all token sequence are guaranteed to be the RPNs of valid formulaic alphas.
(+)
Alpha#4: (-1 TsRank (Ranank (low), 9)
Transition kernel. In this MDP, the transition kernel is deterministic. Given 2y and or, the environment feedbacks the next state Æt-1 by appending a, to the end of z₁, unless Or is the SEP token or I2 has reached its maximum length. In such cases, this episode of aggntt enviirnnvvee interaction terminates, and the environment presents 2g to the agent to initiate new episode.
Talt
(0)
BEG
Low
Rank
TsRank
Mul
SEP
Reward. The key component of the alpha discovery process is the design of reward. Intuitively, the reward is expected to quantify the contribution of the newly discovered formu- laic alpha on the current alpha pool F, which consists of at most P different formulaic alphas collected in previous episodes. As in Yu al. (2023), we set Ft 0 for any incomplete token sequence, the formulaic alpha is only partially formed. Once the token sequence is completed, is parsed into a formulaic alpha. If the parsed alpha is invalid, Ti -1; otherwise, the formulaic alpha is evalu- ated by the environment through the following steps: (1) The new formulaic alpha is added to alpha pool to create an extended alpha pool; (2) A linear model fitted based on the extended alpha pool select up to P principal formulaic alphas with the most significant contribution, and the alpha pool is updated accordingly; (3) The meta-alpha is obtained based on the updated alpha pool and the fitted linear model;
Figure 1. (a) Formulaic expression of Alpha#44 factor in
Kakushadze (2016). (b) Its expression tree. (c) Its RPN repre- sentation.
Intuitively, the RPN representation can be viewed as a trajec- tory, documenting the sequential actions taken by an agent, when selecting which token to place at each position of for- mula. As result, the task of discovering formulaic alphas can be regarded sequential decision-making problem by designing particular MDP.
3.2. Specification of MDP
After introducing the RPN representation, we specify the alpha discovery MDP, as illustrated in Figure 2. Below, we show each component in this MDP.
(4) The reward r, is calculated by the increase in IC of the meta-alpha based on the updated alpha pool compared to that based on previous alpha pool. See Algorithm for more details.
State and action. In accordance with the GP-based meth- ods, the agent manipulates token sequences, prompting us to consider a token-based state set X. Specifically, each state observation 222 E X' corresponds to token sequence representing the currently generated expression, with initial state X, = zo being the BEG token. To maintain the in- terpretability of discovered alphas, we restrict any state to have fewer than 20 tokens. Aligned with the design of state, each action a A is token. However, only a subset of A
Discount factor. Although long alphas tend to lack gen- eralizability and interpretability, they often are more so- phisticated and predictive for stock trends. Considering
For example, time delta token must be followed by time- series operator. See Appendix of Yu al. (2023) for more details on this aspect.
4

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
Algorithm 1 Pseduo code for calculating rt.
3.3. AlphaQCM
Input: training samples {(H,-1, }. new alpha fnew, and alpha set F = {f1. .p.....f... Output: updated alpha set F and reward i'
To solve the issues of non-stationarity and reward-sparsity in alpha discovery MDP, our AlphaQCM method employs the QCM method (Zhang & Zhu, 2023) learn an unbiased variance of rewards. This variance is further used as bonus to guide the agent in exploring the environment, thereby improving the agent's 's training efficiency. Below, we outline the core modules of our AlphaQCM method.
Calculate alpha values and normalize them
f1++1 fnew
ap.s f- fp(H,-1) for
P +1
ap p.s f- By., atd(a
-Mannaaa for p = 1..... P +
# Fit a linear model based on the extended alpha pool
A. + [or,,))...,, 0pp++1,. -X
3.3.1. QCM
3 argming E. lyy, A E ER
We begin with introducing how to learn variance from quan- tiles via the QCM method, which can be extended to most existing DRL frameworks. Recall that the goal of DRL algorithm is to model the discounted cumulative reward Z (zz a) with its quantiles (x(r, a) for k = =..... KK, as in (2). According to the Cornish-Fishhe expansion (Cornish & Fisher, 1938), the quantiles of ("(x,,) a) are linked its moments as follows:
# Obtain the updated alpha pool and meta-alpha
if P + + then
F - (S.....,SPP... {1-+1}
ā. - A.B E REN
else
1 - argmiup 13pl. where 3 = =.,,,..,@p++.)))
3pr+i)
F - [f1 f
+1
ā. + A-A 3p. E N
end if
O2 a) (2, F,0(a, ww (n (3)
# Calculate IC based on F and reward rz
IC(F*) < Mcan (Corr(á ys))
where D2 = (dx: ox - 1. 1 - 3ox) and
rt 1.+1C4 IC(F)) - IC(FF)
0(z,a) (Vh(z,a) a), Vh(x,a)s(z,,) / /h(x, a) |K[x a) - 3
Return F and rt
6
24
this trade-off, we restrict the max length of episode in state setting and hence set y 1.
Specifically, dy is the -h quantile of standard Gaussian distribution, Q" (x, a), h(x, a). s(z, a), and k(z, a) denote the mean, variance, skewness, and kurtosis of 2" 2x a), respectively, and wx(x,, a) represents the expansion error.
Non-stationariit. Since the reward function varies across episodes (with different formulaic alphas collected by the al- pha pool), the alpha discovery MDP is clearly non-station -tn-stationary. Specifically, when the agent discovers a new formulaic al- pha that proves to be effective (as indicated by high re- ward from the environment), the environment incorporates this formulaic alpha into the alpha pool. In the subsequent episode, the similar formulaic alphas to the recently col- lected one tend to earn little reward, since the alpha pool has been updated to capture the predictive information re- lated to this type of alpha. As result, once the alpha pool is updated, the reward function of the MDP changes, cessitating numerous agent-environment interactions and extensive training to re-learn it.
In practice, the true value of (2, a) in (3) is unknown. Al- though the estimated quantile 0e a) is expected to oscil- late around 0g (z,, a), it is always biased due to the presence of non-stationarity. Therefore, substituting Og (3r, a) with 0g (I, a) in (3) yields the following expression:
0z a) = [(x, a) + Q" (xx a) + Φ1,((z,) a) + (z)
(4) where ((x, a) and Ex a) represent the deterministic bias and stochastic residual, arising from the presence of both saasstttionartt and expansion error, respectively.
Reward-sparsity. In addition to the non-stationarity, the alpha discovery MDP exhibits reward-sparsity from two perspectives. First, the reward can only be non-zero when the episode ends (i.e., when new formulaic alpha is gen- erated); otherwise, it must be zero. Second, due the low signal-to-noise ratio of market datasets, most discovered alphas are meaningless and not beneficial to the alpha pool, leading to zero rewards. Consequently, there are numerous zero rewards in the transitions, resulting in an inefficient and unstable training process for the agent.
Clearly, (4) can be interpreted as linear regression model, where (I, a) and & serve the observable response and regressor, respectively, while ((x, + Q" (ar, a). 8(x, a) and e ( a) correspond to the intercept, regression coeffi- cients, and residual, respectively.
With K samples {(0 (z, a). $x)} in hand, we can obtain the ordinary least square (OLS) estimators h(,, a). 8(2, a), and k(x, a) by solving the above linear regression model
5

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
(4). Under some mild conditions", the consistency of these moment estimators is guaranteed by:
encodes the token sequence 2i into vector representation, and a fully-connected head FC(() produces the quantiles:
Proposition 3.1. Suppose that Assumptions C.I and C.2 hold. Then, h(z, a) _ h(æ,a), a), ((z,a) _s s(x, a) and a(x,a) _, k(r, a) as K -> oc, where _> denotes con- vergence in probability.
+ (2,) FC(((x)) o v(T)) E RRAAIK
where (x²,) includes (g( a) for a A and A
1,... K. With (xx;) in hand, h(z,, a) can be computed via the QCM method for a E A. Then, the agent selects an exploratory action de to enhance training efficiency:
Although the MDP is non-stationary, Mar,a) remains unbiased, whereas there is no such guarantee for the vanilla quantile-based variance estimator", even in station- ary MDPs (Bellemare et al., 2023). As mentioned in Mavrin et al. (2019), M(r, a) captures both parametric and intrinsic uncertainties, which can be attributed to non-stationarity and reward-sparsity, respectively. Using f(zz a) as an ex- ploration bonus, our agent tends to explore the most uncer- tain states, which also lead to the most informative expe- riences for overcoming the challenges of non-stationarity and reward-sparsity. By training on these informative expe- riences, the agent mitigates the negative impacts of reward sparsity and non-stationarity as much as possible and effi- ciently learns from the dynamic environment.
a: = arrgax aEA
(5)
where U(z, a) is computed by the Q network of the DQN algorithm", and À is tuning parameter to control the degree of risk fisk-rreference. See Figure 3 for a visual illustration.
In this paper, the Q network employs separate LSTM feature extractor and fully-connected head to transform the Ty into Q values:
( = FC(((a)) E IR A!,
Lastly, it should be noted that we cannot estimate Q' a) using the QCM method, since the estimated intercept cor- responds to ((x, a) + (*, a) in (4), making (Iz, a) and Q"(x,a a) are unidentifiable. Therefore, we use separate RL algorithm to learn Q" (zz, a). In such a non-stationary MDP, the traditional DRI, algorithms yield biased Q esti- mates, as they estimate (z, a) by directly taking expec- tation of Zg.- (x. a). This biased Q issue in non-stationary MDPs is somewhat inevitable, but using the QCM method can alleviate the negative impacts from non-stationarity. The underlying reason is that using QCM method enhances training efficiency, regardless of whether the bias caused by non-stationarity exists. By improving training efficiency, the agent requires fewer agent-environment interactions and less training time to re-approximate Q* Q*(x,aa) (Ir.a..
where (xx)) includes Q(2,, a) for E A.
Morcover, the quantile network is trained using the quantile temporal difference error (Dabney et al., 2018a). while the Q network is trained with the squared temporal difference errors (Mnih et al., 2015). To save space, more technical details and hyperparameters used in our AlphaQCM method are specified in Appendix E, while some unmentioned tech- nical details are consistent with the backbones.
4. Experiments
Since Yu et al. (2023) is the most closely related work, our experiments are also conducted on Chinese A-share stock market datasets to capture the 20-day future stock returns. To evaluate the impact of the complexity of the considered financial system on performance, we consider three different stock pools: (1) the largest 300 stocks (CSI 300), (2) the largest 500 stocks (CSI 500), and (3) all stocks (Market) listed on the Shanghai and Shenzhen Stock Exchanges. As one might expect, the more stocks involved in the dataset, the more challenging it becomes to discover synergistic formulaic alphas, as the system becomes more complex and chaotic. Moreover, each dataset is split chronologically into training set (2010/01/01 to 2019/12/31), validation set (2020/01/01 to 2020/12/31), and test set (2021/01/01 to 2022/12/31).
3.3.2. DRL BACKBONE
After showing how to use the QCM method to obtain vari- ance from quantiles, we elaborate on the backbone used to estimate the quantiles and Q function. In this paper, we adopt the IQN algorithm (Dabney et al., 2018a) to learn the quantiles, and apply the DQN algorithm (Mnih et al., 2015) to learn the Q function.
Specifically, when the agent observes from the environ- ment, T is sampled from the uniform distribution over (0, 1), and it is subsequently fed into the quantile network together with xy In this network, a r-embedding network v(-) trans- forms T into embeddings, a LSTM feature extractor (b)))
We consider the following four kinds of baseline methods for comparison:
See Appendix C for more details. 'The vanilla quantile-based variance estimator is defined in Appendix D.
'Recall that, for a certain J. some actions are invalid, as men- tioned in Section 3.2. Hence, we set the Q(2, a) oc for these invalid actions to mask them.
6

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
Quantile Network
(xx)
R(x+)
Quantile
QCM
Variance
((x,)) a R(x))
de
Xz
Action
Action
Score
BEG
Rank
Action
Mask Invalid Actions
Action
Network
(xx)
Figure 3. An illustration of action selection in our AlphaQCM framework.
(I) Alpha101 comaan-desiged formulaic alphas): Fix the alpha pool as the formulaic alphas provided by Kakushadze (2016), and fit linear model to form a mega-alpha.
deviations of IC values across eight different methods in CSI 300, CSI 500, and Market datasets. From this table, we can draw the following conclusions:
(I) Our AlphaQCM method with the highest value out- performs all competitors, regardless of the stock pool con- sidered. Moreover, the AlphaGen method, which is the most closely related baseline method, ranks second. The advantage of the AlphaQCM method over the AlphaGen method becomes more significant as the number of stocks in the dataset increases. This superior performance may attributed to the fact that the AlphaGen method totally ignores the non-stationary and reward- -parrse issues, while these issues become more pronounced as the the concerned system becomes more complex and chaotic.
(2) MLP. XGBoost, LightGBM (ML-based non- formulaic alphas): Use the MIP model, XGBoost model, or LightGBM model to form a mega-alpha.
(3) GP w/o filter, GP w/ filter (GP-based formulaic al- phas): Use the GP method to generate expressions and apply top-P performing alphas without or with mutual IC filter to form mega-alpha.
(4) PPO w/ filter, AlphaGen (RL-based formulaic al- phas): Use the PPO with mutual IC filter Alpha- Gen method find the optimal alpha pool and then form linear mega-alpha.
(2) For the GP method, incorporating the mutual IC filter im- proves its performance in the CSI 300 and CSI 500 datasets but results in worse performance in the Market dataset. This finding highlights the limitation of using the mutual IC filter to find synergistic formulaic alphas when large number of stocks are involved, as commonly observed in modern portfolio selection.
To account for the effect of stochasticity in the training pro- cess, evaluate each indeterministic experimental combi- nation with 10 different random seeds. More details about these baseline methods are provided in Appendix F.
Following the existing literature (Yu et al., 2023; Cui et al., 2021; Lin et al., 2019a;b), we choose the IC as the most important metric to evaluate the out-of-sample performance. See Appendix for the definition of IC.
(3) The machine learning methods slightly underperform the Alpha101 method in the CSI 300 and CSI 500 datasets, while they outperform the Alpha101 method in the Mar- ket dataset, with the exception of the MLP method. This observation suggests that when facing big data, the formu- laic alphas discovered by human experts may lose their advantage the non-formulaic alphas. However, the RL- based methods surpass both human experts and machine learning methods, with the discovered alphas remaining interpretable.
The source code and related source of this work are available at https: / // thub. com /ww.houFaaaaa////a AlphaQCM for reproducibility.
4.1. Impact of Methods
We first assess how the alpha-generating methods affect the out-of-sample performance of the formed meta-alphas. For a fair comparison, we regard (alpha pool size) as a hyperparameter and choose it based on performance on the validation set. Table reports the means and standard
4.2. Impact of QCM Method and DRL Backbones
One of our key contributions is using the QCM method to overcome the challenges of non-stationarity and reward-

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
Table 1. Out-of-sample IC values across different methods.
Table 2. Out-of-sample IC values across different action selection criteria and DRL backbones.
CSI 300
CSI
500
Market
CSI 300
CSI
500
Market
Method
Mean
Std
Mean
Std
Mean
Std
Variance
Mean
Std
Mean
Std
Mean
Std
Alpha101
3.44%
4.38%
3.15%
Panel A: QRDQN as backbone
MLP
1.99%
0.24%
2.72%
0.65%
2.81%
0.72%
XGBoost
3.19%
0.81%
4.31%
0.96%
4.07%
1.22%
No Vanilla QCM
6.96%
1.64%
8.54%
1.56%
7.06%
1.92%
LightGBM
2.93%
0.76%
4.16%
0.81%
4.28%
0.93%
6.14%
1.23%
8.80%
1.34%
7.60%
1.17%
7.59%
0.81%
9.08%
1.07%
9.12%
1.74%
GP w/o filter
2.01%
1.46%
1.79%
1.62%
1.32%
2.01%
GP w/ filter
3.71%
2.01%
4.52%
1.93%
0.84%
2.27%
Panel B: IQN backbone
PPO w/ filter
1.14%
1.71%
0.98%
1.36%
2.15%
1.86%
No Vanilla
7.17%
2.40%
8.58%
1.47%
7.04%
1.82%
AlphaGen
8.13%
0.94%
8.08%
1.23%
6.04%
1.78%
6.16%
1.73%
8.75%
1.03%
8.42%
1.59%
QCM
8.49%
1.03%
9.55%
1.16%
9.16%
1.61
AlphaQCM 8.49%
1.03%
9.55%
1.16%
9.16%
1.61 %
sparsity. To see it, we consider two competitors below:
4.3. Impact of Domain Knowledge
Until now, the AlphaQCM method has discovered formu- laic alphas in completely data-driven manner, neglecting the valuable insights offered by economic experts in the field of alpha discovery. One potential approach to incorpo- rate the domain knowledge of experts into our AlphaQCM algorithm is to encode the formulaic alphas proposed in Kakushadze (2016) into token sequences and initialize the replay memory with these corresponding trajectories'. In other words, we restrict the agent to first learn from the formulaic alphas discovered by human experts and then find new alphas in a data-driven way.
1. No variance: Fix À= in (5) to remove the impact of variance in action selection, meaning that the QCM method has no effect in the training process.
2. Vanilla variance: Replace M(z, a) in (5) with the vanilla quantile-based variance estimator defined in Appendix D.
Moreover, since our AlphaQCM method relies on the DRL backbone employed to learn quantiles, a natural question arises: which type of DRL backbone is more effective for alpha mining? To answer it, we alter the AlphaQCM method by choosing the QRDQN algorithm (Dabney et al., 2018b) as the backbone, which serves as another benchmark.
To check whether the domain knowledge enhances the ef- ficiency of the AlphaQCM method, we report the perfor- mance of AlphaQCM method with and without domain knowledge in Table 3. From this table, we observe that there is some initial gain when the AlphaQCM method leverages domain knowledge, as the AlphaQCM method with domain knowledge outperforms the one without do- main knowledge in Panels A and B. However, with more engerneenvrommment interactions and training as in Panels and D, the AlphaQCM method in completely data-driven manner achieves higher IC values. This is perhaps because the agent with domain knowledge is prone to fall into local optima by imitating the experts. Hence, to achieve bet- ter final performance, we suggest applying the AlphaQCM method without domain knowledge.
Table 2 reports the results of the conducted ablation study. From this table, we observe that regardless of the backbone employed, using the QCM variance always earns the highest IC value, whereas the action selection based on the vanilla variance results in better performance than that based on no variance only in CSI 500 and Market datasets. Addition- ally, from the viewpoint of convergence performance, using variance to encourage exploration consistently brings lower standard deviations of IC values. These findings imply that using variance for exploration enhances training efficiency and convergence performance in the alpha discovery MDP, and at the same time, a better quality of variance estimation leads to a better empirical performance.
4.4. Impact of Stock Market
We also find that the IQN algorithm outperforms the QRDQN algorithm in most cases, except for no variance in Market dataset and vanilla variance in CSI 500 dataset. However, the differences in performance between the IQN and QRDQN backbones are marginal in these two excep- tions. These empirical results demonstrate the superior per- formance of using the IQN algorithm as the backbone for the AlphaQCM method compared to the QRDQN algorithm.
To show the generalizability of our AlphaQCM method, we conduct additional experiments using the U.S. stock market dataset, specifically the largest 500 stocks (S&P 500). All
Forr example, as in Figure 1, we encode the mathematical expression of "Alpha#4" into its RPN representation, which is a token sequence basically. Then, the generating process of this token sequence regarded a trajectory for agent-environment interactions, which are used to initiate the replay memory.
8

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
variance derived from potentially biased quantiles. This approach enables the efficient alpha discovery in the non- stationary and reward-sparse MDP. To implement the Al- phaQCM method, we employ the IQN algorithm as the backbone to obtain quantiles, while approximating the Q function using the DQN algorithm. Through extensive ex- periments on three real-world datasets, we demonstrate the superior advantage of our AlphaQCM method over the pre- vious state-of-the-an alpha discovery approaches. The abla- tion studies further highlight the contribution of the QCM method, the robustness of DRL backbone, parameter size, and other settings..
Table Out-of-sample IC values of the AlphaQCM methed with or without domain knowledge.
CSI
300
CSI 500
Market
Domain
Mean
Std
Mean
Std
Mean
Std
Panel A: After 10% Training
w/
4.93%
0.71%
5.76%
0.68%
6.02%
0.92%
w/o
4.27%
1.75%
5.68%
1.51%
5.87%
1.34%
Panel B: After 20% Training
6.32%
1.29%
7.01%
1.77%
6.85%
1.44%
w/o
5.54%
0.78%
6.43%
1.38%
6.43%
2.83%
Panel C: After 50% Training
Overall, the AlphaQCM method serves as a powerful tool for discovering synergistic formulaic alphas, with its su- perior capability allowing for -on-stationarity and reward- sparsity in the alpha discovery process.
w/
6.41%
1.47%
7.15%
1.22%
7.33%
1.56%
w/o
6.82%
1.35%
7.57%
2.12%
7.48%
1.84%
Pancl D: After 100% Training
w/
8.17%
1.17%
8.96% 9.55%
1.51%
8.60%
1.23%
w/o
8.49%
1.03% %
1.16% %
9.16% %
1.61% %
A Acknowledgements
The authors are very grateful to the reviewers, the Area Chairs, and the Program Committee members for their con- structive suggestions and comments, leading to a significant improvement of the paper. Zhoufan Zhu's work is supported by the Fundamental Research Funds for the Central Univer- sities (no. 20720251053) and Ke Zhu's 's work is supported by the GRF, RGC of Hong Kong (nos. 17302424, 17304723, 17312622, and 17304421).
experimental settings, except for the stock selection, remain consistent with those in Table 1.
The results, shown in Table 4, demonstrate that AlphaQCM outperforms all competitors on both the CSI 500 and S&P 500 datasets, indicating its robustness across different mar- ket conditions. However, most methods exhibit a decline in IC performance on the S&P 500 dataset compared to the CSI 500. This may be due the greater efficiency of the U.S. stock market, which is more sensitive to breaking news, making it more challenging for models to capture predictable stock trends from formulaic alphas.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Table 4. Out-of-sample IC values on CSI 500 and S&P 500 datascts.
CSI
500
S&P 500
References
Method
Mean
Std
Mean
Std
Alpha101
4.38%
3.12%
Barberis, N. Handbook of Behavioral Economics: Applica- tions and Foundations. Elsevier, 2018
MLP
2.72%
0.65%
2.61%
0.49%
XGBoost
4.31%
0.96%
3.08%
0.67%
Bellemare, M. G., Dabney, W., and Munos, R. A distribu- tional perspective on reinforcement learning. Proceed- ings of the 34th International Conference on Machine Learning, pp. 449-458. 2017.
LightGBM
4.16%
0.81%
3.29%
0.56%
GP w/o filter
1.79%
1.62%
1.88%
1.29%
GP w/ filter
4.52%
1.93%
4.27%
1.39%
PPO w/ filter
0.98%
1.36%
2.03%
1.88%
AlphaGen 8.08%
1.23%
7.48% 5
0.77%
Bellemare, M. G., Dabney, W., and Rowland, M. Distribu- tional Reinforcement Learning. MIT Press, 2023.
AlphaQCM
9.55%
1.16%
8.46%
0.89%
Bellman, R. Dynamic programming. Science, 153:34-37, 1966.
5. Conclusion
Cornish, E. A. and Fisher, R. A. Moments and cumulants in the specification of distributions. Revue de I' I'Institut international de Statistique, pp. 307-320, 1938.
This paper proposes a novel DRL method, AlphaQCM, for alpha discovery in the realm of big market data. Unlike the existing methods in the literature, the key idea of the AlphaQCM method relies on the unbiased estimation of
*See Appendix G for the ablation study.

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
Cui, C., Wang. W., Zhang, M., Chen, G., Luo, Z., and Ooi, B. C. Alphaevolve: A learning framework to discover novel alphas in quantitative investment. In Proceedings of the 2021 International Conference on Management of Data, SIGMOD '21, pp. 2208-2216 ACM, 2021.
International Conference on Neural Information Process- ing Systems, Red Hook, NY, USA, 2019. Curran Asso- ciates Inc.
Lin, X., Chen, Y., Li, Z., and He, K. Stock alpha mining based on genetic algorithm. Technical report, Huatai Securities Research Center, 2019a. URL https:
Dabney, W., Ostrovski, G., Silver, D., and Munos, R. Im- plicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Confer- ence on Machine Learning, volume 80, pp. 1096-1105. PMLR, 2018a.
/ //crrm.ttsee com cttp.//ccccccc//////////010.
pdf.
Lin, X., Chen, Y., Li, Z., and He, K. Revisiting stock alpha mining based on genetic algorithm. Technical report, Huatai Securities Research Center, 2019b. URL https:
Dabney, W., Rowland, M., Bellemare, M. G., and Munos, R. Distributional reinforcement learning with quantile regression. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Inno- vative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'1S/AAA TAI/////AA///A/AAA 18/EAAT 18. AAAI Press, 2018b.
/ //crm. ht com. 2tt//////011/11000000111 a
pdf.
Mavrin, B., Yao, H., Kong, L., Wu, K., and Yu, Y. Distri- butional reinforcement learning for efficient exploration. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 4424 4434. PMLR, 2019.
Ding, Q., Wu, S., Sun, H., Guo, J., and Guo, J. Hicrarchi- multi scale gaussian transformer for stock movement prediction. In Proceedings of the Twenty-Ninth Inter- national Joint Conference on Artificial Intelligence, pp- 4640-4646. International Joint Conferences on Artificial Intelligence Organization, 7 2020.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve- ness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level con- trol through deep reinforcement learning. Nature, 518: 529-533, 2015.
Feng, F., He, X., Wang, X., Luo, C.,, Liu, Y., and Chua, T.-S. Temporal relational ranking for stock prediction. ACM Transactions Information Systems, 37:1-30, 2019.
Petersen, B. K., Landajuela, M., Mundhenk, T. N., Santiago, P., Kim, S. K., and Kim, J. T. Deep symbolic regres- sion: Recovering mathematical expressions from data via risk-seeking policy gradients. In Proceedings of the International Conference on Learning Representations, 2021.
Huber, P. Robust Estimation of Location Parameter. The Annals of Mathematical Statisties, 35:73-101, 1964.
Jegadeesh, N. Evidence of predictable behavior of security returns. The Journal of Finance, 45:881-898, 1990.
Jegadeesh, N. and Titman, S. Returns buying winners and selling losers: Implications for stock market efficiency. The Journal of Finance, 48:65-91, 1993.
Real, E., Liang, C., So, D. R., and Le, Q. V. Automl- 7ero: Evolving machine learning algorithms from scratch. In Proceedings of the 37th International Conference on Machine Learning. JMLR.rg, 2020.
Kakushadze, Z. 101 Formulaic Alphas. March 2016.
Schaul, T. Quan, J., Antonogiou, I., and Silver, D. Priori- tized experience replay, 2016.
Koa, K. J., Ma, Y., Ng. R., and Chua, T.SS. Diffusion variational autoencoder for tackling stochasticity in multi- step regression stock price prediction. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 1087-1096, 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms, 2017.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning 200M), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Yang, D., Zhao, L., Lin, Z., Qin, T., Bian, J., and Liu, T.-Y. Fully parameterized quantile function for distributional reinforcement learning. In Proceedings of the 33rd Inter- national Conference on Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Lecarpentier, E. and Rachelson, E. Non-stationary markov decision processes a worst-case approach using model- based reinforcement learning. In Proceedings of the 33rd
Yang, X., Liu, W., Zhou, D., Bian, J., and Liu, T. Qlib: An ai-oriented quantitative investment platform. 2020.
10

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
Yu, S., Xue, H., Ao, X., Pan, F., He, J., Tu, D., and He, Q- Generating synergistic formulaic alpha collections via reinforcement learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, pp. 5476-5486. Association for Computing Machinery, 2023.
Zhang, N. and Zhu, K. Quantiled conditional variance, skewness, and kurtosis by fornish-fisher expansion, 2023.
Zhang, T., Li, Y., Jin, Y., and Li, J. Autoalpha: an efficient hierarchical evolutionary algorithm for mining alpha fac- tors in quantitative investment, 2020.

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
A. Evaluation Metric for Alphas
To measure the performance an alpha, it is common practice to use the IC as the evaluation metric. For a given alpha pool F, the IC for this alpha pool F is defined as:
ya)l = Σ Ei= (ā,,
1 5
-as) Us)
IC) =
8
>i=1 (āi,s - 2=)²Ei=1 (Jiis - ys)
N
where {( , element of O₂ and Ys. respectively,
}} is the sample of meta-alphas based on F and true returns {3a)) āi,o and Wi, are i-th
N aiss and V.
1 N N Vi.e-
1
Σ 1=1
N
Here, Corr(ā, ys) calculates the cross-sectional correlation to measure the predictive power of F at time s, while the Mean operator used obtain a time-averaged result over the period containing all of S timepoints.
B. Available Features and Operators
Table B.5. Description of available features and operators.
Tokens
Description
Features
Openinghhighloow/doooigg/wwwwe price or volume of stock i at time s.
Constant Time delta
Number from f-30, -00, -5, -2, -1, -1.50.5, -0.01, 0.01, 1, 1, 5, 10, 30). Integer from {10, 20, 30, 40, 50), which is used in the time-series operators.
Time-series Operators
(ei d)
Return the value of Ui where d the time delta and ui,s is feature of stock i time s.
TsRank (Hi,s; d)
Return the rank of R4,, among {ur,,y..., ui, Return the mcan/mediiam/sum//ttaddaa deviation/variance of {ui,..... Wi Return the maximun/minimum value of (ui.s:... Mi.s-d). Return the weighted/exponentialy weighted moving average of {u.......
Mean/ Mean//Me/aaaa Med/Sem//aaaa Std/ Sta/Var (ui d)
Mux/
Mux/Min Min (Wi,s d)
WMA EMA/EMA (Ui d)
ui,s-d}-
Cov/Corr (Ui,s1 Zi,5 d)
Return the covariance/coorelation between Wi,s and 2i,a based on samples {u 21,a) ..., (Ut,--d 2,,-4)), where uy, and 24,s are features of stock at time
Cross-sectional Operators
Sign (ui,s)
Return if the value of ug, is positive, otherwise return O.
Abs (wi,a)
Return the absolute value of Ug,s-
Log (ui,s)
Return the logarithmic value of us.s-
Rank (u5,e)
Return the rank of u,,, among "i......,,yy..
Add/ aad//ub//in Sub/Mull Div (ui,,, 2i,s)
Return the the result of http://hhraacngggmmmiiiiign/nnii ur,s and 2i,--
Greater Less (ui,es 25,))
Return greater/less value of us,s and 2i,0-
12

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
C. Asymptotic Properties
To establish the asymptotic properties of the QCM estimators, some mild assumptions are required. For ease presentation. define
Φ1
of
3
((r, + (xr, y /((z,a) h(a,a))(z.a) 6 y (b(,aa ((2,a) 24
ei a) E2(2,
1
2
02
302
a((,a)=
and (xx
OK
OK
3Φx
sx(2,a)
under some notations in Section 3.3.1. Then, Proposition 3.1 holds if the following two classical assumptions in the regression literature hold:
Assumption C.I. $ 8 positive definite.
Assumption C.2. D'e(z, a)/K K ", 0 as + co.
To be specific, from the regression model (4), we can first obtain the following matrix form:
i cz a) = ( a) + s(x, a).
where (xr, a) = (0, (x), (x, a) Next, under Assumption D.1, the OLS estimator for 8(2, a) is
is the vector of estimated quantiles.
4 (xx 8(2, a) = ( (
which entails
(x,, a) J(0, a) = (Φb b'0 0'0(r, a) 5(z,a) a)
4'0 2. a) + E(x, a)) 8(z,a)
4 D'e(r,a) a)
K
K
P, 0,
where the second equation follows from the linear structure between quantiles and moments (Cornish-Fisher expansion). the third equation follows from rearranging terms, and the last convergence follows from Assumption C.2 and Slutsky's theorem. Hence, this proves the consistency of
8 8 a) (z) (1.
In other words, we have
(x, P4 ((z, a) (x
(2 2 a),
(x, -, Vh(a, a)s(s,
6
(1 a) P, v h[,, a) [k(2, a)
24
Finally, applying continuous mapping theorem, we obtain
T( a) = (x P, h(x,a),
8(x,a) 65,(2, a)
p s(,
(2
M(r,a)= = 248 a)
+3 P, k(æ, a).
(2
13

---

for k' = I....,'', where ((x a) is the target Q network output, and 0w (z, a) is the target quantile network output. In the IQN framework, besides T for quantile network, there is another 7 = ( F independently sampled for the target quantile network. Notably, the target networks share the same architecture with their online counterparts but differ in AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
This completes the technical proof of Proposition 3.1.
Moreover, if assume D'E(z, a) / - 0 almost surely K in Assumption C.2, all of convergence results in Proposition 3.1 hold almost surely. Interested readers could refer to Zhang & Zhu (2023) for more comprehensive analysis on the QCM method.
D. Vanilla Quantile-based Variance Estimator
Recall that the widely used quantile representation Ze, (x, a), defined in (2), approximates (² a) with a mixture of Dirac distributions. With this approximation, Var Z" (æ, a)] can be approximated by the vanilla quantile-based variance estimator Var[Z (x,, a)). where
K
Var Z0 0, (x,0) a)] = E
(Th - rk-1) ( (a((, a) ElZz, (c, (r)
Here, (2 for k I...., are the -h quantile estimates, and
El a]] Σ (T-+1 17
E. Additional Implementation Details
E.1. Training Procedure
Motivated by (1), for each transition (Ii, de, Te, 1141,, the quantile temporal difference error (Dabney et al., 2018a) defined as
, = + 70w Fi+1: argmax U(ore, a'EA
0& (1 at)
network parameters, which are frozen but periodically synchronized from the online network.
Based on (ō,, for batch of ((12, at, Pty Ft+1)) the Huber loss (Huber, 1964) for optimizing the quantile network is defined as
K' ((w*) Σ ΣΣ P
(6x,),)),
tEbulch k=
where wf includes all network parameters in the quantile network, and Φ (8k, (ōu. / 0)] I..(6e,)'))) with x
2 Sh.
if w.l S
Lx (6k.
otherwise
i being a hyperparameter, and I(.) being the indicator function.
Meanwhile, network is optimized by minimizing the sum squared temporal difference errors:
((w) = Σ r: max @(F++1,a') - Q(2,, at)
d'EA
tEbatch
where w represents network parameters in the Q network.
14

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
Note that we equip the agent with the prioritized experience replay method (Schaul al., 2016). While the classical experience replay method samples transitions uniformly from a replay memory, the prioritized experience replay method improves sampling efficiency by replaying more frequently transitions from which there is more to learn. Specifically, it samples transitions with prior probability related to the last encountered quantile temporal difference error, where
Pr ΣΣ Prv (ō 2))
with 17 being hyperparameter. Clearly, the agent equipped with prioritized experience replay can utilize the transitions guided by the QCM method more efficiently.
E.2. Network Hyperparameters
Our network-related hyperparameters are consistent with those in Yu et al. (2023) for fair comparison. Specifically, in both the Q network and quantile network, the LSTM feature extractor <)(+) has 2-layer structure with hidden layer dimension of 128 with dropout rate of 0.1, and the fully connected heads have two hidden layers of 64 dimensions. Moreover, the T-embedding network maps each quantile level into 64-dimensional embedding, as defined in Dabney et al. (201 8a).
E.3. DRL Hyperparameters
Besides the network-related hyperparameters, some additional hyperparameters that our DRL algorithm inherits from the IQN and DQN algorithms are listed in Table E.6.
Table E.6. Additional hyperparameters.
Hyperparameter
Values
Min history start learning
10,000
e-greedy
0.01
Memory size
100,000
Learning rate
5e-5
Optimizer
Adam
Online network update interval (replay period)
Target network update interval
5,000
Batch size
128
K (length of T in the IQN algorithm)
64
K' (length of + in the IQN algorithm)
64
K (constant in the Huber loss)
1.0
7) (constant in the prior probability)
0.5
À (tuning parameter)
0.5, 1,2
P (Alpha pool size)
10, 20, 50, 100
Total step
250,000 (P 10).
300,000 (P 20).
350,000 (P 50).
400,000 (P = 100)
F. Details About Baseline Methods
To evaluate the performance of our AlphaQCM algorithm, we consider four types of baseline methods in Section 4. The first type of baseline methods aims to measure human-level performance in alpha discovery, serving as a benchmark for the remaining machine- based methods. In this category, we apply the well-known Alpha101 method as a representative, which employs the 101 formulaic alphas from Kakushadze (2016) to construct mega-alphas. To ensure model interpretability, these alphas combined linearly based the samples in the training and validation sets", using squared loss function. The
We download the data 101 formulaic alphas from "www.ricequant.com"
15

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
comparison between the AlphaQCM and Alpha101 methods aims evaluate whether the RL-based alpha generator can outperform human experts.
Next, the second type of baseline methods directly applies some end-to-end machine learning models to capture the linkage between stock features and their future returns. Clearly, these Saaiine-leanning-beee alphas lack interpretability but are straightforward to generate. Following Yu et al. (2023), we implement the MLP, XGBoost, and LightGBM methods using the open-source library Qlib (Yang et al., 2020), with pre-specified hyperparameters. The comparison between the AlphaQCM method and the second type of baseline methods to verify whether the combination of interpretable alphas can outperform complex and non-interpretable alphas.
Moreover, the third type of baseline methods depends on the genetic programming algorithm. This type of methods employs the GP method to generate alphas in a one-by-one manner, with the IC being the fitness measure. Specifically, for the GP w/o filter method, the top-/" generated alphas with the highest ICs within the training set are used to form mega-alpha via a linear model, which is fitted on the validation set. In contrast, the GP w/ filter method selects the top- P performing alphas with an additional mutual-IC filter, ensuring that any pair of alphas in the set does not have a mutual IC higher than 0.7. The comparison between the AlphaQCM and the GP-based methods aims probe the limitations of the GP method when involving large populations.
Lastly, the AlphaGen method (Yu et al., 2023) is the most closely related competitor to our ACPhaQCM method, which also belongs to the category of RL-bbased methods. The primary difference between these two methods is the algorithm used for discovering alphas. The comparison between the AlphaQCM and AlphaGen methods aims to check whether the simple PPO algorithm is adequate for such a non-stationary and reward-sparse MDP.
G. Additional Experiments
G.1. Impact of Parameter Size
As mentioned in Appendix E.2, the network-related hyperparameters for both the AlphaGen and AlphaQCM methods are kept consistent to ensure a fair comparison. However, due to differences in their network architectures, the total parameter size of these two methods varies. Specifically, the AlphaGen method employs two types of networks: an actor network and a critic network, with a total of 298, 609 trainable parameters. In contrast, the AlphaQCM method utilizes four networks: an online Q-network, an online quantile network, and their respective target networks (with frozen parameters), resulting in 572, 000 trainable parameters. Thus, the AlphaQCM method has nearly twice the number of trainable parameters compared to AlphaGen, with the same network-related hyperparameters.
To examine whether the superior performance of the AlphaQCM method over the AlphaGen method stems from its ability to handle non-stationary and reward- -parse alpha discovery MDP rather than the increased model complexity, we design two additional variants: a larger AlphaGen method and a smaller AlphaQCM method. The larger AlphaGen method has 564,953 trainable parameters, achieved by setting the hidden dimensions of the LSTM feature extractor to 180, with fully connected heads comprising two hidden layers of 90 dimensions each. Conversely, the smaller AlphaQCM method has 297, 070 trainable parameters, achieved by setting the hidden dimensions of the LSTM network to 90, with fully connected heads comprising two hidden layers of 48 dimensions each. These configurations ensure that the larger AlphaGen method aligns with the parameter size of the proposed A AlphaQCM method, while the smaller AlphaQCM method matches the complexity of the original AlphaGen method.
Table G.7 reports the IC values for methods with different parameter sizes. From this table, we first observe that both methods are robust to changes in parameter size. Additionally, the AlphaQCM method consistently outperforms the AlphaGen method across different parameter scales, with substantial performance gaps. These findings further emphasize the capability of the AlphaQCM method in solving the non-stational stationary and reward-sparse alpha discovery MDP effectively.
G.2. Impact of Model Architecture
In this subsection, we conduct an ablation study to assess whether the MAMBA block could provide efficiency and scalability advantages for the AlphaQCM method. Specifically, we replace the original LSTM module with an MAMBA block, using 128diimensional hidden layer and 4 attention heads, while keeping the rest of the network architecture unchanged. The results from the ablation study are shown in Table G.8. Here, we only run 3 random seeds for the MAMBA- -based AlphaQCM method.
16

---

AlphaQCM: Alpha Discovery with Distributional Reinforcement Learning
Table G.7. Out-of-sample IC values for methods with different parameter sizes.
CSI 300
CSI 500
Market
Method
Mean
Std
Mean
Std
Mean
Std
Panel A: Small-parameter method (nearly 300K parameters)
AlphaGen
8.13%
0.94%
8.08%
1.23%
6.04%
1.78%
AlphaQCM
8.23%
0.90%
9.68% %
1.07% %
9.12%
1.47%
Panel B: Large-parameter method (nearly 570K parameters)
AlphaGen
8.09%
1.33%
8.40%
1.48%
6.67%
2.03%
AlphaQCM
8.49
1.03%
9.55%
1.16%
9.16% %
1.61 %
As seen, there no significant difference between the IC values from the LSTM-based and MAMBA-based AIPhaQCM methods. This may due the relatively short token sequence (i.e., the formula of alpha), which consists of only the top 20 tokens. Consequently, the scalability advantages of the MAMBA block do not appear in this case. Additionally, there no notable efficiency difference in terms of time, the majority of the time is spent agent- agent-envirroment interaction.
Table G.8. Comparison between LSTM and MAMBA models.
CSI 300
CSI 500
Market
Model
Mean
Std
Mean
Std
Mean
Std
LSTM
8.49%
1.03%
9.55%
1.16%
9.16%
1.61 %
MАMBA
8.56%
1.05%
9.32%
1.16%
9.13%
1.63%
G.3. Impact of Expanded Dataset
In this subsection, we expand the testing set in the paper from the period (2021/01/01 to 2022/12/31) to the period (2021/01/01 to 2024/12/31), while the training and validation periods are kept. Apart from the extended time span, all other experimental settings remain consistent with those in Table 1.
Table G.9. Out-of-sample IC values with expanded testing set.
CSI 300
CSI 500
Market
Method
Mean
Std
Mean
Std
Mean
Std
Alpha101
3.02%
4.11%
3.78%
MLP
1.47%
0.26%
2.15%
0.66%
2.04%
0.76%
XGBoost
1.80%
0.97%
3.16%
1.11%
3.25%
1.33%
LightGBM
1.85%
0.79%
2.32%
0.84%
2.37%
1.06%
GP w/o filter
1.15%
1.87%
1.04%
1.65%
0.89%
2.02%
GP w/ filter
2.47%
2.28%
3.54%
2.14%
0.56%
2.57%
AlphaGin 4.13%
0.95%
4.19%
1.39%
3. 1.9
1.94%
AlphaQCM
5.48%
1.17 %
5.87 %
1.33%
4.83
1.79%
The updated experimental results are shown in Table G.9. From this table, we can see that the advantages our AlphaQCM method remain evident over the baseline methods. However, nearly all methods show a noticeable decrease in out-of-sample IC values. This issue could potentially be addressed by re-fitting the models in these methods whenever new information available.
17


Reviews:
[{"review_id": "gcOwwdmJib", "reviewer": "Reviewer_5Vkr", "summary": "This paper proposes a distributional reinforcement learning-based alpha discovery process for algorithmic trading in the stock market. Motivated by the quantile conditional moments (QCM) method, the authors provide an unbiased estimation of variance from quantiles to improve the performance of discovering synergistic formulaic alphas. This estimated variance is used to the bonus of the Q values on the action selection process in RL training. In experiments, the proposed method outperforms several traditional baselines and AlphaGen algorithm in the offline RL trading benchmarks.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments and suggestions, which give us a great help to improve our article in the future. Limited by the max length, we hope you will find that our responses have successfully addressed the important issues you have raised.\n\n---\n\n## Q1\n*The authors present a theoretical statement in Proposition 3.1 to estimate the unbiased variance for the bonus in Equation (5), which is the main claim of this paper. However, they provide this proposition without any proof.*\n\n**A1:** Thank you for your thoughtful comment. We did not include the detailed proof in the original submission, as this paper is primarily application-driven. Indeed, the proof is relatively straightforward. ***We now provide the technical proof of Proposition 3.1 below and will include it in the appendix in the revised version.***\n\nTo begin, we recall the linear regression model (4) from the manuscript:\n$$\\hat{\\theta}_k(x, a)=\\zeta(x, a)+Q^*(x, a)+\\Phi'_k\\delta(x, a)+\\varepsilon_k(x, a),$$\nfrom which we have the matrix form:\n$$\\hat{\\theta}(x, a)=\\Phi\\Delta(x, a)+\\varepsilon(x, a),$$\nwhere $\\hat{\\theta}(x, a)=(\\hat{\\theta}_1(x, a),\\dots,\\hat{\\theta}_K(x, a))'$, $\\Delta(x, a)=(\\zeta(x, a)+Q^*(x, a),\\delta(x,a)')'$, $\\Phi$ and $\\varepsilon(x, a)$ are defined in the Appendix D of manuscript.\n\nNext, assuming Assumption D.1 hold, the least squares estimator for $\\Delta(x, a)$ is given by:\n$$\\hat{\\Delta}(x, a)=(\\Phi'\\Phi)^{-1}\\Phi'\\hat{\\theta}(x, a).$$\n\nAccompanied with Assumption D.2 and Slutsky’s theorem, we can derive:\n$$\\hat{\\Delta}(x, a)-\\Delta(x, a)=(\\Phi'\\Phi)^{-1}\\Phi'[\\Phi\\Delta(x, a)+\\varepsilon(x, a)]-\\Delta(x, a)=(\\Phi'\\Phi)^{-1}\\Phi'\\varepsilon(x, a)= (\\frac{\\Phi'\\Phi}{K})^{-1}\\frac{\\Phi'\\varepsilon(x, a)}{K}\\xrightarrow{p}0.$$\nIt indicates the consistency of $\\hat{\\Delta}(x, a)=(\\hat{\\delta}_1(x, a), \\hat{\\delta}_2(x, a), \\hat{\\delta}_3(x, a), \\hat{\\delta}_4(x, a))'$. In other words, we have $\\hat{\\delta}_1(x, a)\\xrightarrow{p}\\zeta(x, a)+Q^*(x, a)$, $\\hat{\\delta}_2(x, a)\\xrightarrow{p}\\sqrt{h(x, a)}$, $\\hat{\\delta}_3(x, a)\\xrightarrow{p}\\frac{\\sqrt{h(x, a)}s(x, a)}{6}$ and $\\hat{\\delta}_4(x, a)\\xrightarrow{p}\\frac{\\sqrt{h(x, a)}[k(x, a)-3]}{24}.$\n\nFinally, applying continuous mapping theorem, we obtain $\\hat{h}(x, a)=\\hat{\\delta}_2^2(x, a)\\xrightarrow{p}h(x, a)$, $\\hat{s}(x, a) = \\frac{6\\hat{\\delta}_3(x, a)}{\\hat{\\delta}_2(x, a)} \\xrightarrow{p} s(x, a)$, and $\\hat{k}(x, a) = \\frac{24\\hat{\\delta}_4(x, a)}{\\hat{\\delta}_2(x, a)} + 3 \\xrightarrow{p} k(x, a)$.\n\nThis completes the technical proof of Proposition 3.1.\n\n---\n\n## Q2\n*The second problem is that the target tasks seems offline RL frameworks, but the proposed method is an online RL algorithm.*\n\n**A2:**  While the stock data is fixed during the learning process, the alpha discovery task should be solved in an online framework. ***The key reason is that all experiences (i.e., the tuples of state, action, next state, and reward) are collected through agent-environment interaction, rather than being pre-provided.*** As noted in Levine et al. (2020), an online RL algorithm updates the policy with streaming data collected by the agent itself. In contrast, offline RL uses a dataset of experiences collected by an external (potentially unknown) behavior policy, and this dataset is not altered during training. \n\n---\n\n## Q3\n*The key evaluation criteria of this paper is IC, ... However, the paper does not provide any detailed explanation ...*\n\n**A3:**  As a standard metric for evaluating alpha discovery ability, IC is basically time-series averaged Pearsons’ correlation. A higher IC value indicates that the set of discovered alphas is more powerful in predicting stock trends, which, in turn, can lead to greater financial profits for real traders. \n\n---\n\n## Q4\n*In addition, the previous baseline, alphagen also provides Rank IC to help the readers to check the performance ..., It seems to provide more baselines in alphagen, such as PPO\\_filters to compare the performances of RL based algorithms.*\n\n**A4:** Limited by the max length of rebuttal, we have included the out-of-sample IC and Rank IC values for AlphaGen, AlphaQCM and PPO in the following table, where values inside parentheses are the standard deviations. As shown, the AlphaQCM method still demonstrates significant advantages over the comparison methods, even though achieving a high Rank IC value is not the primary goal.\n\n|Method|CSI300 IC|CSI300 RankIC|CSI500 IC|CSI500 RankIC|Market IC|Market RankIC|\n|-|-|-|-|-|-|-|\n|PPO w/ filter|1.14(1.71)|3.02(2.88)|0.98(1.36)|2.79(1.46)|2.15(1.86)|2.58(1.04)|\n|AlphaGen|8.13(0.94)|9.40(1.10)|8.08(1.23)|8.71(1.67)|6.04(1.78)|7.49(2.25)|\n|AlphaQCM|8.49(1.03)|9.88(1.35)|9.55(1.16)|9.24(1.53)|9.16(1.61)|9.71(2.02)|\n\n---\n\n## Due to length limitations, if you want, we can answer your unresolved questions in the next phase.\n\n[1] Levine, S., A. Kumar, G. Tucker, and J. Fu (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems.", "claims_and_evidence": "The authors present a theoretical statement in Proposition 3.1 to estimate the unbiased variance for the bonus in Equation (5), which is the main claim of this paper. However, they provide this proposition without any proof. If there is the proof, please let me know.\n\nThe second problem is that the target tasks, CSI300,500 seems offline RL frameworks, but the proposed method is an online RL algorithm. It is not clear that why the proposed method works in this offline RL setting.. However, they provide this proposition without any proof.", "methods_and_evaluation_criteria": "The key evaluation criteria of this paper is IC, information coefficient, which is based on the Pearsons' correlation coefficient. However, the paper does not provide any detailed explanation of this metric, and it is hard to understand what is the exact meaning of out-of-sample IC values in Tables.\n\nIn addition, the previous baseline, alphagen also provides Rank IC to help the readers to check the performance of algorithms.\n\nI think the authors need to add this details to improve the quality of this submission.", "theoretical_claims": "I have tried to check the correctness of theoretical claims, but the authors does not provide the proof of proposition 3.1.\nIf any, please let me know.", "experimental_designs_or_analyses": "The basic experimental design is followed by alphagen paper, which seems valid.\nHowever, there is no enough explanation of the specific metric, such as out-of-values and the ablation study to help the readers to understand how the proposed bonus works to find the better alpha.", "relation_to_broader_scientific_literature": "This work is for algorithm trading in finance to achieve the better profit.", "essential_references_not_discussed": "Although the authors provide the key references, QCM methods and Cornish-Fisher expansion, the detailed explanation is not enough.", "other_strengths_and_weaknesses": "Using the distributional RL properties to analyze the uncertainty of discovering alpha is quite interesting to me, but the quality of the paper should be elaborated to satisfy the standard of this venue.", "other_comments_or_suggestions": "It seems to provide more baselines in alphagen, such as PPO_filters to compare the performances of RL based algorithms.", "questions_for_authors": "A I mentioned above, can you provide the proof of proposition 3.1 and the more details of equation (5), which is the key part to explain how the prosed method works in the action selection process.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jALnBXPKL5", "reviewer": "Reviewer_YFUK", "summary": "This paper introduces AlphaQCM, a novel distributional reinforcement learning (DRL) method for discovering synergistic formulaic alphas in finance. The authors conceptualize the alpha discovery process as a non-stationary and reward-sparse Markov Decision Process (MDP) and propose AlphaQCM to address these challenges. AlphaQCM leverages the IQN algorithm to learn quantiles of cumulative discounted rewards and employs the Quantiled Conditional Moment (QCM) method to estimate unbiased variance, even under non-stationarity. This variance is then used as an exploration bonus to guide the agent in navigating the vast search space of formulaic alphas. Empirically, AlphaQCM is shown to outperform competitors, including AlphaGen and GP-based methods, on real-world Chinese stock market datasets, particularly when dealing with larger datasets and more complex financial systems. The main algorithmic idea is the integration of QCM within a distributional RL framework to handle non-stationarity and reward sparsity, leading to more efficient and effective alpha discovery.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and positive attitude towards our work. To enhance the quality of this paper, we have carefully considered your suggestions and questions. Below, we provide a point-by-point response to your comments:\n\n---\n\n## Q1 from \"Methods And Evaluation Criteria\":\n*However, a notable limitation of the evaluation is its focus solely on the Chinese A-share stock market. ... To strengthen the evaluation, future work could explore the performance of AlphaQCM on datasets from diverse global markets to assess the broader applicability and robustness of the method.*\n\n**A1**: To demonstrate the generalizability of our AlphaQCM method, we have conducted additional experiments using the U.S. stock market dataset (S&P 500). The results show that AlphaQCM continues to substantially outperform all competitor methods on both the CSI500 and S&P 500 datasets. For detailed results, please see A2 in the response to Reviewer JbCG.\n\n---\n\n## Q2 from \"Theoretical Claims\":\n*While Proposition 3.1 regarding the consistency of moment estimators is mentioned, the paper does not delve deeply into formal proofs or extensive theoretical analysis.*\n\n**A2**: Thank you for your understanding regarding our focus on the application. To complete the theoretical foundation, we have provided the technical proof of Proposition 3.1 in this rebuttal. If you are interested, please refer to A1 in the response to Reviewer 5Vkr for the detailed proof. Moreover, we will include the proof in the Appendix in the revised version.\n\n---\n\n## Q3 from \"Other Strengths And Weaknesses\":\n*While the paper mentions Chinese A-share stock market datasets, it lacks specific details about the exact data sources used (e.g., specific data vendors, data cleaning procedures). This could slightly hinder reproducibility.*\n\n**A3**: Thank you for your insightful suggestion. In this paper, we use the Chinese stock data from the Baostock database and the U.S. stock data from WRDS database. The data cleaning procedures follow those outlined in [1], and the corresponding codes are provided for reproducibility.\n\n---\n\n## Q4 from \"Other Strengths And Weaknesses\":\n*The definition of the \"Market\" dataset is not explicitly clarified. It is mentioned as \"all stocks,\" but it’s unclear if this refers to CSI-1000 or a different universe of stocks. Clarifying this definition would improve clarity.*\n\n**A4**: Apologies for the confusion. Regarding the \"Market\" stock pool, we refer to all stocks listed on the Shanghai and Shenzhen Stock Exchanges. We will clarify this definition in the revised version to ensure better clarity.\n\n---\n\n## Q5 from \"Other Strengths And Weaknesses\":\n*As mentioned earlier, the evaluation is limited to the Chinese stock market. The generalizability to other markets needs further investigation to fully assess the robustness of AlphaQCM.*\n\n**A5**: Please refer to A1 above on this matter.\n\n---\n\n## Q6 from \"Other Strengths And Weaknesses\":\n*While the paper emphasizes formulaic alphas as being more interpretable than black-box ML models, it doesn’t deeply explore the actual interpretability of the alphas discovered by AlphaQCM or provide examples of the discovered formulas and their economic intuition.*\n\n**A6**: Thank you for your insightful question. To be honest, the discovered alphas can vary across different random seeds, and interpreting these alphas often requires solid financial background knowledge. As a result, we have chosen not to delve into this aspect in the manuscript.\n\nTo address your question, we provide two easy-to-interpret discovered alphas based on the Market dataset:\n\n1. $$\\text{RANK(Volume} \\times \\frac{\\text{HIGH}}{\\text{VWAP}} + \\text{CONSTANT}) / \\text{CLOSE}$$\n\n2. $$\\text{WMA(ABS}\\left(\\frac{\\text{CLOSE}}{\\text{OPEN}} \\times \\text{CONSTANT}\\right), 30)$$\n\n---\n\n**Explanation**:\n- In the first discovered alpha, $\\frac{\\text{HIGH}}{\\text{VWAP}} \\times \\text{Volume} + \\text{CONSTANT}$ can be interpreted as a volume-adjusted price upside variation. The calculated values are then cross-sectionally normalized using the RANK operator and further normalized by the closing price. This alpha suggests that future stock trends are influenced by intraday price variation.\n\n- On the other hand, the second alpha measures the absolute values of intraday returns and smooths them over a 30-day window via the WMA operator. This can be viewed as a special form of short-term reversal [2], a widely recognized formulaic alpha in finance.\n\n---\n\n## References\n[1] Yu, S., H. Xue, X. Ao, F. Pan, J. He, D. Tu, and Q. He (2023). Generating synergistic formulaic alpha collections via reinforcement learning. *In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23*, pp. 5476–5486. Association for Computing Machinery.\n\n[2] Jegadeesh, N. (1990). Evidence of predictable behavior of security returns. *The Journal of Finance*, 45, 881–898.", "claims_and_evidence": "Yes, the claims made in the submission are generally well supported by clear and convincing evidence. The central claim that AlphaQCM effectively addresses the challenges of non-stationarity and reward sparsity in alpha discovery is substantiated by the empirical results. Specifically, the paper demonstrates through extensive experiments that AlphaQCM consistently achieves superior Information Coefficient (IC) values compared to various baselines across different Chinese stock market datasets (CSI300, CSI500, and Market). The ablation studies further strengthen the claim by isolating the contribution of the QCM method and showing its advantage over using no variance or a vanilla variance estimator. The paper emphasizes the robustness of AlphaQCM and its improved performance in more complex market settings, which is also supported by the increasing performance gap relative to baselines as the stock pool size increases. The evidence presented, particularly in Tables 1, 2, and 3, appears to convincingly support the claim of improved alpha discovery efficacy using AlphaQCM, especially in non-stationary environments.", "methods_and_evaluation_criteria": "The proposed method, AlphaQCM, and the evaluation criteria are generally appropriate for the problem of formulaic alpha discovery in finance. The conceptualization of alpha discovery as an MDP is a reasonable and increasingly adopted approach in this domain. The use of distributional reinforcement learning, and specifically the integration of QCM, is a novel and potentially impactful methodological contribution to address the identified challenges of non-stationarity and reward sparsity.\n\nHowever, a notable limitation of the evaluation is its focus solely on the Chinese A-share stock market. While the Chinese market is a significant and complex market, the generalizability of the findings to other global markets (e.g., US, European, or emerging markets with different market microstructures and regulatory environments) remains an open question. The evaluation criteria, primarily the Information Coefficient (IC), is a standard and accepted metric in quantitative finance for evaluating alpha performance, making it a relevant choice. To strengthen the evaluation, future work could explore the performance of AlphaQCM on datasets from diverse global markets to assess the broader applicability and robustness of the method.", "theoretical_claims": "Not Applicable. The paper is primarily focused on algorithmic development and empirical validation rather than presenting novel theoretical claims with formal proofs. While Proposition 3.1 regarding the consistency of moment estimators is mentioned, the paper does not delve deeply into formal proofs or extensive theoretical analysis. The strength of the paper lies in its methodological innovation and empirical demonstration of performance. Therefore, the absence of extensive theoretical claims is not a weakness in this context.", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound and valid. The authors have included a number of strong and relevant baselines for comparison, including:\n\n* Alpha101 (Human-designed alphas): Representing human expert knowledge, providing a benchmark for machine-driven discovery.\n\n* MLP, XGBoost, LightGBM (ML-based non-formulaic alphas): Representing end-to-end machine learning approaches, allowing for comparison against complex, but less interpretable, methods.\n\n* GP w/o filter, GP w/ filter (GP-based formulaic alphas): Representing the current mainstream approach for formulaic alpha discovery, allowing for direct comparison to existing formulaic methods.\n\n* AlphaGen (RL-based formulaic alphas): Representing the most closely related RL-based baseline, highlighting the incremental improvement of AlphaQCM.\n\nFurthermore, the ablation studies are well-designed to isolate the impact of key components of AlphaQCM:\n\n* Variance Methods: Comparing QCM variance to no variance and vanilla variance, demonstrating the effectiveness of QCM.\n\n* DRL Backbones: Comparing IQN and QRDQN backbones, showing the robustness across different distributional RL algorithms.\n\n* Domain Knowledge: Assessing the impact of initializing the replay buffer with expert-designed alphas, providing insights into the role of prior knowledge.\n\n* Parameter Size: Analyzing the performance with varying parameter sizes for both AlphaQCM and AlphaGen, demonstrating robustness and consistent outperformance.\n\nThe use of 10 random seeds for each experimental setting to account for stochasticity is also a good practice. The results are presented clearly, with means and standard deviations of IC values, enabling a robust comparison of the methods.", "relation_to_broader_scientific_literature": "The key contribution of this paper is primarily related to the scientific literature within the domain of algorithmic trading and quantitative finance, specifically in the sub-area of formulaic alpha discovery. The paper builds upon and extends the existing literature on using machine learning, and particularly reinforcement learning, for financial signal generation. It directly relates to prior work that utilizes genetic programming and reinforcement learning for discovering formulaic alphas, such as the AlphaGen method.\n\nWhile the paper makes a valuable contribution to this specific niche, its broader contribution to the general machine learning literature might be considered somewhat limited. The core methodological novelty lies in the application of QCM for variance estimation within a distributional RL framework in the context of alpha discovery. While this is a technically sound and empirically effective approach for the specific problem, it might not represent a fundamentally new machine learning concept with broad applicability across diverse ML domains. However, the effective demonstration of addressing non-stationarity and reward sparsity in a real-world financial application is a valuable contribution to the intersection of ML and finance.", "essential_references_not_discussed": "No, based on my current understanding of the literature related to formulaic alpha discovery and reinforcement learning in finance, there do not appear to be any essential references that are critically missing from the paper. The paper cites relevant works on distributional RL, quantile regression, genetic programming for alpha discovery, and related RL-based alpha generation methods. The references provided seem to adequately contextualize the key contributions within the relevant scientific domain.", "other_strengths_and_weaknesses": "Strengths:\n\n* Originality: The application of the QCM method within a distributional RL framework to address non-stationarity and reward sparsity in alpha discovery is a novel and original contribution.\n\n* Significance: The paper addresses a practically significant problem in quantitative finance – the efficient and robust discovery of formulaic alphas, particularly in complex and dynamic markets. The demonstrated performance improvement over strong baselines highlights the practical significance of AlphaQCM.\n\n* Clarity: The paper is generally well-written and clearly explains the proposed method, experimental setup, and results. The figures and tables are informative and contribute to the clarity of presentation. The supplementary material further enhances clarity and reproducibility.\n\n* Empirical Validation: The extensive empirical validation on real-world datasets, including ablation studies and comparisons to multiple strong baselines, is a significant strength.\n\nWeaknesses:\n\n* Data Source Detail: While the paper mentions Chinese A-share stock market datasets, it lacks specific details about the exact data sources used (e.g., specific data vendors, data cleaning procedures). This could slightly hinder reproducibility.\n\n* \"Market\" Definition: The definition of the \"Market\" dataset is not explicitly clarified. It is mentioned as \"all stocks,\" but it's unclear if this refers to CSI-1000 or a different universe of stocks. Clarifying this definition would improve clarity.\n\n* Geographic Limitation: As mentioned earlier, the evaluation is limited to the Chinese stock market. The generalizability to other markets needs further investigation to fully assess the robustness of AlphaQCM.\n\n* Interpretability of Discovered Alphas: While the paper emphasizes formulaic alphas as being more interpretable than black-box ML models, it doesn't deeply explore the actual interpretability of the alphas discovered by AlphaQCM or provide examples of the discovered formulas and their economic intuition.", "other_comments_or_suggestions": "No further comments or suggestions at this time.", "questions_for_authors": "See the weakness part above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZmQlrtL58V", "reviewer": "Reviewer_JbCG", "summary": "The authors propose a method based on distributional reinforcement learning and QCM to learn a good set of well-formed features (i.e. alphas) for stock market prediction. The proposed method is compared with existing baselines and is shown to outperform these baselines on 3 stock market datasets.\n\n## Update after rebuttal\nI maintain my score, thank you.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review and very helpful feedback. Please let us know whether the points below are sufficient clarification for your concerns.\n\n---\n\n## Q1 from \"Theoretical Claims\": \n*There is not much theoretical result in this work except the consistency of the moment estimator under suitable assumptions. It is unclear to me whether the assumptions make sense in the task setup used in this work and I appreciate it if the authors could clarify this.*\n\n**A1:** Certainly. We are pleased to clarify the meaning and importance of these two assumptions. Recall the two assumptions as follows:\n\n**Assumption D.1:**  $\\Phi' \\Phi \\text{ is positive definite;}$\n\n**Assumption D.2:**  $\\frac{\\Phi' \\varepsilon(x, a)}{K} \\xrightarrow{p} 0  \\text{ as } K \\to \\infty.$\n\nAssumption D.1 is necessary for the existence of least squares estimator in the linear regression model (4) outlined in the manuscript. Verifying and ensuring this assumption are straightforward, as $\\Phi$ is a matrix built upon conditional quantiles of the Gaussian distribution.\n\nAssumption D.2 requires that the weighted sum of the stochastic residuals in the linear regression model (4) becomes negligible with respect to $K$ (i.e., the number of quantiles). The validity of this assumption is very important to ensure the consistency of least squares estimator.\n\nIf you are interested in a more detailed proof, please refer to A1 in the response to Reviewer 5Vkr.\n\n---\n\n## Q2 from \"Experimental Designs Or Analyses\":\n*The experimental design is sensible and the chosen set of baselines seems adequate. The only weakness here is that the evaluation is only restricted to essentially one dataset (and its subsets).*\n\n**A2**: Thank you for your insightful point. In this rebuttal, we have presented additional experimental results using U.S. stock market data. Due to the time constraints for the rebuttal, we are able to include only one stock pool: the largest 500 stocks (S&P 500). Apart from the choice of stocks, all other experimental settings are consistent with those presented in Table 1 of the manuscript. The experiment results are demonstrated in the table below.\n\n---\n| Method         | CSI500 Mean (%) | CSI500 Std (%) | S&P 500 Mean (%) | S&P 500 Std (%) |\n|----------------|-----------------|----------------|------------------|-----------------|\n| Alpha101       | 4.38            | -              | 3.12             | -               |\n| MLP            | 2.72            | 0.65           | 2.61             | 0.49            |\n| XGBoost        | 4.31            | 0.96           | 3.08             | 0.67            |\n| LightGBM       | 4.16            | 0.81           | 3.29             | 0.56            |\n| GP w/o filter  | 1.79            | 1.62           | 1.88             | 1.29            |\n| GP w/ filter   | 4.52            | 1.93           | 4.27             | 1.39            |\n| AlphaGen       | 8.08            | 1.23           | 7.48             | 0.77            |\n| **AlphaQCM**       | **9.55**            | **1.16**           | **8.46**             | **0.89**            |\n---\n\nFrom this table, we can find that the AlphaQCM method maintains significant advantages over all competitors on both the CSI500 and S&P 500 datasets, presenting robust performance across different market conditions. However, compared to the results of the CSI500 dataset, those of the S&P 500 dataset exhibit a decline in IC performance in most methods. This phenomenon may be attributed to the fact that the U.S. stock market is more efficient and influenced by breaking news, making it harder to capture predictable stock trends based on formulaic alphas.\n\n---", "claims_and_evidence": "The main claim of the authors is on the contribution of an exploration bonus based on QCM to address the issues of reward-sparsity and non-stationarity in the formulated MDP tasks. Empirically, the authors show clear improvement in the test performance when the proposed exploration bonus is employed.", "methods_and_evaluation_criteria": "The exploration-exploitation problem is a critical issue in RL and the use of exploration bonus is a well-known method to address the issue. The proposed method exploits distributional RL and QCM to obtain such exploration bonus. While not entirely novel, I believe it does make valuable contribution to the RL literature.", "theoretical_claims": "There is not much theoretical result in this work except the consistency of the moment estimator under suitable assumptions. It is unclear to me whether the assumptions make sense in the task setup used in this work and I appreciate it if the authors could clarify this.", "experimental_designs_or_analyses": "The experimental design is sensible and the chosen set of baselines seems adequate. The only weakness here is that the evaluation is only restricted to essentially one dataset (and its subsets).", "relation_to_broader_scientific_literature": "See \"Methods And Evaluation Criteria\" above.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w4mfTocK1k", "reviewer": "Reviewer_iRjv", "summary": "The paper introduces AlphaQCM, a novel reinforcement learning method for discovering formulaic alphas in finance. It conceptualizes alpha discovery as a non-stationary and reward-sparse Markov decision process and addresses challenges through a Q-learning framework combined with quantile-based variance estimation. The Q function and quantile networks, AlphaQCM navigates large search spaces efficiently, outperforming existing methods like AlphaGen in empirical tests on financial datasets. The results demonstrate AlphaQCM's superiority, particularly in complex market environments, providing a more effective and interpretable approach to quantitative finance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and helpful comments. To improve the quality of this paper, we have carefully considered your suggestions and questions. Limited by the max length, our shorten replies are as follows:\n\n## Q1\n*The claim that AlphaQCM outperforms AlphaGen is supported by results but lacks an in-depth analysis of AlphaGen’s failure cases.*\n\n**A1:**  Thanks for your good comment. While the alpha discovery task is conceptualized as an MDP, the AlphaGen employs a native PPO algorithm to find the optimal alpha discovery issue, ignoring the existence of non-stationarity and reward-sparsity in this MDP. This weakness leads to the potential for improvement in the AlphaGen method. By contrast, our AlphaQCM method aims to address the non-stationary and reward-sparse issue via employing the distributional RL algorithm and the QCM method. \n\nWe hope our answer above clarifies AlphaGen’s failure. In the revised version, we would highlight this in-depth comparison.\n\n---\n\n## Q2\n*Additionally, the assertion that AlphaQCM generalizes well to complex datasets is supported by tests on Chinese stock markets but would benefit from validation across different financial environments.*\n\n**A2:**  \nWe appreciate your valuable comment. In this rebuttal, we have provided additional experimental results using data from the U.S. stock market.  Please refer to A2 in the response to Reviewer JbCG. \n\n---\n\n## Q3\n*Newer architectures like MAMBA could offer efficiency and scalability advantages.*\n\n**A3:**  In this rebuttal, we have conducted an ablation study to assess whether the MAMBA block could provide efficiency and scalability advantages for the AlphaQCM method. Specifically, we have replaced the original LSTM module with a MAMBA block, using a 128-dimensional hidden layer and 4 attention heads, while keeping the rest of the network architecture unchanged. Due to time constraints, we are only able to run 3 random seeds for the MAMBA-based AlphaQCM method. The results from the ablation study are shown in the table below.\n\nAs seen, there is no significant difference between the IC values from the LSTM-based and MAMBA-based AlphaQCM methods. This may be due to the relatively short token sequence (i.e., the formula of alpha), which consists of only the top 20 tokens. Additionally, there is no notable efficiency difference in terms of time, as the majority of the time is spent on agent-environment interaction.\n\n|Model|CSI300 Mean| CSI300 Std| CSI500 Mean| CSI500 Std| Market Mean| Market Std|\n|-|-|-|-|-|-|-|\n|LSTM|8.49|1.03|9.55|1.16|9.16|1.61|\n|MAMBA|8.56|1.05|9.32|1.16|9.13|1.63|\n\n---\n\n## Q4\n*The paper overlooks the potential of large language models (LLMs) for formula discovery.*\n\n**A4:**  \nApologies for the lack of comparison and discussion regarding the literature on LLM-based formula discovery. One such method is the LLM-SR method [1], which you referenced. We have implemented the official code, but unfortunately, ***the alphas we found had less than 3% out-of-sample IC values on the CSI300 and CSI500 datasets. Note that these discovered by the AlphaGen and AlphaQCM methods have the out-of-sample IC values exceeding 8%.***\n\nHowever, we do not dismiss the potential of using LLMs for alpha discovery. The performance of the LLM-SR method is highly influenced by prompts. Furthermore, the stock market dataset is far more complex than the physical and biological datasets considered in [1]. **Some in-depth discussion and comparison would be done in the revised manuscript**, but due to time constraints, we leave it for future work.\n\n---\n\n## Q5 \n*Finally, the dataset only extends to 2022, missing recent financial data from 2023 and 2024.*\n\n**A5:**  \nThank you for your good comment. We have expanded the testing set in the manuscript from the period (2021/01/01 to 2022/12/31) to (2021/01/01 to 2024/12/31), while the training and validation periods are kept.\n\nThe experimental results are shown in the table below. From this table, we can find that the advantages of our AlphaQCM method remain evident over the baseline methods. However, nearly all methods show a noticeable decrease in out-of-sample IC values. This issue could potentially be addressed by re-fitting the models in these methods whenever new information is available.\n\n|Method|CSI300 Mean|CSI300 Std|CSI500 Mean|CSI500 Std|Market Mean|Market Std|\n|-|-|-|-|-|-|-|\n|Alpha101|3.02|-|4.11|-|3.78|-|\n|MLP|1.47|0.26|2.15|0.66|2.04|0.76|\n|XGBoost|1.80|0.97|3.16|1.11|3.25|1.33|\n|LightGBM|1.85|0.79|2.32|0.84|2.37|1.06|\n|GP w/o filter|1.15|1.87|1.04|1.65|0.89|2.02|\n|GP w/ filter|2.47|2.28|3.54|2.14|0.56|2.57|\n|AlphaGen|4.13|0.95|4.19|1.39|3.19|1.94|\n|AlphaQCM|**5.48**|**1.17**|**5.87**|**1.33**|**4.83**|**1.79**|\n\n---\n\n## Due to length constraints, we would be happy to address any remaining questions during the next phase.\n\n[1] Shojaee, P., K. Meidani, S. Gupta, A. B. Farimani, and C. K. Reddy (2025). Llm-sr: Scientific equation discovery via programming with large language models.", "claims_and_evidence": "The claims in the submission are generally supported by empirical evidence, particularly in demonstrating AlphaQCM’s superiority over baseline methods through Information Coefficient (IC) comparisons. The claim that AlphaQCM outperforms AlphaGen is supported by results but lacks an in-depth analysis of AlphaGen’s failure cases. Additionally, the assertion that AlphaQCM generalizes well to complex datasets is supported by tests on Chinese stock markets but would benefit from validation across different financial environments. Strengthening these areas with sensitivity analysis, robustness checks, and broader market applications would make the claims more convincing.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense for the problem of formulaic alphas in finance, as use of reinforcement learning (RL). The benchmark datasets are relevant but limited to Chinese markets, raising concerns about generalizability, testing on other global financial datasets would strengthen the findings. While the paper addresses reward sparsity using distributional RL, alternative methods such as intrinsic motivation or curriculum learning could have been explored. The choice of an LSTM-based architecture is reasonable for sequence modeling, but newer architectures like MAMBA could offer efficiency and scalability advantages. The paper overlooks the potential of large language models (LLMs) for formula discovery. A dedicated formula generator leveraging LLMs could provide a strong baseline or hybrid approach. Finally, the dataset only extends to 2022, missing recent financial data from 2023 and 2024, which is crucial for validating the continued effectiveness of AlphaQCM in evolving market conditions.", "theoretical_claims": "No major inconsistencies shown in the theoretical claims and in the formulas presented", "experimental_designs_or_analyses": "The paper presents a well-structured experimental design, comparing AlphaQCM against multiple baseline methods using Information Coefficient (IC) as primary performance metric. The use of datasets provides a reasonable range of financial environments to assess robustness. The reinforcement learning framework, including the Q-network and quantile-based variance estimation, is theoretically sound, and the experiments incorporate ablation studies to isolate the impact of key components like the QCM method.", "relation_to_broader_scientific_literature": "The contributions of the paper align with broader scientific literature on quantitative finance, reinforcement learning, and explainable AI. By addressing the challenge of discovering formulaic alphas through reinforcement learning, the paper builds upon prior work in both genetic programming-based and RL-based alpha discovery methods while introducing improvements for handling non-stationarity and sparse rewards. Its focus on creating explainable formulas is valuable, as many machine learning approaches in finance rely on black-box models that lack interpretability. But, it could benefit from deeper comparisons with alternative explainability-focused methods, such as symbolic regression or LLM-based formula generation, to better position its contributions within the broader literature.", "essential_references_not_discussed": "The paper miss the reference of LLM for formula research for example [1] \n\n[1] Shojaee, Parshin, et al. \"Llm-sr: Scientific equation discovery via programming with large language models.\" arXiv preprint arXiv:2404.18400 (2024).", "other_strengths_and_weaknesses": "No additional strengths and weaknesses", "other_comments_or_suggestions": "No additional comments", "questions_for_authors": "No additional questions for authors", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces AlphaQCM, which is a distributional reinforcement learning method that conceptualizes alpha discovery as a non-stationary, reward-sparse MDP and effectively utilizes quantile conditional moments for improved exploration and alpha discovery. The reviewers highlighted several notable strengths, including the originality of integrating the QCM method within a DRL framework for this financial application, the practical significance of addressing a challenging problem in quantitative finance, and the strong empirical validation demonstrating superior performance against relevant baselines on complex, real-world datasets. While reviewers noted weaknesses such as the initial omission of a theoretical proof  and limitations in the initial scope of evaluation datasets, the authors addressed many concerns in their rebuttal, and the core contributions and empirical results warrant acceptance. Ultimately, the AC recommends acceptance; however, the authors should revise the paper by incorporating the missing proofs of the theorems and all other reviewer suggestions.
Decision: Accept (poster)