Context:
---

Citation: Lastname, F.; Lastname, F.;
Lastname, F. Title. Journal Not Specifi ed
2024, 1, 0. https://doi.org/
Received:
Revised:
Accepted:
Published:
Copyright: © 2024 by the authors.
Submitted to Journal Not Specifi ed
for possible open access publication
under the terms and conditions
of the Creative Commons Attri-
bution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
A Stock Market Decision-Making Framework Based on
CMR-DQN
Xun Chen
1
, Qin Wang
1
, Chao Hu
2,
* and Chengqi Wang
3
1 School of Information and Communication Engineering, Hainan University, Haikou, China;
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>(X.C.); <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>(Q.W.); <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>(C.W.)
2 School of Electronic Infromation, Central South University, Changsha, China; <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>(C.H.)
* Correspondence: <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Abstract: In the dynamic and uncertain stock market, precise forecasting and decision-making are 1
crucial for profi tability. Traditional Deep Neural Networks (DNN) often struggle with capturing 2
long-term dependencies and multi-scale features in complex fi nancial time series data. To address 3
these challenges, we introduce CMR-DQN, an innovative framework that integrates Discrete Wavelet 4
Transform (DWT) for multi-scale data analysis, Temporal Convolutional Network (TCN) for ex- 5
tracting deep temporal features, and a GRU-LSTM-Attention mechanism to enhance the model’s 6
focus and memory. Additionally, CMR-DQN employs the Rainbow DQN reinforcement learning 7
strategy to learn optimal trading strategies in a simulated environment. CMR-DQN signifi cantly 8
improved the total return rate on six selected stocks, with increases ranging from 20.37% to 55.32%. 9
It also demonstrated substantial improvements over the baseline model in terms of Sharpe ratio 10
and maximum drawdown, indicating increased excess returns per unit of total risk and reduced 11
investment risk. These results underscore the effi ciency and effectiveness of CMR-DQN in handling 12
multi-scale time series data and optimizing stock market decisions. 13
Keywords: Deep reinforcement learning; Algorithmic trading; Deep Q network; Decision optimiza- 14
tion 15
0. Introduction 16
In the current global fi nancial market, the ultimate goal pursued by investors is not 17
only the appreciation of capital but also how to effectively manage and control investment 18
risks during this process. The core of this goal lies in fi nding the optimal balance between 19
risk and reward, to ensure that investments can bring sustainable and long-term returns. 20
These methods are not only time-consuming and labor-intensive, but also prone to the 21
infl uence of human emotions and biases when processing and analyzing large-scale fi nan- 22
cial data, thus affecting the objectivity and effectiveness of decision-making. Especially 23
when faced with rapidly changing market information, human traders fi nd it diffi cult 24
to react accurately in real-time, which poses signifi cant challenges to traditional trading 25
methods in the modern fi nancial market[ 1]. In this context, algorithmic trading, as an 26
innovative trading method, automates trading decision-making and execution through 27
computer programs, and brings new possibilities to modern fi nancial transactions with its 28
effi cient and objective characteristics. 29
The core driving force for the exploration and application of algorithmic trading 30
lies in the use of advanced computing technology to solve the limitations of traditional 31
trading methods. In the practice of algorithmic trading, rule-based methods and machine 32
learning-based strategies are the two main directions. Rule-based algorithmic trading 33
relies on an in-depth understanding of market rules, through the setting of a fi xed set of 34
trading rules to automate buying and selling operations. The advantage of this approach is 35
that it is straightforward and easy to implement, but at the same time, it is also infl exible 36
and may not be able to adapt in time in the face of rapid changes in the market [2]. In 37
Version July 10, 2024 submitted to Journal Not Specifi ed https://www.mdpi.com/journal/notspecifi edVersion July 10, 2024 submitted to Journal Not Specifi ed 2 of 18
contrast, machine learning-based algorithmic trading is more capable of data processing 38
and adaptability by training models to identify and learn complex patterns in the market. 39
Deep learning models, in particular, are able to automatically extract valuable features from 40
large amounts of historical data to support more accurate trading decisions [3]. However, 41
the ultimate goal of both rule-based and machine learning methods is to ensure the high 42
complexity and uncertainty of the fi nancial markets for strategy execution, and developing 43
trading strategies that can effectively capture market opportunities while adapting to their 44
dynamic changes remains a major challenge in the current algorithmic trading fi eld[ 4]. In 45
the past decade, deep learning and reinforcement learning (RL) technology have shown 46
great potential and application value in the optimization of fi nancial trading systems. Deep 47
learning provides a new perspective for the analysis and prediction of fi nancial markets 48
through its ability to automatically identify complex patterns and features in massive 49
amounts of data. Reinforcement learning, especially its branch of deep reinforcement 50
learning (DRL), has become a cutting-edge technology to solve sequential decision-making 51
problems by simulating the learning mechanism to fi nd the optimal action strategy in 52
continuous interaction with the environment [5]. In particular, DRL’s major breakthroughs 53
in mimicking human decision-making processes, such as AlphaGo’s victory over the World 54
Go Champion, further validated its ability to deal with complex decision-making problems. 55
In recent years, deep learning and reinforcement learning technologies have made 56
remarkable breakthroughs and successes in many fi elds, however, there are still many 57
challenges in applying these cutting-edge technologies to the fi eld of fi nancial trading, 58
especially in the development of effi cient trading algorithms. The high degree of uncer- 59
tainty in fi nancial markets, the non-stationary nature of data, and the need to make precise 60
decisions in a very short period of time have greatly increased the complexity of designing 61
and implementing effective trading strategies. In addition, an ideal fi nancial trading system 62
not only needs to pursue maximum returns, but also fully considers the importance of risk 63
control in the strategy to ensure that the trading strategy adopted has a high degree of 64
robustness and reliability [6]. 65
Despite these challenges, initial progress has been made in the application of deep 66
reinforcement learning (DRL) techniques, especially deep Q networks (DQNs) [7] and 67
its variants, in fi nancial transactions. Researchers began to try to use DQN technology 68
to model and analyze the fi nancial market, and designed intelligent algorithms to auto- 69
matically fi nd and use trading opportunities in the market. The goal of these algorithms 70
is to intelligently maximize return on investment and Sharpe ratios in a volatile market 71
environment, thereby improving the overall performance of trading strategies [8]. 72
DQN and its variants make decisions directly from high-dimensional raw data by 73
learning optimal action strategies, which provides a new perspective for analysis and 74
forecasting in fi nancial markets. Especially when working with fi nancial time series data, 75
DQN is able to identify and exploit hidden patterns and trends in historical information to 76
predict future market dynamics. Compared with traditional fi nancial analysis methods, 77
this model-based learning method can more fl exibly adapt to market changes and provide 78
more accurate and stable trading decision support. 79
The main contribution of this paper is to propose an innovative Deep Reinforcement 80
Learning (DRL) algorithm, the Composite Multi-Role Deep Q Network (CMR-DQN), which 81
is designed to improve the accuracy and effi ciency of stock market forecasting and decision- 82
making. By combining the Discrete Wavelet Transform (DWT), the Temporal Convolutional 83
Network (TCN), and the GRU-LSTM-Attention mechanism, CMR-DQN optimizes the 84
processing power of traditional deep neural networks (DNNs) in many aspects, especially 85
in the ability to parse complex fi nancial time series data. This algorithmic framework not 86
only enhances the performance of the model in capturing long-term dependencies and 87
multi-scale features, but also effectively improves the quality of stock market decision- 88
making through advanced learning strategies. 89
In summary, our research makes the following major contributions: 90
First, the CMR-DQN model proposed in this paper innovatively solves the limitations 91Version July 10, 2024 submitted to Journal Not Specifi ed 3 of 18
of traditional deep neural networks in processing complex fi nancial time series data by 92
combining Discrete Wavelet Transform (DWT), Time Convolutional Network (TCN) and 93
GRU-LSTM-Attention mechanism. This combination not only demonstrates excellent capa- 94
bilities in multi-scale analysis, but also signifi cantly improves the accuracy of stock market 95
forecasts and the quality of decision-making through fi ne feature extraction and effi cient 96
use of key information points. 97
Second, CMR-DQN introduces the Rainbow DQN reinforcement learning strategy to 98
achieve self-learning and strategy optimization in a simulated stock trading environment. 99
The adoption of this strategy not only enhances the model’s adaptability to market dynam- 100
ics, but also ensures that the model can continue to optimize trading decisions in the face 101
of changing market conditions to achieve long-term profi tability and risk control. 102
Third,in addition to the fundamental indicators such as Close, Open, High, Low, and 103
Volume, our CMR-DQN algorithm incorporates a diverse set of technical indicators in- 104
cluding RSI, ROC, CCI, and MACD. Notably, our innovation extends to the utilization of 105
EXPMA and VMACD indicators, which enrich the depth and sophistication of our trading 106
model 107
Finally, this study also incorporates six stocks from both international and domestic 108
markets and thoroughly demonstrates the superior performance of CMR-DQN through 109
comparisons with traditional strategies as well as various strategies based on deep rein- 110
forcement learning. 111
1. Related Work 112
The core driving force for the exploration and application of algorithmic trading lies 113
in the use of advanced computing technology to solve the limitations of traditional trading 114
methods. In the practice of algorithmic trading, rule-based methods and machine learning- 115
based strategies are the two main directions. Rule-based algorithmic trading relies on an 116
in-depth understanding of market rules, through the setting of a fi xed set of trading rules 117
to automate buying and selling operations. The advantage of this approach is that it is 118
straightforward and easy to implement, but at the same time, it is also infl exible and may 119
not be able to adapt in time in the face of rapid changes in the market. 120
1.1. Deep reinforcement learning in trading 121
In the fi eld of fi nancial transactions, the revolutionary application of deep reinforce- 122
ment learning (DRL) is attracting increasing attention from industry and academia. DRL 123
combines the data processing capabilities of deep learning with the decision-making ca- 124
pabilities of reinforcement learning, showing great potential to solve complex fi nancial 125
decision-making problems. Since the fi rst successful application of the Deep Q Network 126
(DQN) algorithm by Mnih and Silver and the performance of human level or even beyond 127
humans in multiple tasks, DRL technology has made remarkable progress in the research 128
and application of fi nancial trading strategies[ 9,10] . In addition to the above-mentioned 129
work, DRL technology is also being applied to a wider range of fi nancial transactions. For 130
example, Li demonstrated the ability of DRLs to manage market shocks and transaction 131
costs by effectively solving the optimal execution problem in high-frequency trading by 132
employing an improved DRL model [11]. At the same time, Ma et al. proposed a paral- 133
lelized deep reinforcement learning (DRL) framework for stock trading, which combines 134
multiple neural network models and reinforcement learning strategies to simultaneously 135
analyze real-time market data and long-term historical trends [12]. In addition, DRLs have 136
also shown their unique value in risk control and management. 137
Liu and his team have developed a new algorithmic trading strategy based on deep 138
reinforcement learning (DRL), which combines fi nancial market data, news sentiment anal- 139
ysis, and technical indicators. They employ an actor-critic approach and a duel network 140
architecture that effectively captures the key features of time series data and candlestick 141
charts with the aim of optimizing the trading decision-making process [13]. Similarly, Fan 142
and Peng further explored the application of DRL in fi nancial stock investment manage- 143Version July 10, 2024 submitted to Journal Not Specifi ed 4 of 18
ment by combining the methods of DQN and LSTM networks, demonstrating its potential 144
to improve investment effi ciency and decision-making quality[ 14]. The DADE-DQN model 145
proposed by Huang demonstrates the effectiveness of DRL in improving the performance 146
of stock trading strategies by introducing dual-action selection and dual-environment mech- 147
anisms, as well as combining long short-term memory (LSTM) and attention mechanisms 148
[15]. This innovative DRL application not only optimizes the trading decision-making 149
process, but also improves the overall return of the strategy. 150
1.2. Optimization of deep reinforcement learning algorithms 151
Advancements in DRL technology are a key driver of continuous innovation in the 152
fi nancial trading space. With the continuous evolution of algorithms, learning mechanisms, 153
and network structures, DRLs provide new perspectives and tools for solving complex 154
fi nancial decision-making problems. For example, the dual DQN algorithm proposed by 155
Hasselt et al. solves the problem of possible overestimation of the action value function in 156
the traditional Q learning process, and signifi cantly improves the accuracy of estimation 157
and the performance of the model by introducing the dual learning goal [16]. In addition, 158
the preferential experience replay technique introduced by Schaul et al. improved the 159
learning effi ciency and stability of the model in complex environments by assigninga 160
higher resampling priority to important learning experiences [17]. 161
The introduction of the dueling network structure is an important innovation in the 162
structure of the DRL model. The work by Hessel et al. demonstrated unprecedented data 163
effi ciency and fi nal performance on the Atari 2600 benchmark by extending and combining 164
six independent DQN algorithms into an ensemble model, the Rainbow DQN [18]. Wang et 165
al. optimized the model’s estimation of the value of behavior in different states by dividing 166
the network into two independent paths representing the state-value function and the 167
state-action dominance function, further enhancing the fi ne-grained control in the decision- 168
making [19]. These technological advances not only improve the theoretical robustness 169
of the model, but also provide practical benefi ts for practical applications, such as the 170
development of fi nancial trading strategies. In addition, the Deep Recurrent Q Network 171
(DRQN) enables the DRL model to better process and memorize sequence information by 172
introducing recurrent neural networks (RNNs), especially LSTM units, which is particularly 173
important for fi nancial transaction scenarios rich in time series data[ 20]. Chen researched 174
a deep reinforcement learning algorithm based on Duel Deep Cyclic Q Network (Dueling 175
DRQN) to improve the accuracy of stock price forecasting by combining the advantages 176
of the recurrent neural network model in processing sequence data with the ability of 177
the Q algorithm to adaptively learn stock price patterns and trends [21]. Ye and Schuller 178
proposed a trading model that mimicks human trader behavior, which aims to improve the 179
co nsistency of machine trading algorithms with human behavior by combining supervised 180
learning, single-step and multi-step Q learning, and imitation learning, and using discrete 181
wavelet transform (DWT) to process data, and verify its performance beyond the baseline 182
model for a variety of U.S. stocks through actual backtesting [22]. Shah employ a range 183
of integrated methods, including rainbow DQN, GRU, and LSTM, for real-time stock 184
market forecasting and trading signal generation. They measure the effectiveness of 185
these technologies by looking at forecast accuracy and return on investment (ROI). When 186
comparing deep learning models, especially GRU and LSTM, rainbow DQNs have been 187
found to perform best among models such as rainbow DQN, DQN, and dual DQN [23]. 188
1.3. Stock Data Denoising 189
In conventional methods, Fourier Transform (FT) is commonly used for analyzing the 190
frequency domain information of signals [24]. However, it is not suitable for directly apply- 191
ing to non-stationary signals, such as stock price data [25]. Wavelet Transform (WT), on the 192
other hand, overcomes the drawbacks of FT in analyzing and reconstructing non-stationary 193
signals [26]. Researchers have successfully applied the Discrete Wavelet Transform (DWT) 194
method to stock prediction tasks, achieving promising results [27]. Wang introduced a 195Version July 10, 2024 submitted to Journal Not Specifi ed 5 of 18
neural network architecture named Multiscale Long Short-Term Memory Network (mL- 196
STM), which integrates wavelet transform (WT) and Long Short-Term Memory (LSTM). 197
The mLSTM incorporates the DWT method into the deep learning framework to obtain 198
multiscale subsequences of different frequencies, which are independently modeled by 199
LSTM. [28]. 200
In addition, by introducing methods such as dilated convolutions and residual con- 201
nections into Convolutional Neural Networks (CNNs), Temporal Convolutional Networks 202
(TCNs) have gained popularity in sequence modeling tasks. Bai, Kolter, and Koltun 203
demonstrated that TCNs have surpassed the performance of RNNs in a range of sequence 204
modeling tasks [29]. Gone proposed a Multi-Stage TCN-LSTM Hybrid Attentive Network 205
(MSHAN) for stock trend prediction, leveraging historical stock data, technical indicators, 206
and weighted social media information to outperform baselines and enhance trading profi ts 207
through accurate directional movement prediction of daily stock prices [30]. Dechun Wen 208
introduced MWDINet, a stock price prediction framework integrating wavelet decomposi- 209
tion, Hull Moving Average, and autocorrelation correction modules. Experimental results 210
demonstrate its superiority over existing models, showcasing its potential for accurate 211
stock price forecasting [31]. 212
2. Models and methods 213
In the fi nancial markets, traders face the daunting task of reducing risk while simul- 214
taneously increasing profi tability. Inspired by the DQN algorithm, we propose a novel 215
approach called CMR-DQN (Compound Multiscale Representation Deep Q-Network) 216
aimed at addressing this challenge. 217
CMR-DQN is an extension of the DQN algorithm specifi cally designed for fi nancial 218
trading problems. Typically, DQN-based reinforcement learning agents require a large 219
number of training events to learn the optimal strategy and maximize cumulative rewards. 220
However, in stock trading, due to high data noise, randomness, and limited real stock 221
data, this may lead to overfi tting on the test set and degraded performance. Our goal is 222
to overcome overfi tting and effectively explore the best strategies with limited data. The 223
model fl owchart is illustrated in Figure 1.
Figure 1. The architecture of CMR-DQN framework.
224
2.1. DWT-TCN 225
DWT-TCN (Denoising Wavelet Transform Temporal Convolutional Network) is a 226
deep learning model designed specifi cally for denoising tasks in time series data. It inte- 227
grates the advantages of Discrete Wavelet Transform (DWT) and Temporal Convolutional 228
Network (TCN). Its core objective is to effectively capture both long-term and short-term 229
dependencies in time series data to enhance denoising performance. 230Version July 10, 2024 submitted to Journal Not Specifi ed 6 of 18
The processing pipeline of the model involves three key steps. Firstly, the input time 231
series data undergoes processing in the wavelet transform layer, where it is decomposed 232
into approximation coeffi cients and detail coeffi cients of different scales to obtain feature 233
representations. Subsequently, these feature representations are passed to the temporal 234
convolutional network for further processing. The temporal convolutional network consists 235
of stacked convolutional layers and activation functions, aimed at extracting long-term and 236
short-term dependencies in the time series data. Finally, the output of TCN is processed 237
through a global average pooling layer to aggregate features, generating the fi nal denoised 238
time series data. The entire process aims to effi ciently denoise time series data by combining 239
the processing capabilities of wavelet transform and temporal convolutional networks. The 240
specifi c process is illustrated in Figure 2.
Figure 2. The structural diagram of DWT-TCN
241
2.1.1. Discrete Wavelet Transform (DWT) 242
The Discrete Wavelet Transform (DWT) is a commonly used technique in signal 243
processing [32], which provides a method for multiscale analysis of signals by decomposing 244
them into different scales of approximation parts Ajand detail parts Dj. In the DWT 245
layer, the input time series data x(t) is decomposed into different levels of approximation 246
coeffi cientsA j+1 and detail coeffi cientsD j+1, expressed as Equation 1: 247
Aj+1, Dj+1
= DWT(x(t)) (1)
These coeffi cients represent the time-frequency features of the signal at different scales. 248
Through this decomposition, we can better understand the patterns of change in time 249
series data at different time scales, thus more effectively capturing the data’s characteristic 250
information. 251
2.1.2. Temporal Convolutional Network(TCN) 252
TCN represents a signifi cant advancement in deep learning-based forecasting, build- 253
ing upon enhancements to CNN architectures. While CNNs fi nd widespread use in image 254
processing, adapting them effectively for time series prediction necessitated tailored re- 255
fi nements. Through the integration of specifi c requirements and features of time series 256
prediction, researchers successfully achieved superior results with TCN [33]. By incorporat- 257
ing dilated convolutions alongside causal convolutions, TCN expands the receptive fi eld 258
and refi nes accuracy. Its structural diagram is depicted in Figure 2. 259
2.2. Rainbow DQN 260
Rainbow DQN (Rainbow Deep Q-Network) stands as an innovative advancement 261
in deep reinforcement learning, aiming to overcome the limitations of standard Deep Q- 262Version July 10, 2024 submitted to Journal Not Specifi ed 7 of 18
Network (DQN) algorithms [34]. It integrates various enhancements such as prioritized 263
experience replay, double Q-learning, dueling architecture, multi-step learning, distri- 264
butional RL, and noisy networks, offering improved sample effi ciency, faster learning, 265
and enhanced generalization capabilities. Rainbow DQN is particularly notable for its 266
application in predicting buy/sell signals for stock prices using integrated deep learning 267
techniques. This comprehensive approach makes Rainbow DQN a potent tool for address- 268
ing a wide range of reinforcement learning challenges. The specifi c working diagram of 269
Rainbow DQN is shown in Figure 3.
Figure 3. Working Diagram of Rainbow DQN
270
Prioritized Experience Replay Buffer Tom proposed using prioritized experience 271
replay buffer instead of the ordinary replay buffer, assigning higher priority to important 272
experiences based on the TD error (?) [17], The Temporal Difference Error, as in Equation 273
2, measures the discrepancy between predicted and actual outcomes in reinforcement 274
learning. 275
?t = Rt+1
+ ?maxa ?q(St+1, a, w) ? ?q(St, At, w) (2)
The priority of pt the experience is considered to be its magnitude and is stored in the 276
buffer, as shown in Equation 3. 277
pt =| ?t | +e (3)
Dueling Architecture Sampling experience tuples with probabilities P (i) during batch 278
creation operates akin to the SoftMax function, facilitating a reduction in the frequency of 279
batch updates. To address experience tuples that are starved for attention, reintroducing 280
uniform random sampling with a constant value ? can be benefi cial. Adjustments to the 281
update rule are necessary to counteract biases stemming from the uneven distribution of 282
priority values, as shown in Equation 4. 283
P (i)
= pi
a
/( ? k pk
a
) (4)
Dueling Networks introduced a novel approach utilizing two separate streams in 284
the neural network architecture [19]. One stream, known as the value stream (V(s)), is 285
dedicated to directly estimating the value function for states. This is particularly effective 286
because states often exhibit minimal variation. The other stream, termed the advantage 287
stream (A(s, a)), focuses on estimating the advantage for each action, thereby capturing 288
the differences in action outcomes within each state. By aggregating the outputs from both 289
streams, the model can derive action values (Q(s, a)) effectively. This innovative framework 290
enhances the network’s ability to understand and represent the nuanced relationships 291
between states and actions. The architecture and formulas are represented as shown in 292
Figure 4 and Equation 5. 293
Q(s, a) = V(s) + {A(s, a) ? 1/|A|[ ? a A(s, a)]} (5)Version July 10, 2024 submitted to Journal Not Specifi ed 8 of 18
Figure 4. Dueling Architecture Network
294
Double DQN The main difference between Double DQN and traditional DQN lies in 295
how they calculate the target Q-values [35]. Compared to standard DQN, Double DQN is 296
better equipped to overcome the problem of overestimation present in DQN. This is because 297
DQN relies solely on the current network when selecting the optimal action, which can 298
lead to an overestimation of Q-values, thereby affecting training effectiveness and stability. 299
In contrast, Double DQN addresses this issue by utilizing the target network to evaluate 300
the Q-values of the optimal actions, effectively reducing the impact of overestimation and 301
enhancing training stability and effectiveness. Specifi cally, the formula for calculating the 302
target Q-values is as shown in Equation 6. 303
Yt = Rt+1
+ ?Q
?
' (St+1, arg max
a
Q
?
(St+1, a)) (6)
where Yt represents the target Q-value, Rt+1 is the reward from the environment, ? is 304
the discount factor, St+1 is the next state, Q
?
' is the output of the target network, Q
? is 305
the output of the current network, and arg maxa Q
?
(St+1, a) denotes selecting the action 306
with the highest Q-value according to the current network. This approach helps reduce 307
overestimation, enhancing learning stability and making the algorithm more robust. 308
Distributional RL Distributional Reinforcement Learning (Distributional RL) is a 309
class of value-based Reinforcement Learning (RL) algorithms. Classical value-based RL 310
methods attempt to model the cumulative return using expected values, represented 311
by value functions V(x) or action-value functions Q(x, a), However, in this modeling 312
process, complete distributional information is largely lost. As illustrated in Equation 7, 313
Value distributional reinforcement learning aims to address this issue by modeling the 314
distribution Z(x, a) of the random variable representing the cumulative return, rather 315
than only modeling its expectation [36]. This approach provides a more comprehensive 316
understanding of uncertainty and variability in the RL problem, enabling agents to make 317
more informed decisions. 318
Q(xt, at) = E Z(xt, at) = E
" ?
?
i=1
?t+iR(xt+i, at+i
)
#
(7)
Noisy Nets In Rainbow DQN, the NoisyNets module introduces noise parameters 319
into the weights and biases of the neural network to increase its exploration and stability 320
[37]. NoisyNets parameterizes noise to introduce randomness into the network, aiding in 321
better exploration of the state space during training and enhancing convergence speed and 322
generalization capability Specifi cally, for a linear layer with inputs x and outputs y , and 323
weights W and biases b NoisyNets add parameterized noise to these weights and biases. 324Version July 10, 2024 submitted to Journal Not Specifi ed 9 of 18
The noise parameters are denoted by µ ? where ? is a zero-mean noise vector with a fi xed 325
statistical distribution. The formal representation is as shown in Equation 8. 326
y
def
= (µW
+ ? W
? ?W
)x + µb
+ ? b
? ?b (8)
Here µW and µb represent the mean parameters for the weights and biases respectively, 327
? W and ? b denote the standard deviation parameters, ?W and ?b re noise vectors with 328
zero mean and fi xed statistical properties, and ? represents element-wise multiplication. 329
By adjusting µ and ?, the distribution and intensity of the noise can be controlled, thus 330
affecting the exploration and generalization capabilities of the network. 331
Multi-Step Q-Learning Multi-Step Q-Learning introduces Multi-Step TD targets, 332
which accumulate rewards from multiple future time steps to more accurately estimate the 333
long-term value of state-action pairs [38]. These targets cover rewards from the current time 334
step t up to t+n and use a discount factor ? to control the importance of future rewards. 335
Based on these targets, Multi-Step Q-Learning updates Q-values to more accurately refl ect 336
the infl uence of long-term rewards. The update rule is as shown in Equation 9: 337
Q(St, At) < Q(St, At) + ?[Gt:t+n ? Q(St, At)] (9)
Q(St, At) represents the estimated Q-value after taking action At in state st, ? is the learning 338
rate controlling the magnitude of updates, and Gt:t+n is the multi-step return from time 339
step t to t + n as in Equation 10: 340
Gt:t+n = Rt+1
+ ?Rt+2 + ?
2
Rt+3 + . . . + ?
n?1
Rt+n + ?
n
Q(St+n, At+n) (10)
By employing this approach, Multi-Step Q-Learning enhances the learning effi ciency and 341
performance of agents, enabling them to better cope with the complex dynamics of their 342
environment. 343
2.3. GRU-LSTM-Attention 344
In Rainbow DQN, we use the GRU-LSTM-Attention model instead of the original DNN 345
as the Q-value estimator. This model combines GRU, LSTM, and attention mechanisms 346
to better process sequence data. It can effectively capture long-term dependencies in 347
sequence data and improve the model’s ability to understand sequence data. Within 348
the model architecture, the outputs of the GRU layer serve as inputs to the LSTM layer, 349
and the outputs and fi nal state of the LSTM layer are passed to the attention mechanism. 350
Specifi cally, the output ht from the GRU layer, representing the hidden state at the current 351
time step, is used as input to the LSTM layer. Conversely, the outputs and fi nal state hn 352
from the LSTM layer are utilized as inputs to the attention mechanism, where hn denotes 353
the hidden state at the last time step. 354
For GRU, the inputs xt and the previous hidden state ht?1 undergo a series of opera- 355
tions to produce the current hidden state ht, as shown in Equation 11: 356
rt = ?(Wr · [ht?1, xt] + br)
zt = ?(Wz · [ht?1, xt] + bz)
?ht = tanh(Wh
· [rt · ht?1, xt] + bh
)
ht = (1 ? zt) · ht?1
+ zt · ?ht
(11)Version July 10, 2024 submitted to Journal Not Specifi ed 10 of 18
For LSTM, the inputs xt and the previous hidden state ht?1 are processed to produce the 357
current hidden state ht and cell state Ct, as shown in Equation 12: 358
ft = ?(Wf
· [ht?1, xt] + bf
)
it = ?(Wi
· [ht?1, xt] + bi
)
?Ct = tanh(WC
· [ht?1, xt] + bC
)
Ct = ft · Ct?1
+ it · ?Ct
ot = ?(Wo · [ht?1, xt] + bo)
ht = ot · tanh(Ct)
(12)
The LSTM outputs and fi nal state hn are passed to the attention mechanism. The attention 359
mechanism calculates attention scores eti between each LSTM output hi and hn. These 360
scores are normalized to obtain attention weights ati, representing the importance of each 361
LSTM output . Finally, the context vector ct at time step t is computed as a weighted sum 362
of the LSTM outputs, using the attention weights ati, as show in Equation 13: 363
eti
= score(hi, hn)
?ti
=
exp(eti
)
? T
j=1
exp(etj
)
ct =
T
?
i=1
?ti
· hi
(13)
By embedding the GRU-LSTM-Attention model into Rainbow DQN, we leverage its 364
ability to capture long-term dependencies and dynamically attend to important parts of 365
the sequence, thereby enhancing the performance and effectiveness of Rainbow DQN. 366
2.4. CMR-DQN Training 367
In summary, Algorithm 1 presents the pseudocode of the entire CMR-DQN algorithm, 368
which forms the foundation of our proposed trading strategy. Throughout the trading 369
process, this strategy undergoes continual updates based on daily observations, ensuring 370
ongoing learning and refi nement of the trading approach. 371
3. Experiments and Results 372
In this section, we will provide a detailed overview of establishing the environment, 373
setting up and executing experiments. We begin by exploring the process of selecting the 374
dataset and associated features. Following this, we delve into a detailed discussion of 375
experiment settings. Finally, we analyze and discuss the experimental results, evaluating 376
the effectiveness of the proposed model across six different stock index datasets. 377
3.1. Datasets 378
To comprehensively assess the performance and robustness of our model, we have 379
selected a range of stock indices from different regions and economic backgrounds as our 380
experimental subjects. These indices include the Dow Jones Industrial Average (DJI), the 381
CAC 40 Index (FCHI) from France, the Nasdaq Composite Index (IXIC), the Nikkei 225 382
(N225), as well as two Chinese stocks: Ping An Bank (SZ000001, abbreviated as PA) and 383
Ziguang Guowei Co., Ltd. (SZ000938, abbreviated as ZG). Our dataset spans from July 1, 384
2011, to December 20, 2021. 385
3.2. Trading Experimental Setup 386
The trading experiment setup includes the following key components: 387
1. Processing Initialization: We conducted normalization on each input variable to en- 388
sure consistent scaling and prevent gradient explosions. This normalization resulted 389Version July 10, 2024 submitted to Journal Not Specifi ed 11 of 18
Algorithm 1 CMR-DQN Algorithm
Data:Time series data denoised using DWT; Each trading signal represents one of three
possible actions
Initialize CMR-DQN network parameters ? and target network parameters ?target
Initialize replay buffer D
Initialize exploration rate ?
Initialize data and other parameters
for episode = 1 to M do
Initialize state s
for time step t = 1 to Tmax do
Choose action a based on current state s and exploration rate ?
if random number < exploration rate ? then
Randomly choose action a
else
Choose action a = argmax(Q(s, a; ?))
end if
Execute action a, observe reward r and new state s
'
Store (s, a, r, s
'
) tuple in replay buffer D
Sample minibatch of transitions (sj, aj, rj, s
'
j
) from replay buffer D
for each sample transition (sj, aj, rj, s
'
j
) do
if state s
'
j
is terminal state then
Set target value yj
= rj
else
Set target value yj
= rj
+ ? · max(Q(s
'
j
, a
'
; ?target))
end if
end for
Compute loss L and update CMR-DQN network parameters ? via gradient
descent: L =
1
N ? (yj
? Q(sj, aj; ?))2
if the current time step is the time to update the target network then
Update target network parameters ?target = ?
end if
end for
end forVersion July 10, 2024 submitted to Journal Not Specifi ed 12 of 18
in a mean of 0 and a standard deviation of 1 for the normalized data, thereby en- 390
hancing stability during the training process. In the CMR-DQN framework, model 391
initialization is a critical step to ensure network weights start training from a rea- 392
sonable point. Proper initialization facilitates faster convergence and helps avoid 393
undesirable local optima; 394
2. Evaluation metrics: The performance metrics for the trading experiment include 395
several key indicators. The Average Annualized Return Rate (AAR) measures the 396
average annual return on investment. The Maximum Drawdown (MDD) evaluates the 397
largest peak-to-trough decline in investment value over a specifi c period. The Sharpe 398
Ratio (SP) quantifi es the risk-adjusted return of the investment strategy, refl ecting its 399
effi ciency. The Cumulative Return (CR) represents the total return on investment over 400
the entire trading period; 401
3. Comparison Experiment: To objectively evaluate the strengths and weaknesses of 402
the CMR-DQN algorithm, we set the initial capital to 200,000 and compared it with 403
other trading methods (such as B&H and DQN). Additionally, by comparing it with 404
Rainbow DQN and CMR-DQN without the Rainbow components (i.e., DQN-GLA), 405
we conducted an ablation study to further validate its performance. 406
During the training process of our CMR-DQN model, we utilized graphical represen- 407
tations to visualize different functions. Figure 5 illustrates both the variation of cumulative 408
rewards and the fl uctuation of the loss function during training. This refl ects the model’s 409
progress in learning and optimizing its decision-making strategies, as well as its perfor- 410
mance in parameter optimization and data fi tting, across six distinct datasets. 411
Observing these two plots enables us to simultaneously understand the model’s 412
performance in reward acquisition and parameter optimization, while also allowing for 413
a comparative analysis of its performance across diverse datasets. These visualizations 414
provide comprehensive monitoring of the training process, aiding in the assessment of 415
the model’s performance across various environments and guiding further adjustments in 416
training strategies. 417
(a) PA (b) ZG (c) DJI
(d) FCHI (e) IXIC (f) N225
Figure 5. The accumulation of rewards and the variation trend of the loss function during the training
process of the CMR-DQN model on six datasets.
By observing the results from these two charts, it is evident that the CMR-DQN 418
model has converged in both cumulative reward and loss functions, indicating stable and 419
promising performance during training. This convergence not only showcases the model’s 420
capability in exploring and exploiting potential returns in the stock market but also refl ects 421
its fl exibility in dynamically adjusting and optimizing trading strategies. The model can 422
adeptly adapt to evolving market conditions and adjust decisions promptly to maximize 423
portfolio effi ciency. 424
3.3. Feature Selection 425
In the CMR-DQN algorithm, we utilize a variety of features for stock prediction. These 426
features encompass fundamental stock indicators such as closing price, opening price, 427Version July 10, 2024 submitted to Journal Not Specifi ed 13 of 18
highest price, lowest price, and trading volume, along with a series of technical indicators 428
including Relative Strength Index (RSI), Rate of Change (ROC), Commodity Channel Index 429
(CCI), and Moving Average Convergence Divergence (MACD). Particularly noteworthy is 430
our innovative inclusion of Exponential Moving Average (EXPMA) and Volume-weighted 431
Moving Average Convergence Divergence (VMACD) in the feature selection process. 432
3.3.1. Exponential Weighted Moving Average (EXPMA) 433
The Exponential Moving Average (EXPMA), also known as the Daily Average Index, 434
is a variant of the moving average line. It smooths out the fast and slow moving average 435
lines at different speeds and assesses market buy and sell signals based on the crossover 436
of these two lines. Compared to the traditional Moving Average (MA) indicator, EXPMA 437
places more emphasis on daily price changes. While the traditional MA relies on the 438
arithmetic average of price fl uctuations N days ago, infl uenced by the highs and lows ofN 439
days prior, it may not immediately refl ect real-time price changes. EXPMA, on the other 440
hand, is a trend-based indicator that gives more weight to the daily price performance, 441
thereby overcoming lag to some extent, making it an effective medium-term indicator. The 442
calculation formula for EXPMA is as shown in Equation 14: 443
EXPMA = ? ? Close + (1 ? ?) ? EXPMAprev (14)
Here,Close represents the closing price of the day, ? s the smoothing coeffi cient typically 444
ranging from 0 to 1 and EXPMAprev denotes the EXPMA value of the previous day. The 445
primary purpose of EXPMA is to identify buy and sell opportunities and predict future 446
price trends. However, it’s essential to note that while EXPMA refl ects daily price changes, 447
it may not accurately predict price peaks and bottoms. 448
3.3.2. Volume-weighted Moving Average Convergence Divergence (VMACD) 449
In fi nancial market technical analysis, the Moving Average Convergence Divergence 450
(MACD) indicator is commonly used. However, MACD typically relies solely on price data, 451
particularly the closing price, without considering volume data. To address this limitation 452
and incorporate volume data into the analysis, the Volume-weighted Moving Average 453
Convergence and Divergence (VMACD) indicator is introduced. The calculation formula 454
for VMACD is as shown in Equation 15: 455
DIFF : EMA(VOL, SHORT) ? EMA(VOL, LONG)
DEA : EMA(DIFF, M)
VMACD : DIFF ? DEA
(15)
Here, DIFF represents the difference between two volume Exponential Moving Averages 456
(EMAs), one with a shorter period (SHORT) and the other with a longer period (LONG). 457
DEA is the EMA of the DIFF values over a specifi ed period (M). Finally, the VMACD line is 458
derived by subtracting DEA from DIFF. 459
In practical applications, similar to MACD, VMACD signals potential market trends. 460
However, VMACD may anticipate market movements a few days earlier than MACD. 461
Specifi cally, when trading volume signifi cantly deviates from its normal level and prices 462
increase, it often signals the beginning of a market reversal. Crossing the zero line is also 463
considered a signal of a market reversal. 464
It’s essential to note that while VMACD provides valuable insights into market dy- 465
namics, it should be used in conjunction with price-based indicators for comprehensive 466
analysis. Additionally, VMACD is commonly utilized alongside other indicators like KD 467
and KDJ for enhanced decision-making in trading strategies. 468
3.4. Experimental Results 469
This section presents a comparison between the proposed CMR-DQN framework and 470
fi ve evaluation methods. The experimental results on six stock index datasets are shown 471Version July 10, 2024 submitted to Journal Not Specifi ed 14 of 18
in Table 1. Trading performance is assessed using four performance metrics: Cumulative 472
Return (CR), Sharpe Ratio (SR), Annualized Return (AR), and Maximum Drawdown 473
(MDD). 474
Table 1 highlights the performance of different trading methods across six assets. 475
Among the listed metrics, the CMR-DQN strategy achieved the highest scores, indicating 476
its superior performance. Compared to other strategies, CMR-DQN attained signifi cantly 477
higher cumulative returns and demonstrated the smallest maximum drawdown, under- 478
scoring its robust profi tability and risk mitigation across the six datasets. For instance, on 479
the IXIC dataset, the CMR-DQN strategy achieved a cumulative return of 49.15%, outper- 480
forming all other methods. Its Sharpe Ratio was 0.99, refl ecting excellent risk-adjusted 481
returns. Additionally, the CMR-DQN strategy had the lowest maximum drawdown of 482
16.35% on the IXIC dataset, further highlighting its effectiveness. 483
Similar observations can be made for the other assets listed in the table. In terms 484
of cumulative returns, Sharpe Ratio, and annualized returns, CMR-DQN consistently 485
outperformed benchmark methods, demonstrating its superior performance. These re- 486
sults validate the proposed CMR-DQN framework’s effectiveness in generating profi table 487
trading strategies with low risk.
Table 1. The trading performance of six assets across different models
Asset Metric B&H DQN
Rainbow
DQN
DQN-GLA CMR-DQN
PA
CR[%] -19.07 -24.46 8.16 14.62 28.22
SR -0.22 -0.28 0.11 0.16 0.29
AR[%] -33.24 43.29 13.76 25.15 50.48
MDD[%] 56.42 59.97 33.98 29.84 20.92
ZG
CR[%] -15.58 12.13 0.63 23.23 31.26
SR -0.16 0.14 0.01 0.27 0.33
AR[%] -27.52 18.03 1.04 40.97 48.30
MDD[%] 52.35 31.97 39.70 23.86 18.73
DJI
CR[%] 29.03 5.54 45.05 45.09 52.32
SR 0.30 0.09 0.48 0.49 0.69
AR[%] 51.98 9.38 84.28 84.39 99.73
MDD[%] 20.70 36.02 12.85 12.79 7.91
FCHI
CR[%] -1.54 45.03 42.70 53.16 55.23
SR -0.02 0.48 0.46 0.72 0.73
AR[%] -2.03 84.29 79.42 101.5 106.04
MDD[%] 41.03 12.88 13.61 6.52 6.18
IXIC
CR[%] 4.82 -0.62 17.54 8.81 20.37
SR 0.03 -0.01 0.21 0.11 0.24
AR[%] 8.17 -1.02 30.43 14.89 35.64
MDD[%] 37.34 40.71 27.58 34.06 25.59
N225
CR[%] 11.52 35.86 36.32 37.02 41.96
SR 0.13 0.35 0.38 0.41 0.45
AR[%] 19.64 65.50 66.42 67.83 77.89
MDD[%] 32.05 17.42 16.33 15.78 14.02
488
Figure 6 displays the cumulative asset growth curves for various strategies, including 489
B&H, DQN, Rainbow DQN, DQN-GLA, and the newly proposed CMR-DQN. It is evident 490
from the graph that the performance of the CMR-DQN strategy surpasses that of the 491
others. CMR-DQN not only signifi cantly reduces loss risk but also achieves higher returns. 492
Moreover, under the CMR-DQN strategy, total asset growth is smoother compared to the 493
benchmark strategy. 494Version July 10, 2024 submitted to Journal Not Specifi ed 15 of 18
The performance shown in Figure 6 aligns with the performance metrics in Table 7. 495
Across different asset categories, CMR-DQN consistently outperforms other benchmark 496
methods, demonstrating its effectiveness in generating profi table trading strategies. This 497
strategy not only avoids substantial losses but also delivers additional returns compared to 498
benchmark and alternative methods. 499
These fi ndings further validate the superiority of the CMR-DQN framework in cre- 500
ating profi table trading strategies. By successfully balancing risk and return, CMR-DQN 501
offers investors a more stable and potentially more profi table investment approach than 502
traditional strategies and other deep reinforcement learning methods.
(a) 001 (b) 938
(c) DJI (d) FCHI
(e) IXIC (f) N225
Figure 6. Results of Different Models on Six Datasets
503
3.5. Discussion 504
This paper introduces and explores the CMR-DQN algorithm, a DQN extension 505
designed to enhance stock trading strategies. Through a comprehensive performance 506
evaluation of different trading methods across six assets, the results in Table 7 clearly 507
demonstrate that the CMR-DQN strategy consistently excels across all assets. Notably, 508
the CMR-DQN strategy consistently achieved the highest cumulative returns among all 509
comparative methods, highlighting its advantages in optimizing trading decisions and 510
capturing market opportunities. 511
The CMR-DQN strategy excels not only in returns but also in risk management. By 512
maintaining consistently low maximum drawdowns, the CMR-DQN strategy effectively 513
protects investors’ capital amid market volatility while achieving stable returns under con- 514
trolled risk. This capability is repeatedly validated in this study, supported by mathematical 515
models and empirical data. Specifi cally, the CMR-DQN strategy showcased its profi tability 516
and risk mitigation abilities across six datasets, which holds signifi cant implications for 517
policymakers, traders, and fi nancial experts. 518
Additionally, we compared multiple deep reinforcement learning (DRL) strategies 519
with traditional strategies (such as B&H, DQN, Rainbow DQN, and DQN-GLA) under the 520
same conditions, further affi rming the superiority of the CMR-DQN algorithm. Regardless 521
of the trading objectives, the CMR-DQN algorithm consistently outperformed benchmark 522
methods, achieving notable cumulative returns and Sharpe ratios. For instance, i n the 523Version July 10, 2024 submitted to Journal Not Specifi ed 16 of 18
FCHI dataset, the CMR-DQN strategy achieved a cumulative return of 55.23% and a Sharpe 524
ratio of 0.73, highlighting its ability to generate profi table trading strategies while effectively 525
managing risk. 526
In conclusion, the CMR-DQN algorithm has demonstrated its effectiveness and supe- 527
riority in stock trading strategies through empirical research. The algorithm not only excels 528
in cumulative returns and risk management but also showcases strong adaptability across 529
various market conditions. These results lay a solid foundation for further research and ap- 530
plication of DRL-based trading strategies, providing valuable new tools and methodologies 531
for traders and fi nancial decision-makers. 532
4. Conclusion 533
The CMR-DQN model integrates Discrete Wavelet Transform (DWT), Time Convo- 534
lutional Network (TCN), and a GRU-LSTM-Attention mechanism to enhance traditional 535
deep neural networks for complex fi nancial time series data analysis. This innovative 536
approach excels in multi-scale analysis, signifi cantly improving stock market predictions 537
and decision-making by effectively extracting features and utilizing key information points. 538
Additionally, CMR-DQN incorporates the Rainbow DQN reinforcement learning strategy, 539
optimizing simulated stock trading through self-learning and dynamic strategy optimiza- 540
tion. The model leverages various technical indicators, such as Close, Open, High, Low, 541
Volume, RSI, ROC, CCI, MACD, and introduces EXPMA and VMACD indicators for more 542
sophisticated trading decisions. 543
Comprehensive evaluations on six international and domestic stocks demonstrate 544
the superior performance of CMR-DQN in stock market prediction and trading decisions. 545
However, there is potential for improvement by diversifying feature indicators and refi ning 546
complex trading mechanisms. Future research could explore additional technical indicators, 547
market signals, and macroeconomic data to enhance the model’s sensitivity and accuracy. 548
Further optimization of trading decision mechanisms, incorporating sophisticated strategy 549
combinations and refi ned risk management models, could improve its stability and prof- 550
itability. Extending the model’s application to broader markets and asset classes like futures 551
and forex would also verify its generalizability and applicability. Continuous refi nement 552
and in-depth research could establish CMR-DQN as a vital tool for market analysis and 553
trading strategy support.

---
Role:
Act as AI expert in mathematics. 
Provide a continuous, logically coherent response without using bullet points or numbered lists.
Adjust your writing style to more looks like human. 
---
Tasks:
Write professional-grade summarization and expert recenze.
Find few advantages of proposed approach.