# A Stock Market Decision-Making Framework Based on CMR-DQN
## 3 Models and Methods
### Rainbow DQN
\[Y_{t}=R_{t+1}+\gamma Q_{\theta^{\prime}}(S_{t+1},\arg\max_{a}Q_{\theta}(S_{t+1},a)), \tag{6}\]  
where \(Y_{t}\) represents the target \(Q\)-value, \(R_{t+1}\) is the reward from the environment, \(\gamma\) is the discount factor, \(S_{t+1}\) is the next state, \(Q_{\theta^{\prime}}\) is the output of the target network, \(Q_{\theta}\) is the output of the current network, and \(\arg\max_{a}Q_{\theta}(S_{t+1},a)\) denotes selecting the action with the highest \(Q\)-value according to the current network. This approach helps reduce overestimation, enhancing learning stability and making the algorithm more robust. Figure 4: Dueling architecture network. Distributional RL: Distributional reinforcement learning (distributional RL) is a class of value-based reinforcement learning (RL) algorithms. Classical value-based RL methods attempt to model the cumulative return using expected values, represented by value functions \(V(x)\) or action-value functions \(Q(x,a)\). However, in this modeling process, complete distributional information is largely lost. As illustrated in Equation (7), value distributional reinforcement learning aims to address this issue by modeling the distribution \(Z(x,a)\) of the random variable representing the cumulative return, rather than only modeling its expectation [37]. This approach provides a more comprehensive understanding of uncertainty and variability in the RL problem, enabling agents to make more informed decisions. \[Q(x_{t},a_{t})=\mathbb{E}Z(x_{t},a_{t})=\mathbb{E}\bigg{[}\sum_{i=1}^{\infty }\gamma_{t+i}R(x_{t+i},a_{t+i})\bigg{]} \tag{7}\]  
NoisyNets: In Rainbow DQN, the NoisyNets module introduces noise parameters into the weights and biases of the neural network to increase its exploration and stability [38]. NoisyNets parameterizes noise to introduce randomness into the network, aiding in the better exploration of the state space during training and enhancing convergence speed and generalization capability Specifically, for a linear layer with inputs \(x\) and outputs \(y\), and weights \(W\) and biases \(b\), NoisyNets add parameterized noise to these weights and biases. The noise parameters are denoted by \(\mu\)\(\Sigma\), where \(\epsilon\) is a zero-mean noise vector with a fixed statistical distribution. The formal representation is as shown in Equation (8). \[y\stackrel{{\text{def}}}{{=}}\quad(\mu_{W}+\Sigma_{W}\odot \epsilon_{W})x+\mu_{b}+\Sigma_{b}\odot\epsilon_{b} \tag{8}\]  
Here, \(\mu_{W}\) and \(\mu_{b}\) represent the mean parameters for the weights and biases, respectively; \(\Sigma_{W}\) and \(\Sigma_{b}\) denote the standard deviation parameters; \(\epsilon_{W}\) and \(\epsilon_{b}\) represent noise vectors with zero mean and fixed statistical properties; and \(\odot\) represents element-wise multiplication. By adjusting \(\mu\) and \(\Sigma\), the distribution and intensity of the noise can be controlled, thus affecting the exploration and generalization capabilities of the network. Multi-Step \(Q\)-Learning: Multi-Step \(Q\)-Learning introduces multi-Step TD targets, which accumulate rewards from multiple future time steps to estimate the long-term value of state-action pairs more accurately [39]. These targets cover rewards from the current time step \(t\) up to t+n and use a discount factor \(\gamma\) to control the importance of future rewards. Based on these targets, multi-step \(Q\)-learning updates \(Q\)-values to more accurately reflect the influence of long-term rewards.