# A Stock Market Decision-Making Framework Based on CMR-DQN
## 3 Models and Methods
### Rainbow DQN  
Rainbow DQN (Rainbow Deep Q Network) stands as an innovative advancement in deep reinforcement learning, aiming to overcome the limitations of standard Deep Q Network (DQN) algorithms [18]. It integrates various enhancements such as prioritized experience replay, double Q-learning, dueling architecture, multi-step learning, distributional RL, and noisy networks, offering improved sample efficiency, faster learning, and enhanced generalization capabilities. Rainbow DQN is particularly notable for its application in predicting buy/sell signals for stock prices using integrated deep learning techniques. This comprehensive approach makes Rainbow DQN a potent tool for addressing a wide range of reinforcement learning challenges. The specific working diagram of Rainbow DQN is shown in Figure 3.  
Prioritized Experience Replay Buffer: [PERSON] proposed using prioritized experience replay buffer instead of the ordinary replay buffer, assigning higher priority to important experiences based on the TD error (\(\delta\)) [19]. The temporal difference error, as in Equation (2), measures the discrepancy between predicted and actual outcomes in reinforcement learning. \[\delta_{t}=R_{t+1}+\gamma max_{a}\hat{q}(S_{t+1},a,w)-\hat{q}(S_{t},A_{t},w) \tag{2}\]  
The priority of \(p_{t}\): the experience is considered to be its magnitude and is stored in the buffer, as shown in Equation (3). \[p_{t}=\mid\delta_{t}\mid+e \tag{3}\]  
Figure 3: Working diagram of Rainbow DQN. Dueling Architecture: Sampling experience tuples with probabilities \(P_{(i)}\) during batch creation operates akin to the SoftMax function, facilitating a reduction in the frequency of batch updates. To address experience tuples that are starved for attention, reintroducing uniform random sampling with a constant value \(\alpha\) can be beneficial. Adjustments to the update rule are necessary to counteract biases stemming from the uneven distribution of priority values, as shown in Equation (4). \[P_{(i)}={p_{i}}^{a}/(\Sigma_{k}{p_{k}}^{a}) \tag{4}\]  
Dueling networks introduced a novel approach utilizing two separate streams in the neural network architecture [21]. One stream, known as the value stream \((V(s))\), is dedicated to directly estimating the value function for states. This is particularly effective because states often exhibit minimal variation. The other stream, termed the advantage stream \((A(s,a))\), focuses on estimating the advantage for each action, thereby capturing the differences in action outcomes within each state. By aggregating the outputs from both streams, the model can derive action values \((Q(s,a))\) effectively. This innovative framework enhances the network's ability to understand and represent the nuanced relationships between states and actions. The architecture and formulae are represented as shown in Figure 4 and Equation (5). \[Q(s,a)=V(s)+\{A(s,a)-1/|A|[\Sigma_{a}A(s,a)]\} \tag{5}\]  
Double DQN: The main difference between double DQN and traditional DQN lies in how they calculate the target \(Q\)-values [36]. Compared to standard DQN, double DQN is better equipped to overcome the problem of overestimation present in DQN. This is because DQN relies solely on the current network when selecting the optimal action, which can lead to an overestimation of \(Q\)-values, thereby affecting training effectiveness and stability. In contrast, Double DQN addresses this issue by utilizing the target network to evaluate the \(Q\)-values of the optimal actions, effectively reducing the impact of overestimation and enhancing training stability and effectiveness. Specifically, the formula for calculating the target \(Q\)-values is as shown in Equation (6).