## 6 Deep Reinforcement Learning for Imbalanced Classification
### Deep Q-Network Based Imbalanced Classification Algorithm  
#### 6.3.1 Deep Q-Learning for ICMDP Subsection  
According to ICMDP, the classification policy \(\pi\), shown in (23), is considered as a function. This function receives a sample and returns the probabilities of both labels.  
\[\pi(a|s)\ =P(a_{t}=a|s_{t}=s) \tag{23}\]  
The classifier agent aims at accurately identifying as many training data samples as possible. Since the classifier agent obtains a positive reward for correctly identifying the sample, it then accomplishes its objective through maximization of the cumulative rewards \(g_{t}\), shown in (24).  
\[g_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k} \tag{24}\]  
In the RL, the function which is the Q function is used to calculate the quality of the association between the state and the action. The Q function is shown in (25).  
\[Q^{\pi}(s,a)\ =E_{\pi}[g_{t}|s_{t}=s,\ a_{t}=a] \tag{25}\]  
Based on the Bellman equation [36], the Q function is defined in (26).  
\[Q^{\pi}(s,a)\ =E_{\pi}[r_{t}+\gamma Q^{\pi}(s_{t+1},a_{t+1})|s_{t}=s,\ a_{t}=a] \tag{26}\]  
In the ICMDP, through resolving the optimal \(Q^{*}\) function, the classifier agent maximizes the cumulative rewards. The optimal classification policy \(\pi^{*}\) is defined as the greedy policy under the optimal \(Q^{*}\) function shown in (27).  
\[\pi^{*}(a|s)\ =\ \left\{\begin{array}{rl}1,&if\ a=\arg max_{a}Q^{*}(s,a)\\ 0&else\end{array}\right. \tag{27}\]  
The optimal \(Q^{*}\) function, which is shown in (28), is obtained through substituting (27) into (26).  
\[Q^{*}(s,a)\ =E_{\pi}[r_{t}+\underbrace{\gamma max}_{a}Q^{*}(s_{t+1},a_{t+1})| s_{t}=s,\ a_{t}=a] \tag{28}\]  
The Q functions are written down through a table in the low-dimensional finite state space. By contrast, the Q functions are solved based on the deep Q-learning algorithm in high-dimensional continuous state space. In the deep Q-learning algorithm, the interaction  
We replace (32) into (34), (33) into (35) and add the derivative of \(L_{+}(\theta_{k})\)) and \(L_{-}(\theta_{k})\), and we then obtain (36).  
\[\begin{split}\frac{\
abla L(\theta_{k})}{\
abla\theta_{k}}& =-2\sum_{m=1}^{D_{min}+D_{maj}}((1-t_{m})\underbrace{\gamma max}_ {a_{m}}Q(s_{m}^{\prime};a_{m}^{\prime};\theta_{k-1})\\ &\quad\quad-Q(s_{m},a_{m};\theta_{k}))\frac{\
abla Q(s_{m},a_{m}; \theta_{k})}{\
abla\theta_{k}}\\ &\quad\quad-2\sum_{i=1}^{D_{min}}(-1)^{1-I(a_{i}=I_{i})}\ \frac{\
abla Q(s_{i},a_{i};\theta_{k})}{\
abla\theta_{k}}\\ &\quad\quad-2\lambda\sum_{i=1}^{D_{min}}(-1)^{1-I(a_{j}=I_{j})}\ \frac{\
abla Q(s_{j},a_{j};\theta_{k})}{\
abla\theta_{k}}\end{split} \tag{36}\]  
where \(t_{m}=0\) if \(terminal=False\), and \(t_{m}=1\) if \(terminal=True\).  
Based on (36), the second part indicates the minority class, and the third part illustrates the majority class. In the case of the imbalanced dataset, if the value of \(\lambda\) is equal to 1, the immediate rewards of the two classes are identical. The value of the second part is smaller than that of the third part, as the number of samples in the minority class is less than that in the majority class. Therefore, the model is biased to the majority class. If the \(\lambda\) value is less than 1, \(\lambda\) will mitigate the immediate rewards of negative samples and will weaken their influences on the loss function of the Q network.