## 6 Deep Reinforcement Learning for Imbalanced Classification  
### Imbalanced Classification Markov Decision Process  
Reinforcement learning (RL) algorithms incorporated deep learning (DL) in order to defeat world champions at the Go game and human experts who perform various Atari video games. We express the problem of classification as a guessing game. As shown in Figure 11, the agent receives a sample \(s_{t}\) at each time step and classifies what the sample is, and thus the environment returns it an immediate reward \(r_{t}\) and provides the next sample \(s_{t+1}\). If the agent accurately classifies the sample category, the agent receives a positive reward through the environment. If the agent does not accurately classify the sample category, the agent receives a negative reward through the environment. The agent may accurately guess the samples as much as possible to accumulate the maximum rewards. In order to obtain this accumulation, the agent learns an optimal behavior from its interaction with the environment [4]. The Imbalanced Classification Markov Decision Process (ICMDP) framework is formalized in this paper. The purpose of the ICMDP framework is to decompose the classification task of the imbalanced data into a sequential decision-making problem. For instance, the imbalanced training dataset is \(D=\{(x_{1},l_{1}),\ (x_{2},l_{2}),\ \dots,\ (x_{n},l_{n})\ \}\), where \(x_{i}\) is expressed as the \(i^{th}\) sample, and \(l_{i}\) is defined as the label of the \(i^{th}\) sample. Based on the ICMDP, we propose to train a classifier as an agent with many components, as follows. \begin{table}
\begin{tabular}{c c} \hline \hline
**AUC Value** & **Performance of Model** \\ \hline
0.9–1.0 & Excellent \\
0.8–0.9 & Very good \\
0.7–0.8 & Good \\
0.6–0.7 & Fair \\
0.5–0.6 & Poor \\ \hline \hline \end{tabular}
\end{table} 
Table 4: AUC Performance. **State 5**: The state is identified by the training sample. The agent receives the first sample \(x_{1}\) as an initial state \(s_{1}\) at the beginning of training. The state \(s_{t}\) of the environment at each time step corresponds to the sample \(x_{t}\). When the new episode launches, the order of samples in the training dataset is shuffled in the environment. **Action \(A\)**: The action relates to the training dataset label. The action which is taken by the agent aims at predicting the label of the class. In case of the binary classification, the action is defined as \(A=\{0,1\}\) where \(0\) shows the minority class and \(1\) illustrates the majority class. **Reward \(R\)**: The reward \(r_{t}\) is the response from the environment to measure the success or failure of the agent-based actions. In the imbalanced dataset, we want to direct the agent to learn the policy of optimal classification, so the absolute reward value of sample in the majority class is lower than that in the minority class. It means that if the agent correctly or incorrectly recognizes the minority class sample, the environment will feedback a larger reward or punishment to the agent, respectively. **Transition probability \(P\)**: The transition probability \(P(s_{t+1}|s_{t},a_{t})\) is deterministic in the ICMDP. Based on the order of samples in the training dataset, the agent can move from the current state \(s_{t}\) to the subsequent state \(s_{t+1}\). **Discount factor \(\gamma\)**: The discount factor is defined as \(\gamma\), which belongs to \([0,1]\). The discount factor aims at balancing the immediate and future rewards. **Episode**: The episode is a transition trajectory, which is a sequence of states and actions influencing those states. If all samples in the training dataset are classified or the agent misclassifies the sample from the minority class, an episode will complete. **Policy \(\pi_{\theta}\)**: The policy \(\pi_{\theta}\) is defined as a mapping function \(\pi\): \(S\ \rightarrow\ A\), where \(\pi_{\theta}(s_{t})\) is the action which is performed by the agent in state \(s_{t}\).