# Delay-Tolerance-Based Mobile Data Offloading Using Deep Reinforcement Learning
## 2 Delay-Tolerance-Based Mobile Data Offloading Using Deep Reinforcement Learning
### Transmission Control by Deep Reinforcement Learning
As the input parameter of the RL server, the UE information consists of the control target UE and the UE connecting to the same eNB. Although the QoS level can be determined based only on \(content_{n}\) and \(content_{dt}\) of the control target UE, we introduce relative information to assign the UE priority in detail. The relative information to be introduced includes the maximum, median, and minimum values of \(content_{n}\) and \(content_{dt}\). We can expect that the RL server decides the QoS level to be assigned to control the target UE in consideration of the UE connected to the same eNB by sending the relative information of the UE to the RL server. We also define the current time as a learning parameter, because it is used to evaluate the action of the RL server while taking the time information into consideration. Furthermore, we define all the parameters as one parameter in order to avoid learning that not converging too many states although a parameter can be defined for each QoS level. We define the priority of the bandwidth allocated to the UE as action \(a\). There are five QoS levels. The bandwidths are allocated in the order of QoS1 to QoS3. Furthermore, we assign QoS0 to a non-MDOP UE and an MDOP UE that has content that exceeds the delay tolerance. In MDOP, the content is sent at the maximum transmission rate when the delay tolerance of the content is exceeded. Moreover, we assign QoS4 to the UE that does not need to allocate bandwidth. Although as the learning progresses, the RL server can learn to avoid allocating a bandwidth to a specific UE, and the learning will become efficient if this control is given to the RL server as an action. Hence, the RL server assigns one of QoS1 to QoS4 to the UE. The role of the RL server is to determine action \(a\) using collected UE and eNB information via the MDOP server, which is the priority of the bandwidth to be allocated to the UE. The actual transmission rate is determined by the MDOP server based on the QoS level assigned by the RL server. The MDOP server allocates the bandwidth in order from the UE with a high QoS level. Finally, the reward \(r\) is derived by comparing the current load \(L_{t+1}\) and the ideal load \(L_{ideal}\) for each eNB. \(L_{ideal}\) is constant regardless of time. Action \(a\) is evaluated using the reward function. The RL server then learns based on this evaluation result. A reward function of the proposed method is shown in Algorithm 1. First, we compare \(L_{t+1}\) with \(L_{ideal}\) in order to evaluate action \(a\) in terms of whether the load balancing of eNB is achieved. Reward \(r\) is a positive reward when \(L_{t+1}\) is lower than \(L_{ideal}\). In contrast, reward \(r\) is a negative reward when \(L_{t+1}\) exceeds \(L_{ideal}\). Thus, we set that the positive reward as \(+1\) and the negative reward as the value determined according to the difference of \(L_{t+1}\) and \(L_{ideal}\). If the negative reward value is a fixed discrete value, action \(a\) is evaluated in the same manner regardless of the difference of \(L_{t+1}\) and \(L_{ideal}\) when \(L_{t+1}\) exceeds \(L_{ideal}\). Therefore, we define \begin{table}
\begin{tabular}{c c} \hline \hline
**Value** & **Attribute** \\ \hline Remaining amount of content (\(content_{n}\)) & UE information \\ Delay tolerance time of content (\(content_{dt}\)) & UE information \\ Maximum \(content_{n}\) of UEs & Relative information \\ Median \(content_{n}\) of UEs & Relative information \\ Minimum \(content_{n}\) of UEs & Relative information \\ Maximum \(content_{dt}\) of UEs & Relative information \\ Median \(content_{dt}\) of UEs & Relative information \\ Minimum \(content_{dt}\) of UEs & Relative information \\ Available bandwidth & eNB information \\ Current load & eNB information \\ Ideal load & eNB information \\ Current time & eNB information \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Learning parameters of the proposed method at \(t\). a negative reward using continuous values to avoid such an evaluation.