# Delay-Tolerance-Based Mobile Data Offloading Using Deep Reinforcement Learning
## 3 Evaluation and Discussion
### Basic Evaluation  
First, we performed a basic evaluation to confirm that the proposed method can perform appropriate control in the scenario where the priority to be allocated is explicit. Table 3 shows the parameters of \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Value** \\ \hline Transmission power of UE & 23 [dBm] \\ Transmission power of eNB & 46 [dBm] \\ Cellular bandwidth & 10 [MHz] \\ Frequency & 2.0 [GHz] \\ Antenna & Omnidirectional antenna \\ TCP & New Reno \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Network environment model. Figure 3: Network architecture of the proposed method. the basic evaluation scenario. Moreover, Figure 4 presents the topology of the basic evaluation. In the basic evaluation, the number of UE is 2, and the content delay tolerance held by each is 60 s and 80 s. The content size is set such that it can be sent when allocating the bandwidth preferentially to the UE with a short delay tolerance. Thus, it is ideal that the RL server learns the control to assign a high priority to the UE that has the delay tolerant 60 s content. We set the episode end time \(t_{end}\) as 100 in Algorithm 1 as the simulation time is 100 s. We set the maximum send rate of the UE as 1000 Kbyte/s and ideal load of eNB to 500 Kbyte/s. Furthermore, the mobility model of the UE is stationary, and the initial position is fixed. We used no-control as a comparison target for the proposed method. Figure 5 shows the training curves tracking the total reward and average q-value in the basic evaluation. We made the RL server learn the basic scenario until 200 epochs or approximately 40,000 data. One epoch means that the scenario for basic evaluation was executed once in the simulation. Based on Figure 5a, we confirmed that the total reward value increased as the number of epochs increased. At the same time, it was confirmed that the average Q-value also tends to converge. We think that the RL server learned the QoS allocation method for load balancing as the learning proceeds since the RL server gets a higher reward as the time to accommodate the current load in the ideal load \(L_{ideal}\) increases. Based on Figure 5, we confirmed that the RL server learned the basic evaluation scenario. Thus, we evaluated the load balancing performance using the learned model. Table 4 shows the excess amount from \(L_{ideal}\) and the transmission amount sent within \(L_{ideal}\), and Figure 6 presents the variation in the QoS level assigned to each UE from generation the content to data transmission completed. Table 4 confirmed that the proposed method reduced the excess data amount and increased the transmission amount sent as compared with the no-control case. In particular, we confirmed that the proposed method reduced the excess data amount by 78% and increased the transmission amount sent by 47% from the viewpoint of the bandwidth utilization efficiency. Furthermore, it can be observed from Figure 6 how the proposed method performed the QoS assignment. The QoS levels are high, middle, low, and empty in order from 0 to 4. The QoS level 5  
Figure 4: Topology of basic evaluation. \begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Parameters**} & \multicolumn{2}{c}{**Value**} \\  & **UE1** & **UE2** \\ \hline Simulation time & 100 [s] & \\ Number of eNB & 1 & \\ Maximum data reception amount & 625 [Kbyte/s] & \\ of eNB & 500 [Kbyte/s] & \\ Ideal load & 500 [Kbyte/s] & \\ Number of UE & 2 & \\ Max send rate of UE & 1000 [Kbyte/s] & \\ Movement method & Stationary & \\ Initial position of UE & Fixed & \\ Data generation interval & 100 [s] & \\ Content size & 30 [MB] & 10 [MB] \\ Delay tolerance of the content & 60 [s] & 80 [s] \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Scenario for basic evaluation. indicates that the content has been completely sent.