## 4 Proposed DQN-Based Offloading Decision Algorithm
### DQN-Based Offloading and Compression Decision
(**a**) represents the figure of \(R_{combi}\) values that is a combination of \(R_{delay}\), \(R_{energy}\), and \(R_{capacity}\). (**b**) is a graph of \(R_{delay}\) per time step, (**c**) is about \(R_{energy}\). Lastly, we conduct the simulations in order to present the novelty of DQN-based algorithm than the other methods. One of the offloading decision algorithms is designed based on well-known greedy algorithms, as discussed in [36]. We remind readers that our proposed BRL-based offloading decisions will be made based on the DQN-based framework under the consideration of our rewards (i.e., combination of delay, energy, and capacity). On the other hand, the greedy algorithm can be used for offloading decision-making based on uploading data sizes (i.e., related to delays), i.e., this algorithm firstly sorts its own data to transmit at first. Then, it transmits the data sequentially. As shown in Figure 7, this greedy-based algorithm introduces more average delays than our proposed DQN-based algorithm. Note that the average delay is defined as follows when \(N\) number of packets are simulated for the average delay calculation:

\[\frac{1}{N}\sum_{i=1}^{N}\left(t_{i}^{\text{Departure}}-t_{i}^{\text{arrival}}\right) \tag{12}\] \begin{table}
\begin{tabular}{c c c c} \hline  & **DQN Based** & **Local Computing** & **Full Offloading** \\ \hline Energy consumption & 52.448 & 315.830 & 71.845 \\ Execution delay & 153.320 & 563.830 & 122.374 \\ \hline \end{tabular}
\end{table} 
Table 2: Energy consumption and latency values with 10,000 tasks. Figure 5: Accumulated reward. Figure 6: Performance evaluation results (comparison among DQN-based, full offloading, and local offloading): execution delay (**left**) and energy consumption (**right**). where \(t_{i}^{\text{Departure}}\) and \(t_{i}^{\text{Arrival}}\) are the times when the \(i\)-th packet is left from the queue and when the \(i\)-th packet is arrived at the queue, respectively. As the average data size for offloading increases, the delays increase. In the DQN-based algorithm, optimal control will be made which considers delay, energy, and capacity, during offloading decisions. Thus, the delays are smaller than the greedy-based decision-making. As shown on the right-hand side of Figure 7, our proposed DQN-based decision-making algorithm outperforms the greedy; and the gaps become larger when the average data sizes increase. ## 6 Concluding Remarks and Future Work

This paper proposes a novel deep Q-network based offloading decision-making method in mobile edge computing systems. For utilizing real-time and high-quality communications in mobile edge networks, efficient offloading algorithms and task scheduling in MEC servers are essentially required. In this paper, therefore, a new dynamic offloading algorithm is proposed, and the algorithm improves its performance based on DQN. For performance evaluation, the results of our simulation can be changed due to environment settings or reward decision parameters. According to the simulation-based evaluation, we verify that the proposed DQN-based algorithm achieves desired performance. As future research directions, various factors can be considered in order to conduct more realistic performance evaluation (e.g., channel allocation, channel quality, and mobility of devices) and real-world implementation (e.g., with Raspberry Pi). In addition, more detailed discussions about energy consumption and energy-efficiency are desired. Therefore, we will include this in our future investigation. Moreover, the computational overhead is definitely considerable for our future work because considering the overhead is meaningful in mobile edge computing research. Lastly, the proposed algorithm in this paper runs in a centralized manner. It would be much scalable and practical if the computation can be done in a distributed manner. Therefore, our next work should be dedicated to the design and implementation of fully distributed multi-agent reinforcement learning algorithm which does not require centralized computation.