## 4 Proposed DQN-Based Offloading Decision Algorithm
### DQN-Based Offloading and Compression Decision
The output size is 2, which stands for the number of dimensions of actions (offloading or not). The assumed values can be updated depending on the development of hardware and computing systems. Despite these parameter updates, our scheme can be superior to the other static schemes. Furthermore, we can conduct more data-intensive simulations with various settings or real-world implementation as one of future research directions. ### Simulation Results and Discussions

In this paper, the values of \(R_{delay}\) and \(R_{energy}\) are no more than 1. In addition, we set the ratio of results by local computing and the OCD algorithm as an indicator of the reward. Furthermore, we set \(R_{capacity}\) as a negative reward in order to have 20% of \(R_{combi}\) in (6). According to this setting, the maximum value of \(R_{combi}\) is 0.8. We can see the rewards per time step in Figure 4. The red graph presents the case where \(y=0\). In the reward of delay and the reward of energy graph, \(D_{dqn}\) and \(E_{dqn}\) have the same values with \(D_{local}\) and \(E_{local}\) when the proposed OCD algorithm decides to take an action for non-offloading, i.e., local computing. As a result, the reward values are to be 0, when the OCD algorithm decides offloading \(D_{dqn}\); and \(E_{dqn}\) has a value other than zero. If the values of delay and energy consumption are less than \(D_{local}\) and \(E_{local}\), the rewards are above the \(y=0\). On the other hand, they are below \(y=0\) if \(D_{dqn}\) and \(E_{dqn}\) are greater than the values by local computing. In Figure 4, the first graph is the combination of the below two graphs. It is as shown in Figure 5, and it stands for the reward that the agent receives at every time step. In Figure 5, the red line presents the accumulated reward. Although the OCD algorithm always makes optimal offloading decisions, the reward is not always positive. As a result, the total value of the reward can decrease when the negative reward is accumulated. In particular, there is a constant return of negative rewards from 130 to 150 intervals. We compare the result of 30,000 tasks performed by OCD algorithm with the results performed by fully local computing (i.e., no offloading) and edge computing. The executed simulation gets results as shown in Figure 6 and Table 2. If there are tasks that are not processed at time step \(t\) due to capacity limit, the tasks are accumulated as the tasks to be handled at the next time step. Thus, in the execution delay and energy consumption side, local computing continues to increase the values. Furthermore, from the time step 30, the time to execute tasks is the longest and it reduces TD's energy as maximum. The full offloading has the biggest values at the first time, and it has the smallest growth rate until all the tasks are executed. The reason why the initial values of full offloading is a longer transmission time. The proposed OCD algorithm (DQN-based) has significantly better performance than the local computing by TD. In terms of execution delay, the value of the proposed OCD algorithm (DQN-based) is the lowest until time step 110. However, as the time step goes by, and the task size increases, the results of the proposed OCD algorithm (DQN-based) are better than the results of full offloading. In the energy consumption side, the proposed DQN-based algorithm always has the smallest values. These results show that the proposed OCD algorithm (DQN-based) is the most suitable in real-time wireless communication applications. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Processing delay at terminal & 1.0 \\ Processing delay at edge server & 0.08 \\ Transmission delay (uploading) & 0.25 \\ Transmission delay (downloading) & 0.05 \\ Processing energy at terminal & 1.0 \\ Processing energy at edge server & 0.05 \\ Transmission energy (uploading) & 0.15 \\ Transmission energy (downloading) & 0.05 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Simulation parameters. Figure 4: Reward of each time step \(t\).