## 4 Proposed DQN-Based Offloading Decision Algorithm
### DQN-Based Offloading and Compression Decision
```
1:Initialize replay memory \(D\) to capacity \(N\)
2:Initialize action-value function \(Q\) with random weights \(\theta\)
3:Initialize target action-value function \(\hat{Q}\) with weights \(\bar{\theta}=\theta\)
4:for episode = 1, \(M\)do
5: Initialize sequence \(s_{1}=\left\{x_{1}\right\}\) and preprocessed sequence \(\phi_{1}=\phi\left(s_{1}\right)\)
6:for\(t=1\), \(T\)do
7: With probability \(\epsilon\) selects a random action \(a_{t}\)
8: Otherwise select \(a_{t}=argmax_{a}Q\left(\phi\left(s_{t}\right),a;\theta\right)\)
9: Execute action \(a_{T}\) in emulator and observe reward \(r_{t}^{\epsilon}\), \(r_{t}^{d}\) and image \(x_{t+1}\)
10: Set \(r_{t}=\alpha\) * \(r_{t}^{\epsilon}+\left(1-\alpha\right)\) * \(r_{t}^{d}+r_{t}^{c}\) with weight parameter \(\alpha\)
11: Set \(s_{t+1}=s_{t}\), \(a_{t}\), \(x_{t+1}\) and process \(\phi_{t+1}=\phi\left(s_{t+1}\right)\)
12: Store transition \(\left(\phi_{t},a_{t},r_{t},\phi_{t+1}\right)\) in \(D\)
13: Sample random minibatch of transitions \(\left(\phi_{j},a_{j},r_{j},\phi_{j+^{\prime}}\right)\) from \(D\)
14:if Episode terminates at step \(j+1\)then
15:\(y_{j}=r_{j}\)
16:else
17:\(y_{j}=r_{j}+\gamma\max_{a^{\prime}}\hat{Q}\left(\phi_{j+1},a;\bar{\theta}\right)\)
18:endif
19: Compute a gradient descent step on \(\left(y_{j}-Q\left(\phi_{j},a_{j};\theta\right)\right)^{2}\) with respect to the network parameters \(\theta\)
20: Every \(C\) steps reset \(\hat{Q}\)=\(Q\)
21:endfor
22:endfor
```  
**Algorithm 1** DQN-based offloading decision ```
1 if task is decided to offload then
2:if\(StandByQcap>=taskdo\)then  compression is not necessary, store the data in the Q
4:else  compress and offload the compressed task to edge server
6:endif
7:else if task is decided to process locally then
8:if\(TDcap>=taskdo\)then  perform the task at terminal device without any negative reward
10:else  perform the task at terminal device
12: a negative reward \(R_{capacity}(r_{t}^{c})\) is added endif
14:endif
```

**Algorithm 2** Threshold-based compression decision

## 5 Performance Evaluation

This section presents the simulation setting and the performance evaluation results of the proposed DRL-based dynamic offloading decision-making algorithm. We simulate the values of execution delay and energy consumption for the offloading decision tasks computed by the proposed algorithm in each time step. We calculate the values for the processing based on (10) and (11), and we compare the values with the cases of fully local computing and fully offloading. ### Simulation Setting

We assume our simulation setting parameters as follows. Each terminal device has the maximal capacity of 200 (normalized); and also has an internal queue (i.e., standBy Q) which has the maximal capacity of 500. In addition, the capacity of edge server is 10,000. This assumption is reasonable because edge servers are generally powerful comparing to user terminal devices, i.e., order of 100 as used in [35]. The capacities of terminal device queue, stanBy Q, and the edge server are set between maximum capacities at the start of the episode. The episode is terminated when the agent performs 30,000 tasks. In addition, the task which the agent should do is selected between 50 and 350 continuously (i.e., uniform distribution) and, if there are any tasks left at the time step, the remaining tasks are accumulated in each time step. We also set the parameters to calculate the execution delay and energy consumption of the terminal device and edge server, as presented in Table 1. The values of each parameter are based on the 100 task executions and they are the representations of ratios for the local computing values when the execution time and energy consumption are set to 1. We also assume that the terminal device can process 100 tasks at one-time step and also the edge server can process 1250 tasks at the same time. In our DQN-based learning architecture, the input size of the model is set to 4. The input size stands for the number of dimension of observation (TD capacity, Q capacity, ES server capacity, and remaining tasks).