## 4 Distributed Computation Offloading and Resource Allocation
### DRL-Based Offloading  
Hence, we provide the offloading technique based on the DRL to improve computing performance, where the DRL functions as an agent, known as a deep Q-network, which combines the traditional RL and DNN to speed up the learning process [4]. For each action in DRL, we have a separate output. State representation is used as the only input to the neural network. DRL is renowned for its ability to reduce the complexity of a system by calculating the Q-values for all conceivable actions for a given state with a single forward pass [3,4]. In addition, better implementation is achieved by utilizing the replay memory and intrinsic generalization capabilities introduced by NN DRL to the minimum interaction with the complex environment. Q-learning in RL often results in an impractical situation. An approximator is used in DRL to estimate the action-value function \(Q(s,a;\omega)\approx Q^{*}(s,a)\)), which notably enhances the system's performance [4]. In Algorithm 2, the training procedure and offloading mechanism are displayed. ```
1:Input:
2:set random weights \(\omega\) for the Q-network \(Q(s,a)\) initialization
3:build a weighted \(\hat{\omega}\) target Q-network \(\hat{Q}(s,a)\)
4: Assign a capacity of \(N_{D}\) to the replay memory \(D\)
5: Determine the mini-batch size \(M\) for training network
6: Establish the maximal training session \(Emax\)
7:Output:
8: Trained Q-network (\(Q(s,a)\))
9:Begin Iteration:
10:for each episode within \(E_{max}\)do
11: Establishing the standard parameters of simulation for the computational offloading environment
12: assign the starting state \(s_{0}=[g_{k}^{mode},f_{l}^{k},R_{k},M_{k}]\)
13:for each decision step \(t=1:T-1\)do
14: Establish a random value \(xx\) between 0 and 1
15:if\(x\leq\epsilon\)then
16: Randomly select action \(a_{t}\) for mode selection and power allocation as \([s_{k}^{m},p_{k}]\)
17:if\(s_{k}^{m}==edge\)then
18: Run algorithm (1) and allocate optimal computing resources to the user \(k\)
19:elseif\(s_{k}^{m}==cloud\)then
20: Run algorithm (1) and allocate optimal computing resources to the user \(k\)
21:else
22: Execute the task with local resources
23:endif
24:else:
25: Take the best possible action \(a_{t}\)
26:\(a_{t}=\underset{a_{t}\in A}{\arg\max}\,Q(s_{t},a_{t};\omega)\) for selecting mode and allocating power as \([s_{k}^{m},p_{k}]\)
27:endif
28: Offload the task \(M_{k}\) of \(k\)-th user
29: Perform action \(a_{t}\) and assess the system's energy
30: Examine the reward \(r_{t}\) based on the formulated problem described in (16). 31: Put the reward \(r_{t}\) along with \(s_{t},s_{t+1}\), and \(a_{t}\) as an interaction sample \((s_{t},a_{t},r_{t},s_{t+1})\) in the replay memory \(D\). 32: From the replay memory \(D\), randomly sample the mini-batch with the transitions \((s_{t},a_{t},r_{t},s_{t+1})\) of size \(M\). 33: By using mini-batch descent gradient on \((r_{t}+\xi\underset{a^{\prime}\in A}{\max}\hat{Q}(s_{t+1},a^{\prime};\hat{ \omega})-Q(s_{t+1},a_{t};\omega))^{2}\) with \(\omega\), to train the Q-network. 34: Update the target Q-network with parameters \(\omega\) to \(\hat{\omega}\) periodically. 35:endfor
36:endfor
```  
**Algorithm 2** DRL Algorithm for Computing Offloading in E-RAN  
The input for the deep neural network (DNN) comprises various parameters related to the current state of the user \(k\), including available communication modes, radio and computing resources at the edge, and the size of the offloading work \([s_{k}^{m},f_{l}^{k},R_{k},M_{k}]\). This information is depicted in both Figure 2 and Algorithm 2. The output formulates the \(Q(s,a,\omega)\) with the weight of \(\omega\) to achieve a more precise estimation based on all possible actions. To establish a balance between exploitation and investigation, the deployed IoT devices choose the offloading policy \(a_{t}\in A\) based on the DNN result, which was received via a \(\epsilon\)-greedy policy.