# The Framework of 6G Self-Evolving Networks and the Decision-Making Scheme for Massive IoT
## 3 The Distributed Task Offloading Scheme for Massive IoT
### System Model
#### 3.1.4 Problem Formulation  
In the massive IoT scenarios, users make their own decisions according the local information without considering the policies of the other users. Thus, for the \(j\)th user, we  
where \(\pi_{j}(t)\) is the policy of the user generating the \(j\)th service at time \(t\), and \(\pi_{-j}(t)\) denotes the policies of the other users. \(\gamma\in[0,1]\) is the discount rate to determine the weight of the future reward. Notice that the above problem is the game process, and there exists a Nash equilibrium (NE) policy \(a_{j}^{*}\) for the user \(j\), that is  
\[\mathcal{R}_{j}(s,a_{j}^{*},a_{-j}^{*})\geq\mathcal{R}_{j}(s,a_{k},a_{-k}),(k \
eq j) \tag{13}\]  
We first use the DQN to get the NE policy, and the target value can be denoted as  
\[y_{j}^{DQN}=R_{j}+\gamma\hat{Q}_{j}(s^{\prime},a;\theta^{-}) \tag{14}\]  
In DQN, the neural network (NN) is used to approximate the \(Q\)-value function, and the loss function can be denoted as  
\[L(\theta)=E[y_{j}^{DQN}-Q(s,a;\theta)]^{2} \tag{15}\]  
Then, to solve the problem of overestimation in typical DQN, we used DDQN [46] to solve the game problem. The target can be denoted as  
\[y_{j}^{DDQN}=R_{j}+\gamma\hat{Q}_{j}(s^{\prime},\underset{a_{j}\in A_{j}}{\arg \max}Q_{i}(s^{\prime},a;\theta);\theta^{-}) \tag{16}\]  
Finally, we further explore the utilization of D3 QN [47] to solve the game problem. Different from DQN and DDQN, the output of D3 QN network consists of theses two parts: state value \(V(s)\) and advantage value \(A(s,a)\), and the target Q value can be denoted as  
\[Q(s,a)=A(s,a)+V(s) \tag{17}\]  
where \(V(s)=E[R_{j}+\gamma V(s_{t+1})]\), \(A(s,a)=A(s,a)-\frac{1}{|A|}\sum A(s,a^{\prime})\), and \(\frac{1}{|A|}\) denotes the advantages of the entire action space. The training stage is completed in the centralized node and is described in Algorithm 1. ```
1:Input the training steps

### Simulation Results

This part shows the simulation results. We consider a square area of length 500 m. The \(\epsilon\)-greedy policy is used when choosing the action, and we set \(\epsilon\in[0,0.9]\). The learning rate is set as the experience value 0.01, and the discount rate is set as \(\gamma=0.99\). The non-negative weights \(\alpha\) and \(\beta\) in Equation (6) are set as the experience values which are respectively 0.5 and 0.0005. The action-selection cost \(\phi_{j}\) in Equation (11) is set as 0.001. The size of the experience replay memory is 500. Figure 2 gives the comparison of convergence performance among different algorithms. We can see that D3 QN converges faster than other algorithms, which shows that D3 QN has the lowest training complexity, and D3 QN is more suitable for massive IoT scenarios. Figure 3 gives the QoE utility of the different algorithms. The horizontal axis represents the number of users who are active and have the service requests at a certain moment. On the other hand, Q-learning is the the most basic reinforcement learning algorithm, and our D3 QN-based offloading and resource allocation scheme is evolved from the DQN algorithm, so we simply use the Q-learning algorithm and the DQN algorithm as the comparison methods, the same as reference [36]. From Figure 3, we can see that our proposed D3 QN-based scheme can achieve the better performance in terms of the QoE utility, especially when the number of users are relatively large. The main reasons for this result are analyzed as follows: on one hand, the value of the resource allocation is continuous, and Q-learning needs do some discretization processing when meeting non-discrete action, which may cause some bias in decision-making; on the other hand, because of the overestimation of the Q valuein DQN, the output policy of DQN may not be the optimal policy. Figure 2: The comparison of convergence performance among different algorithms. From Figures 2 and 3, we can conclude that our proposed algorithm on the basis of 6G self-evolving networks can achieve better performance.