## 2 Materials and Methods
### Introduction of Qlearning Components  
First, it is important to clarify the reason for using Qlearning (QL). When solving dynamic optimization problems, QL has some advantages over commonly used heuristic algorithms. First, QL is better at processing a large amount of network historical data. It can also evaluate the quality of the obtained strategies more easily. Second, the amount of data used by QL is relatively small. QL can quickly obtain the solution to a problem, especially when the solution is related to past experience. Thus, the algorithm has strong adaptability to the environment. Third, using QL can realize the offline training and optimization of the network, which is especially suitable for wireless networks and other scenarios that experience dramatic changes. Finally, compared with the more complex DQN algorithm, QL has a theoretical guarantee of convergence and more easily obtains the solution.  
Figure 3 shows a schematic diagram of reinforcement learning. In the figure, \(S_{t}\) is the current state. For each state of the agent, the algorithm will select an action according to the preset rules. Then the agent obtains the corresponding reward (\(r_{t}\)) and jumps to the next state \(S_{t+1}\) according to the environment. After several episodes, the algorithm will eventually converge on the decision or path with the maximum total reward, i.e., the optimal solution.  
By matching the reinforcement learning model with each component of the optimization problem, it is conceivable that the centralized reinforcement learning agent is the MEC server. The action space includes four sub-spaces, namely the U, V, D and W matrices. Thus, A = [U V D W]. In particular, due to the extra burden associated with continuous variables in solving problems, it is necessary to discretize the power allocation action, i.e., assume that the power can be divided into G gears. Therefore, the elements in the V and W matrices must yield to this condition:  
\[\mathrm{v_{m,i}}\,,\mathrm{w_{m,j}}\mathrm{e}\left\{\mathrm{x_{k}}\Big{|} \mathrm{x_{k}}=\frac{\mathrm{k}}{\mathrm{G}}\mathrm{P_{lim}}\,,\mathrm{k}=0,1, \ldots,\mathrm{G}\right\}, \tag{21}\]  
On this basis, the state will be expressed as the task corresponding to the maximum delay in the network, which can facilitate a more purposeful optimization of the network. According to the index of the current state, the optimal allocation scheme of this network is obvious. The index of the state is equivalent to the index of different tasks in the opinion of user cooperation. Conceivably, user cooperation can help compress the state space according to these indexes. Finally, for each process of action selection and state transfer, since the objective is to minimize the delay and energy consumption, the value of the reward function should be negatively correlated with the value of objective function, as follows:  
\[\mathrm{R(s,a)}=\mathrm{c_{1}}\times(\mathrm{c_{2}}-\mathrm{wsf}(\mathrm{s}, \ \mathrm{a})), \tag{22}\]  
where \(\mathrm{c_{1}}\) and \(\mathrm{c_{2}}\) are two constants and 'wsf' is the weighted sum function shown in Equation (14). In this way, the larger reward represents a smaller value of objective function and better network performance.