## 2 Materials and Methods
### Introduction of Exploration Strategy, Iteration Process and Parameters  
The following section discusses some portions of QL, an algorithm for reinforcement learning. In each loop of the QL algorithm, the MEC server first selects a feasible action from the action space according to the current state. After the server performs this action, the state changes accordingly. The server will record the reward and update the corresponding term in the Q table. Obviously, in the process of algorithm implementation, there are two important mechanisms, namely exploration and updating of the Q table. The random exploration strategy is an important part of exploration. In order to ensure that all feasible actions can be selected by the MEC server with a non-zero probability and the probability of the current optimal solution is always higher than others, the Bozeman  
element in the Q table and its denominator is the sum of all exponential terms in this state. In addition, \(\theta\) is the ability of action differentiation. The rule for updating the Q table is also important. Each time the MEC server receives a reward, its corresponding term in the Q table must be updated. Usually, the Berman equation of reinforcement learning can be used as the rule for updating the Q table:  
\[\text{Q}(\text{s}_{\text{k}},\text{a}_{\text{k}})=(1-\alpha)\text{Q}(\text{s}_ {\text{k}},\text{a}_{\text{k}})+\alpha[\text{R}\left(\text{s}_{\text{k}},\text{ a}_{\text{k}}\right)+\underset{\text{a}_{\text{k}+1}}{\text{max}}\text{Q}(\text{s}_{ \text{k}+1},:)], \tag{24}\]  
where \(\alpha\) is the learning efficiency of the agent, the value of which affects the convergence speed and optimization results of the model, and \(\gamma\) is the discount of the impact of future rewards on the current state. A larger value indicates that the model pays more attention to long-term benefits, and vice versa. \(\underset{\text{a}_{\text{k}+1}}{\text{max}}\text{Q}(\text{s}_{\text{k}+1},:)\) is a term in the Q table that corresponds to the optimal action of the next state \(\text{s}_{\text{k}+1}\) with the prerequisite that the current action is \(\text{a}_{\text{k}}\). The pseudocode of algorithm is depicted in the following Algorithm 1. ```
Input: state and action space, network conditions, initialized reward R = 0, training episodes K Output: optimal resource allocation strategy
1. check the task similarity by the rule of user cooperation
2. initialize Q table, Q = 0
3. choose initial state \(\text{s}_{0}\)
4. for i \(\leq\) K:
5. calculate the exploration strategy according to (16), select the action to be performed according to epsilon and \(\pi\)
6. Update the R matrix according to the result of performing action
7. update Q table according to (17)
8. move to the next state \(\text{s}_{\text{k}+1}\)
9. end for
```  
**Algorithm 1** Pseudocode for Qlearning algorithm with user cooperation  
The algorithm operates under the prerequisites that the MEC server knows the network conditions completely, resource allocation is entirely observable and all actions are interconnected. In addition, in this model, the network state at the next moment is only related to the current state and the action performed. Therefore, the Qlearning problem can be modeled as a Markov decision model. At the beginning of the algorithm, the MEC server needs to initialize the Q table and randomly select the initial state \(\text{s}_{0}\). After obtaining the network state \(\text{s}_{\text{k}}\), the server determines the optimal action by searching the maximum term of the current state in the Q table and calculates the strategy for action exploration. Whenever the MEC server searches the action, it will select the best action with a certain probability. At the same time, in order to ensure that the server always maintains exploration behavior during the learning process to avoid falling into local optimums, the MEC server will also select a random action according to the strategy \(\pi\) with another certain probability.