## 3 Results
### Illustration and Evaluation of Algorithm Performance  
In order to show the advantages of the proposed algorithm, we compared its performance with some heuristic optimization algorithms. Each algorithm experienced the same network change in the simulation. Between every two changes, the network model was trained or iterated a certain number of times. In the following figures, QL and co-QL represent the QL algorithm and QL with user cooperation considered, respectively. ACO, DE, PSO and SA denote the ant colony optimization algorithm, differential evolution algorithm, particle swarm optimization algorithm and simulated annealing algorithm, respectively. These heuristic algorithms were taken for comparison because they are typical and efficient. After continuous research and promotion, these algorithms have shown high efficiency and good performance in solving dynamic optimization problems [23; 24; 25; 26]. Therefore, this simulation compared their performance with QL to illustrate the advantages of reinforcement learning compared with heuristic algorithms in solving dynamic optimization problems. The benchmark in the following figures is the result corresponding to the greedy algorithm with random exploration. This algorithm has the advantages of being simple and fast; however, it is easy to get trapped in a local optimal result. Figure 5 illustrates the convergence speed of the various algorithms in solving optimization problems. The abscissa in Figure 6 represents the iteration turn of the algorithm, and the ordinate is the value of the objective function corresponding to the current optimization result. It can be seen that the convergence speed of QL was only inferior to that of the particle swarm optimization algorithm; however, its optimization result was better. This disadvantage in convergence speed can be compensated by offline training; thus, in most cases, QL has better convergence speed. In a practical network, heuristic algorithms are feasible only when the network fluctuation is small. In the case of obvious network fluctuation, the objective function of the whole network may change significantly, which will make the heuristic algorithm invalid. However, QL is good at dynamic optimization  
Figure 4: Algorithm flow chart. and can adapt well to these network changes. In addition, the convergence speed of QL was further improved and the final optimization result was also better after adopting the view of user cooperation. Figure 6 shows the reward curve of the two QL algorithms. The abscissa represents the iteration turn, and the ordinate is the average reward of the current turn. It can be seen that QL can realize a rapid increase in rewards and eventually converge on a stable result; thus, this algorithm is effective. In addition, from the above two figures, comparing the QL algorithms with and without user cooperation, the one with user cooperation had a better convergence performance. Since the number of tasks in the network is reduced by user cooperation during optimization, it is obvious that the QL with cooperation will obtain more rewards and converge on a better result than the one without cooperation. In order to further analyze the dynamic performance of the QL algorithm in solving optimization problems and explain the functions of user cooperation, including making the best use of network resources and compressing the exploration space of the algorithm, we used the above algorithms to solve several dynamic optimization problems under different conditions of M, N, G, etc. By observing the final result of optimization, the following results were obtained. In particular, for the model described in this paper, the delay is composed of the computation delay and transmission delay. Here, the computation delay depends only on the device itself, which will not change during the simulation. Figure 5: Comparison of convergence speed of algorithms. Figure 6: Reward functions of the two QL algorithms. The change in total network delay already reflects the change in network transmission delay.