## 5 Performance Evaluation
### Experimental Analysis  
In this study, we analyzed the performance of four frameworks during the training process by recording their action selection and reward values, as illustrated in Figure 5a,b. Regarding convergence, the reward value of the PPO algorithm stabilized around step 340, while the A2C algorithm and the DQN algorithm showed stabilization at approximately steps 400 and 430, respectively. However, the random policy consistently demonstrated low reward values from the 57 th step onward and was thus not considered for convergence analysis. Through an in-depth examination of the result, it became evident that reinforcement learning algorithms play a crucial role in offloading decisions. Notably, the PPO algorithm exhibited superior convergence compared to the other three benchmarked algorithms, as shown in Figure 5b. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Values** \\ \hline Learning rate \(\lambda_{a},\lambda_{c}\) (PPO and A2C) & \(1\times 10^{-4}\), \(5\times 10^{-4}\) \\ Learning rate \(\lambda\)(DQN) & \(1\times 10^{-4}\) \\ Discount factor \(\gamma\) & 0.98 \\ Clipping factor \(\epsilon\) & 0.2 \\ Entropy factor \(\psi\) & 0.01 \\ Replay buffer size of DQN & 10,000 \\ Sample size in the empirical buffer & 64 \\ Activation functions and optimizers & [PERSON], [PERSON] \\ Hidden sizes & 128, 64 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Hyperparameters of algorithms. Figure 5: (**a**) Action selection count by different schemes; (**b**) training process for the four schemes. Figure 6a shows the line graphs of the average reward value of the four schemes. The random scheme was in a low reward state compared with the other three schemes, and the average reward fluctuated slightly around \(-24\) after \(57\) steps; the average reward value of the DQN scheme was around \(-5.0\) after convergence, the average reward value of the A2C scheme was around \(-3.7\) after convergence, and the average reward was still on the rise after \(400\) steps, while the average reward value of the PPO scheme was over \(-5.0\) after convergence. The PPO scheme had an average reward value of around \(-3.3\) after convergence and was relatively smooth, better than the other three baselines. In terms of training stability, the stability of the PPO algorithm was better than the other three baselines within \(0\)-\(150\) steps, DQN was more stable between \(151\) and \(400\) steps, and the A2C algorithm was smoother after \(400\) steps. Overall, DQN performed better in terms of stability. From Figures 5b and 6a, we observed that the DRL algorithm effectively adapts to complex edge computing environments through end-to-end optimization and thus achieves better performance compared to the randomness strategy. Whereas the randomness strategy lacks targeted decisions, and its results do not provide meaningful information, we further focused on the DQN algorithm, A2C algorithm, and PPO algorithm. Secondly, as shown in Figure 6b, we recorded the weighted average of the delay and energy consumption during the training process. Time cost represents the cumulative sum of transmission delay and computation delay. Energy cost encompasses the combined sum of transmission energy consumption and computation energy consumption. The ultimate average cost is the weighted sum of time cost and energy cost, with the respective weight parameters for time cost and energy cost determined by the neural network. PPO had the highest average total cost in \(0\)-\(30\) steps, and as the algorithms were trained, it became gradually smoother after \(300\) steps at around \(2.87\), which was the lowest average total cost compared to the other algorithms, and the overall view of the weighted average of the total cost was more stable. As shown in Figure 7a,b, the performance metrics of delay and energy consumption were compared among the different algorithms.