## 2 Methodologies
### Random-Delay-Corrected DRL Framework
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Signaling Request** & **Chain** & **CPU Requirements (Cycles/s)** & **ttl (Time Steps)** \\ \hline _RegistrationRequest_ & \(\langle\)AMF-AUSF-UDM-UDR \(\rangle\) & \(\langle 100,200,400,500\rangle\) & 3000 \\ \hline _ServiceRequest_ & \(\langle\)AMF-AUSF-UDM-UDR \(\rangle\) & \(\langle 200,300,100,400\rangle\) & 2800 \\ \hline _PDUSessionEstablishmentRequest_ & \(\langle\)AMF-SMF-UDM-UPF \(\rangle\) & \(\langle 315,246,177,63\rangle\) & 4000 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameters for signaling requests. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Neural Networks** & **Architecture** & **Learning Rate** & **Training Index** & **Updating Index** & **Batch Size** \\ \hline Evaluation/Target Q & 128 (relu) - 64 (relu) - 3 (linear) & 0.0001 & 10 & 20 & 32 \\ \hline Forward Model & 128 (relu) - 64 (relu) - 8 (sigmoid) & 0.0001 & 10 & \(None\) & 32 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Parameters for neural networks. No-delay scenario. This is an idle scenario with the assumption that the collected state will arrive at the agent immediately without delays; therefore, delayed-DQN works the same as the standard DQN. * Turn-based scenario. There are constant or random delays between the agent and the environment, but the agent will only generate an action when it receives the collected state, whose interaction pattern is shown in Figure 1a. * Constant-delay scenario with the standard DQN. In this scenario, the delays are transformed to be constant delays by carefully choosing the time interval \(\Delta T\) as presented in Section 2.3. Then, the interaction pattern works as shown in Figure 1b. The standard DQN (DQN algorithm with the standard MDP) is used to try to learn an optimal policy for better AMF scaling. * Random-delay scenario with the standard DQN. We design a scenario where the standard DQN tries to learn an optimal policy in a random-delay environment. * Constant-delay scenario with delayed-DQN. Different from the above scenario, this scenario adopts our proposed delayed-DQN to learn an optimal policy. * Random-delay scenario with delayed-DQN. This agent is equipped with the delayed-DQN algorithm to learn from delay-corrected trajectories sampling in a random-delay scenario. ### Evaluation Results

The evaluation results are shown in Figure 6. As we can see from Figure 6a, the no-delay scenario shows the optimal training results compared to others while the turn-based scenario shows the worst performance, which proves that the turn-based pattern cannot adapt to the dynamics in the real-world networks. However, we can still figure out that the periodicity-based pattern shown in Figure 1b,c cannot train an optimal agent when the standard DQN algorithms are used, especially for the random-delay scenario. Although the constant-delay scenario with the standard DQN trains a near-optimal agent, the reward has high fluctuation that can affect the final decisions. Moreover, the random-delay scenario with the standard DQN even has little improvement compared to the turn-based scenario. The results tell us we can not adopt the standard DQN algorithm to the periodicity-based scenarios directly, which cannot guarantee performance. After that, our proposed delayed-DQN based on the random-delay-corrected DRL framework is integrated with the periodicity-based pattern for the validation. The results show that the performance of the delayed-DQN algorithm is very close to the optimal value of the no-delay scenario for both constant-delay and random-delay scenarios. Figure 6b-f evaluate the impact of the state collection time interval \(\Delta T\) on the performance of DQN algorithms. For the evaluation, we set three different \(\Delta T\) values, which are 20, 10, and 5 time steps, respectively. Generally, we can see that it performs better when for \(\Delta T=5\) than the other two scenarios no matter what interaction pattern is chosen.