## 2 Methodologies
### Scenario Abstraction  
Thanks to the efforts by 3 GPP, a new network function, namely NWDAF, is designed to support online closed-loop automation for cloud-native 5G core networks. With similar feedback mechanisms, DRL techniques are expected to be suitable to facilitate AI-native network automation. However, the 0-delay assumptions in the standard DRL algorithms are not effective for the scenarios in real-world environments where observation delays (state collection), reward delays (metrics), and action delays (action execution) exist. Considering the delay features, we summarize three different interaction scenarios, which are illustrated in Figure 1.  
Turn-based scenario (Figure 1a). In this scenario, the environment prepares and sends observations every \(T\) time steps where \(T\geq d1+d2\). The agent generates an action only when it receives an observation. This interaction pattern will bring two-fold drawbacks: (1) the low-efficiency sampling data cannot be used to learn the transfer probability \(p(s^{\prime}|s,a)\) thereby degrading the performance of DRL algorithms; (2) the sparse actions (action execution time interval \(T^{\prime}\)) will not meet the dynamic requirements of diversified and differentiated vertical applications or signaling requests. Periodicity-based scenario with constant delays (Figure 1b). In order to address the issues of Figure 1a, we consider a periodicity-based scenario where the environment collects the state periodically (at a smaller time interval \(T\)). This scenario is expected to capture the time-varying dynamics when \(T\) is carefully determined; therefore, the agent will generate an action at every time step after receiving the first observation and these actions are  
Figure 1: Different interaction scenarios between the agent and the underlying networks considering different delay patterns. \(d1\) and \(d2\) are delays for state collection and action execution, respectively. \(T\) denotes the time interval of state collection and \(T^{\prime}\) is the time interval of action execution. The red symbols such as “\(\kappa\)” (as shown in (c)) mean that the collected states will not be used to generate the next action. The red lines are the processes for state collection, while the green lines are for action execution. Here, we assume that the action delays are constant. executed at each time step for near real-time network controls. The reward metrics go along with observations to evaluate how well the last action executes on the last state. In addition, complete trajectories samples \([s,a,s^{\prime},r]\) can be obtained through periodical state collection to be used to learn the transfer probability \(p(s^{\prime}|s,a)\); however, because of the existence of interaction delays, the input observation of the neural network is delayed for several time steps and changed by the latest executed actions. If we input the delayed observation \(s_{t-d1}\) into the neural network, it will generate a non-optimal action. This is quite different from that of the standard DRL algorithms assuming that the observation is immediate and only changed by the latest one executed action (\(\pi(a_{t}|s_{t})\)). Therefore, one optional solution is to predict the current state (\(\hat{s}_{t}\)) using the latest observation \(s_{t-d1}\) and latest executed actions \((a_{t-d1},...,a_{t-1})\) for generating a new action \(\pi(a_{t}|\hat{s}_{t})\): \(p(s_{t}|s_{t-d1},a_{t-d1},...,a_{t-1})\). In this scenario, the length of \((a_{t-d1},...,a_{t-1})\) is fixed. Periodicity-based scenario with random delays (Figure 1c). Different from Figure 1b, this scenario is more realistic because the delays for state collection and transmission are usually random, which brings greater challenges for trajectory sampling and agent training.