###### Abstract
The results show that (1) the periodicity-based pattern can be more beneficial to capture the network dynamics than the turn-based pattern; (2) the random-delay-corrected DRL framework can improve the performance when adopting the DQN algorithm in the periodicity-based scenarios; (3) smaller state collection time interval \(\Delta T\) can further improve the performance of the delayed-DQN algorithm; (4) the proposed framework is well adopted in a real-world cloud-native 5G core network to enable AI-native network automation. To the best of our knowledge, this is the first paper discussing DRL algorithms applications in a real-world online closed-loop network automation with random delays. Compared with the state-of-the-art works, our contributions are concluded as follows. * We abstract the interaction patterns between the agent and the environment and model them to be PH-MDP, a general model that supports turn-based and one-timestep/multi-timestep constant/random delays scenarios. With PH-MDP, it is not necessary to know the specific delays in advance; the agent can monitor whether an observation is received at regular intervals and obtain the delays from the time stamp carried within the observation; therefore, the PH-MDP is able to be well-adopted in a real-world system. * We prove that the PH-MDP can be transformed into the standard MDP in terms of the transmission probability. We propose a forward model instead of an augmented state space to learn the ground-truth transmission probability from a dynamic action buffer. Due to the uncertainty of delays, we propose a delay-corrected trajectory sampling method to obtain samples according to the time stamps of unordered actions and observations. The PH-MDP, forward model, and delay-corrected trajectory sampling method make up our proposed random-delay-corrected DRL framework. * We talk about the relationship between the time interval \(\Delta T\) of state collection and ground-truth delays \(d1\). We discuss ways to define how long one timestep should be, thereby supporting different scenarios (e.g., turn-based scenario) in the use case. * We develop a PoC prototype to evaluate our proposed framework in a real-world system. The environment is a stateless 5G core network deployed in a Kubernetes-based cloud computing with Prometheus for state collection and envAdapter for network control; the agent is an AI Engine (AIE) supported by NVIDIA Cloud CPU to enable customized DRL algorithms uploading and running. We design a specific DRL algorithm, delayed-DQN, based on the framework to enable automated scaling of AMF and several experiments are conducted to show the performance. * We open all source code in Gitlab (OpenXG: [[http://git.opensource5g.org/openxg/openxg-aaiaas](http://git.opensource5g.org/openxg/openxg-aaiaas)]([http://git.opensource5g.org/openxg/openxg-aaiaas](http://git.opensource5g.org/openxg/openxg-aaiaas)) (accessed on 2 October 2022)) (prototype) of Open-Source Radio Access Network (OS-RAN (OS-RAN: [[http://www.openxg.org.cn/?Tutorials_66.html](http://www.openxg.org.cn/?Tutorials_66.html)]([http://www.openxg.org.cn/?Tutorials_66.html](http://www.openxg.org.cn/?Tutorials_66.html)) (accessed on 2 October 2022)) (The code is available for free for those who agree with the License))) community and Github (delayed-DQN: [[https://github.com/dukl/delayed-dqn](https://github.com/dukl/delayed-dqn)]([https://github.com/dukl/delayed-dqn](https://github.com/dukl/delayed-dqn)) (accessed on 2 October 2022))) (delayed-DQN algorithm). The rest of this paper is organized as follows. Section 2 presents our proposed methodologies including scenario abstraction, modeling, and random-delay-corrected DRL framework. Section 3 presents our developed PoC prototype including a Kubernetes-based cloud environment and NVIDIA Cloud GPU-based AIE.