## 2 Methodologies
### Random-Delay-Corrected DRL Framework
The results prove that frequent state collection with smaller time intervals \(\Delta T\) can better adapt to network dynamics. In summary, Figure 6 proves that (1) the periodicity-based pattern can be more beneficial to capture the network dynamics than the turn-based pattern; (2) the random-delay-corrected DRL framework can improve the performance when adopting the DQN algorithm in the periodicity-based scenarios; (3) smaller state collection time interval \(\Delta T\) can further improve the performance of the delayed-DQN algorithm. ## 5 Conclusions

This paper proposes a random-delay-corrected deep reinforcement learning framework to address the challenges for DRL algorithms application in real-world networks where random delays exist between the agent and the environment. Firstly, the interaction patterns are abstracted to be three scenarios including turn-based, periodicity-based with constant delays, and periodicity-based with random-delays scenarios. Secondly, these interaction patterns are modeled as a partial history-dependent Markov decision process, which extends from the standard Markov decision process. Thirdly, a random-delay-corrected deep reinforcement learning framework with a forward model and delay-corrected trajectory sampling is presented to transform the PH-MDP to the standard MDP, thereby training the agent using the state-of-the-art DRL algorithms. Fourthly, an example DRL algorithm, namely delayed-DQN, based on the framework is proposed. For the evaluation, a PoC prototype for a cloud-native 5G core network is developed and a use case, namely delayed-DQN-enabled AMF (one network function in 5G core network) scaling, is implemented. The evaluation results show that the proposed methodologies perform better in the random-delay networks than others. This paper makes good contributions to DRL-enabled online closed-loop management for real-world networks thereby enabling AI-native network automation towards 6G. Conceptualization, K.D., L.W. and X.W. ; methodology, K.D. ; software, K.D., Y.L. and H.N. ; validation, L.W. and S.H. ; formal analysis, K.D. ; investigation, L.W. ; writing--original draft preparation, K.D. ; writing--review and editing, L.W. All authors have read and agreed to the published version of the manuscript. This research was funded by the National Key Research and Development Program of China under Grant 2019 YFB1803300 and Beijing Natural Science Foundation (L202002). Not applicable. Figure 6: Evaluation results. “TN”, “ND”, “CD”, and “RD” mean turn-based, no-delay, constant-delay, and random-delay scenarios, respectively. “20-TS”, “10-TS”, and “5-TS” mean \(\Delta T\) is set to 20, 10, and 5, respectively. For example, “CD-DQN-5-TS” means a constant-delay scenario where states are collected every 5 time step and the agent adopts the DQN algorithm to make decisions. The results are the average rewards for each 10 episodes. Data Availability StatementOur developed PoC prototype is open to the OS-RAN community ([[http://git.opensource5g.org/openxg/openxg-aiaaas](http://git.opensource5g.org/openxg/openxg-aiaaas)]([http://git.opensource5g.org/openxg/openxg-aiaaas](http://git.opensource5g.org/openxg/openxg-aiaaas)) (accessed on 2 October 2022) ) for anyone who agrees with the License. The authors declare no conflict of interest. ## References

* (1) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Digital-Twin-Enabled 6G: Vision, Architectural Trends, and Future Directions. _IEEE Commun. Mag._**2022**, _60_, 74-80. [CrossRef]
* (2) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; et al. White Paper on 6G Networking. 2020. Available online: [[https://biblio.ugent.be/publication/8668820](https://biblio.ugent.be/publication/8668820)]([https://biblio.ugent.be/publication/8668820](https://biblio.ugent.be/publication/8668820)) (accessed on 30 October 2022). * (3) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Micro-service-based radio access network. _China Commun._**2022**, _19_, 1-15.