## 2 Methodologies
### Random-Delay-Corrected DRL Framework
### The Kubernetes-Based Cloud Environment Deploying 5G Core Network Functions

The development target of the environment of the PoC prototype is to support online state collection and network controls, thereby providing samples for the agent to make intelligent decisions and APIs for executing these decisions. For this purpose, the environment is composed of the following components. Stateless 5G Core. The stateless design of the 5G core network is envisioned to be an essential approach to better deploy 5G core network functions in a cloud environment without signaling message loss [21]. This prototype aims to scale 5G core network resources automatically to process the time-varying signaling messages simulated by UERANSIM (UERANSIM: simulators for 5G gNBs and UEs without air-interface protocols) so that a message-level stateless 5G core network is set up with a set of OpenXG-5 GCore projects ([[http://git.opensource5g.org/dashboard/projects](http://git.opensource5g.org/dashboard/projects)]([http://git.opensource5g.org/dashboard/projects](http://git.opensource5g.org/dashboard/projects)) (accessed on 2 October 2022)). As is shown in Figure 2, the stateless 5G core network consists of RAN Service Integrated Enabler (RISE), AMF, SME, UDM, AUSE, UDR, UDSF, and UPF. RISE is a middleware between UERANSIM and AMF, responsible for the mutual conversion of HTTP1.1 and SCTP protocols to discover AMF instances. The SCTP protocol within AMF is replaced by HTTP1.1 to send/receive non-access-stratum (NAS) messages. Further, the communication contexts within AMF are removed to the unified data storage, UDSF, to make AMF stateless. When receiving one NAS message by AMF, it will try to obtain the corresponding contexts associated with someone's identifier (e.g., SUPI) first. The processing results will also be updated in UDSF to ensure information consistency. UDSF maintains an MYSQL databasestoring contexts such as _gnb_context_, _ue_ngap_context_, and so on. With these network functions, the stateless 5G core network is able to inter-operate with open-source gNB/UE (e.g., UERANSIM) and also some commercial gNBs (e.g., Baicell, Amarisoft gNB) and UEs (e.g., Huawei Mate 30 5G Pro, Hongmi K30). Therefore, some high-level features such as UE-initiated registration and PDU session establishment procedures can be supported for real 3 GPP-defined signaling messages. In order to collect the network-domain and service-domain data within the cloud-native 5G core network, some modifications are made to the state-of-the-art open-source projects. The \"EventExposure\" service within AMF and SMF is implemented to expose network-domain data, such as UEs' SUPI, IP address UEs' location, and so on. For UPF, we implement a non-3 GPP-defined interface for UPF to expose the service-domain data such as uplink and downlink data rate of a certain UE because 3 GPP has not designed the service-based interfaces for UPF. Kubernetes-based Cloud Environment. The stateless 5G core network is implemented to be adaptable in the cloud environment with instances packaged in Docker containers. Kubernetes, an open-source cloud environment, is chosen to orchestrate all NF instances in this PoC prototype. As is shown in Figure 2, we integrate Prometheus into Kubernetes to collect the resource-domain data automatically. Prometheus is an open-source monitor for resource utilization of physical machines, virtual machines, Kubernetes nodes, and Kubernetes Pods deploying NF instances. To make this happen, each Kubernetes worker node is equipped with _node_export_, each Pod is equipped with _cadvisor_, and Prometheus should be configured to support the service discovery procedure within Kubernetes. There exists one situation that one Pod may be alive while the NF instance deployed in this Pod may crash because of some bugs. In this context, we customize a Python service named _nfState.py_ that will work after one Pod is instantiated.