## 2 Methodologies
### Random-Delay-Corrected DRL Framework
### Implementation

The prototype runs on four virtual machines (VMs) deployed on one SANGFOR ([[https://www.sangfor.com.cn/](https://www.sangfor.com.cn/)]([https://www.sangfor.com.cn/](https://www.sangfor.com.cn/)) (accessed on 2 October 2022)) server, including three VMs for deploying a Kubernetes (One master and two work nodes: 8-cores CPU, 8G memory, Ubuntu 18.04; Kubernetes version: 1.17.4)-based cloud environment and one VM (AIE: 32-cores CPU; 32G memory; Ubuntu 18.04; GPU version: NVIDIA A100-PCI; NVIDIA-SMI 470.141.03; CUDA version 11.4; cuDNN version: v8.2.4; installation tutorials for NVIDIA Cloud GPU: [[https://zhuanalan.zhihu.com/p/406815658](https://zhuanalan.zhihu.com/p/406815658)]([https://zhuanalan.zhihu.com/p/406815658](https://zhuanalan.zhihu.com/p/406815658)) (accessed on 2 October 2022)) for deploying an AIE. The communication between Kubernetes nodes is via a Flannel plugin with which a Calico ([[https://www.cnblogs.com/xin1006/p/14989365.html](https://www.cnblogs.com/xin1006/p/14989365.html)]([https://www.cnblogs.com/xin1006/p/14989365.html](https://www.cnblogs.com/xin1006/p/14989365.html)) (accessed on 2 October 2022)) plugin is integrated to maintain the IP addresses of Pods. Each NF is packaged in one Docker instance via one specific Dockerfile shown in Figure 4b to be deployed in a Pod on one work node. * @PREFIX@: The unique name of deployment of the Docker instance, e.g., amf. * @NFNAME@: The unique name of Pod when deploying different instances, e.g., amf. * @IP@: The unique IP address of Pod is set supported by the Calico plugin, e.g., 10.244.1.19. * @NODE@: The selected node to deploy a new Pod instance, e.g., node2. * @NFCONTAINER@: The name of newly created container, e.g., amf-1. * @NFIMAGE@: The selected image file to create a new Pod instance (the image file shall be available on the selected node), e.g., amf:cmcc. * @CPU@: The required CPU resource, e.g., 50 m. If the available CPU resource of the selected node is lower than the required one, the Pod instance cannot be created successfully. * @COMMAND@: The commands to run the scripts (_nfState.py_ for opening the states of the Pod instance) and the executable file of NF (e.g., /amf/build/amf/build/amf -c /home/amf.conf -o). * @PATH@: The file sharing path between the container and the host, e.g., /home/. The whole procedure is illustrated in Figure 4a. Firstly, users upload their 3 rd-party applications packaged in compressed files (e.g., app.tar). The AIE unzips these files to obtain four elements including \"env.yaml\", \"api.yaml\", \"procedure.py\", and \"model.pb\". An example \"env.yaml\" is shown in Figure 4e, which indicates the requirements of the running environment; an example \"api.yaml\" is shown in Figure 4d, which indicates the APIs for data collection and network control. With the \"env.yaml\" file, the AIE creates a new NVIDIA Docker (instance to the application with \"procedure.py\" (logic of the algorithms) and \"model.pb\" (non-trained or trained neural network). The data manager within the AIE periodically collects state data from the underlying environment and provides many fine-grained APIs for data openness. The example data collected by Prometheus are shown in Figure 4c, which can also be displayed graphically via Grafana. ## 4 Use Case: Delayed-DQN-Enabled AMF Scaling

On top of the PoC prototype, we implement a use case to evaluate our proposed random-delay-corrected DRL framework. In the 5G core network, AMF is of vital importance for connecting the radio access networks and core networks, which may result in performance bottlenecks. Our previous work [21] has designed and implemented a message-level stateless AMF that helps to realize signaling-no-loss and UE-unaware scaling; therefore, we choose to implement a delayed-DQN-enabled AMF scaling use case for the validation. The architecture is illustrated in Figure 5.

Within the environment, multiple AMF instances collaborate together to process signaling messages.