## 2 Methodologies
### Random-Delay-Corrected DRL Framework
RISE performs as a middle box between the radio access networks and the 5G core networks, which inherit the SCTP server from AMF. RISE receives all signaling messages and mixes them without considering the procedures or UEs that individual message belongs to. Moreover, stateless AMF is implemented by decoupling the communication contexts from the functional procedures. In this context, RISE can distribute all signaling messages into multiple AMF instances to be processed, thus supporting UE-unaware message-level load balancing. We abstract the working model as the left figure of Figure 5. Each component (e.g., RISE, AMF) is with a message queue for caching unprocessed signaling messages. For example, the purple rectangle named \"msgInRISE\" is the message queue within RISE to cache the upcoming signaling request messages with the Poisson distribution. Additionally, \"msgUpOnRoad\" and \"msgDOnRoad\" represent the uplink/downlink signaling messages that are still on the road to the peer entities for simplification. The service rate of \"msgInRISE\" and \"msgInAMFInst\" is related to the processing capabilities of RISE and AMF instances, while that of \"msgUpOnRoad\" and \"msgDOnRoad\" is related to the uplink and downlink delays between RISE and AMF instances. Moreover, the signaling messages from RISE are distributed to the AMF instance that has a minimum number of unprocessed messages one by one to be balanced among multiple AMF instances. The target is to allocate an adaptive number of stateless AMF instances to process the upcoming request messages at a low cost. \[\mathbb{A}_{avail}=\{a|a\in\mathbb{A},Id(s_{c})\leq Id(a)<ts\} \tag{13}\]0 Procedure, example configuration file, and example data. ### Reinforcement Learning

For this purpose, the problem is modeled as a MDP, including state representation, action definition, and reward description that are illustrated as follows:

State Representation: The state is represented as \(S=(\mathbf{N},\mathbf{D},\mathbf{C}_{amf},\mathbf{N}_{amf})\). \(\mathbf{N}\) is defined as the vector of a number of messages in \"msgInRISE\" (\(N_{rise}\)), \"msgUpOnRoad\" (\(N_{up}\)), and \"msgDOnRoad\" (\(N_{dn}\)) message queues, that is \(\mathbf{N}=(N_{rise},N_{up},N_{dn})\). \(\mathbf{D}\) donates the vector of uplink (\(D_{up}\)) and downlink (\(D_{dn}\)) delays between RISE and AMF instances, that is \(\mathbf{D}=(D_{up},D_{dn})\). \(\mathbf{C}_{amf}\) represents the vector of the number of CPU cores of AMF instances, that is \(\mathbf{C}_{amf}=(C_{amf-1},C_{amf-2},...,C_{amf-N_{max}})\) and \(\mathbf{N}_{amf}\) represents the vector of number of the unprocessed messages within AMF instances, that is \(\mathbf{N}_{amf}=(N_{amf-1},N_{amf-2},...,N_{amf-N_{max}})\), where \(N_{max}\) is the maximum number of AMF instances supported in the system. For AMF instances that are not instantiated, the corresponding number of CPU cores and unprocessed messages is 0. Action Definition: An action is a valid decision on the number of AMF instances, which may be \"\(-1\)\", \"0\", or \"1\". \"\(-1\)\" means that _envAdapter_ deletes one AMF instance if the number of AMF instances is more than 1. The unprocessed messages in the to-be-deleted AMF instance are equally distributed among other live AMF instances. The AMF instance can accept these messages unless the number of unprocessed messages itself does not exceed its 90% of maximum capacity (\(C_{max}\)) so some messages may be discarded. \"0\" means that the environment keeps running without any change in the number of AMF instances. \"1\" means that a new AMF instance will be instantiated if the number of AMF instances is less than \(N_{max}\). Reward Description: A reward is a signal telling the agent how good the current action is doing. The reinforcement learning agent aims to maximize the estimation of discounted accumulative rewards to obtain better long-term performance.