## 2 Methodologies
### Random-Delay-Corrected DRL Framework
In this use case, the agent is envisioned to learn an optimal policy for deploying an adaptive number of AMF instances under UE signaling requests lasting for certain time steps with a Poisson distribution. The reward function is defined as (20). Firstly, the current number of AMF instances \(N_{aft}\) is calculated as (17), where \(N_{cnt}\) denotes the number of AMF instance before executing an action. Then, the average capacity (\(C_{avg}\)) of unprocessed signaling messages among all AMF instances is calculated as (18), where \(N_{total}\) denotes the total number of unprocessed signaling messages in the system, \(S_{cp}\) denotes the processing capability of AMF instances (messages per second), and \(T\) denotes the running time before arriving at next time step. When \(C_{avg}\) exceeds the 0.9 times maximum capacity \(C_{max}\), we set \(C_{avg}=2C_{max}\) to calculate a negative reward value to punish such actions. Then, the basic reward value \(R_{base}\) is calculated as (19), considering the QoS for processing signaling messages and the costs for deploying AMF instances at the same time. In order to limit the reward value in each step to \([0,1]\), the reward value is normalized as defined in (20); therefore, we obtain the final reward value \(R_{final}\). \[\mathbb{A}_{avail}=\{a|a\in\mathbb{A},Id(s_{c})\leq Id(a)<ts\} \tag{13}\]1 Architecture: Delayed-DQN-enabled AMF scaling. \[(N_{aft},R_{punish})=\left\{ \begin{array}{ll}(N_{cnt}+1,0)&\text{$a=1$ and $N_{cnt}<N_{max}$}\\ (N_{cnt},-0.5)&\text{$a=1$ and $N_{cnt}=N_{max}$}\\ (N_{cnt},0)&\text{$a=0$}\\ (N_{cnt},-0.5)&\text{$a=-1$ and $N_{cnt}=1$}\\ (N_{cnt}-1,0)&\text{$a=-1$ and $N_{cnt}>1$}\end{array} \right. \tag{17}\]

\[C_{avg}=\left\{ \begin{array}{ll}N_{total}/N_{aft}-S_{cp}\times T&\text{$C_{ avg}\leq 0.9C_{max}$}\\ 2C_{max}&\text{$C_{avg}>0.9C_{max}$}\end{array} \right. \tag{18}\]

\[R_{base}=\alpha(0.9-C_{avg}/C_{max})/0.9-N_{aft}/N_{max}+R_{punish} \tag{19}\]

\[R_{final}=(R_{base}-(-3.3))/(1-N_{aft}/N_{max}-(-3.3)) \tag{20}\]

### Experiment Setup and Parameters

In this experiment, three types of signaling requests including _RegistrationRequest_, _ServiceRequest_, and _PDUSessionEstablishmentRequest_ were simulated by UERANSIM following the \(\sin(\cdot)\)-like flow pattern (that is \(500\sin(x)+500\)). There are flows varying from \([0,1000]\) in 10 time steps. Each type of signaling request is processed via a service function chain (e.g., _RegistrationRequest_ is processed via a \(\langle\)AMF-AUSF-UDM-UDR\(\rangle\) chain). Further, each type of signaling request has different resource (e.g., CPU) requirements in different network functions and has different \(ttl\). The detailed parameters for signaling requests are shown in Table 1.

In addition, we deploy four instances for each network function except AMF and maximum \(N_{max}=30\) AMF instances in this use case to process these signaling requests. In the beginning, there is only one AMF instance available and the agent will decide to add/delete/maintain the number of AMF instances according to the received state time by time. For each instance, the maximum capacity and CPU are set to be \(C_{max}=5000\) and \(300\) cycles/s. Neural networks' parameters are shown in Table 2.

The experiment will run for \(EPISODE=500\) episodes and 20,000 time steps for each one. In order to explore the possible states in the environment, we set the maximum epsilon and epsilon decay to be \(e_{max}=0.9\) and \(e_{decay}=0.99996\), respectively. At each time step, the agent will randomly choose an action with a probability \(\epsilon\) and then \(\epsilon\) will be changed to be \(\epsilon=\epsilon\times e_{decay}\) until it reaches \(e_{max}\). ### Benchmark Scenarios

To evaluate the performance and feasibility of our proposed random-delay-corrected DRL framework (delayed-DQN in this use case), we design the following scenarios: