## 2 Methodologies
### Random-Delay-Corrected DRL Framework
```
1:Initialize the environment (env) and the agent (agent)
2:Initialize the evaluation network and target network of DQN with weights \(\omega\) and \(\omega^{\prime}\)
3:Initialize the Forward Model with weights \(\theta\)
4:Initialize the action buffer \(\mathbb{A}\) (FIFO-Queue): \(\mathbb{A}=\emptyset\) and \(\mathbb{A}_{avail}=\mathbb{A}\)
5:Initialize the latest observation \(s_{c}\) randomly (\(Id(s_{c})=-1\))
6:Initialize the received observations \(\mathbb{O}=\emptyset\), available \(\mathbb{O}_{avail}=\emptyset\), and rewards \(\mathbb{R}=\emptyset\) by agent
7:Initialize the raw trajectory \(\mathbb{T}=\emptyset\), samples for Forward Model \(\mathbb{D}_{fm}=\emptyset\), and samples for DRLs \(\mathbb{D}_{dtl}=\emptyset\)
8:for\(ep\leq EPISODE\)do
9: Reset env, agent, \(\mathbb{A}\), \(\mathbb{A}_{avail}\), \(s_{c}\), \(\mathbb{O}\), \(\mathbb{R}\), \(\mathbb{T}\), \(\mathbb{D}_{fm}\), and \(\mathbb{D}_{dtl}\)
10:for\(ts\leq T\)do
11:env sends the State \(s_{ts}\) and reward \(r_{ts}\)
12:agent sets \(\mathbb{O}\subseteq\{s_{ts-d_{min}},s_{ts-d_{min}-1},...,s_{ts-d_{max}}\}\); \(\mathbb{O}_{avail}=\{s|s\in\mathbb{O},Id(s)>Id(s_{c})\}\)
13:agent sets \(\mathbb{R}\subseteq\{r_{ts-d_{min}},r_{ts-d_{min}-1},...,r_{ts-d_{max}}\}\)
14:if\(Id(s_{c})==-1\) and \(\mathbb{A}==\emptyset\)then
15:agent adds action \(a_{-1}=\epsilon\) into \(\mathbb{A}\): \(\mathbb{A}.put(a_{-1})\rightarrow\mathbb{A}=\{a_{-1}\}\) and sets \(\mathbb{A}_{avail}=\mathbb{A}\)
16:else
```  
**Algorithm 1** Random-Delay-Corrected DRL Framework with DCTS (e.g., delayed-DQN)```
17:agent adds observations and rewards into \(\mathbb{T}\): \(\mathbb{T}=\mathbb{T}\bigcup\mathbb{O}\bigcup\mathbb{R}\)
18:if\(\mathbb{O}_{\textit{avail}}!=\emptyset\)then
19:agent resets \(s_{c}=\arg\max_{s\in\mathbb{O}_{\textit{avail}}}ld(s)\)
20:agent resets \(\mathbb{A}_{\textit{avail}}=\{a|a\in\mathbb{A},Id(s_{c})\leq Id(a)<ts\}\)
21:agent predicts State \(\hat{s}_{ts}=f(s_{c},A_{\textit{avail}})\) and then executes action \(a_{ts}=\arg\max_{a}Q_{\omega}(\hat{s}_{ts})\) with a probability \(1-e\)
22:agent adds \(a_{ts}\) into \(\mathbb{A}\): \(\mathbb{A}.\textit{put}(a_{ts})\rightarrow\mathbb{A}=\{...,a_{ts-1},a_{ts}\}\)
23:agent adds \(a_{ts}\) into \(\mathbb{T}\): \(\mathbb{T}=\{...,a_{ts}\}\)
24:if\(\exists s,s^{\prime}\in\mathbb{T}\) and \(Id(s)+1==Id(s^{\prime})\)then
25:agent adds sample \(\langle s,a_{Id(s)},r_{Id(s^{\prime})},s^{\prime}\rangle\) into \(\mathbb{D}_{\textit{drI}}\) and sample \(\langle s,a_{Id(s)},s^{\prime}\rangle\) into \(\mathbb{D}_{\textit{fm}}\)
26:agent removes elements \(s\), \(a_{Id(s)}\), and \(r_{Id(s^{\prime})}\) from \(\mathbb{T}\)
27:if\(len(\mathbb{D}_{\textit{drI}})>N_{\textit{mem}}\)then
28:agent samples a random minibatch of \(N_{\textit{mem}}\) samples \(\langle s_{i},a_{i},r_{i+1},s_{i+1}\rangle\) from \(\mathbb{D}_{\textit{drI}}\)
29:agent updates weights \(\omega\) with a gradient step on \(((r+\gamma\max_{a^{\prime}}Q_{\omega^{\prime}}(s^{\prime},a^{\prime})-Q_{ \omega}(s,a))^{2})\)
30: Every \(C\) steps, reset \(\omega^{\prime}=\omega\)
31:if\(len(\mathbb{D}_{\textit{fm}})>M\)then
32:agent samples a random minibatch of \(M\) samples \(\langle s_{i},a_{i},s_{i+1}\rangle\) from \(\mathbb{D}_{\textit{fm}}\)
33:agent updates the Forward Model by minimizing the loss: \(\mathcal{L}(\theta)=\sum_{i=1}^{M}(g_{\theta}(s_{i},a_{i})-s_{i+1})^{2}\)
```

**Algorithm 1**_Cont._

## 3 PoC Prototype: Real-World Cloud-Native 5G Core

In order to evaluate our proposed methodologies, we develop a PoC prototype based on some open-source projects, such as OAI. This prototype aims at managing a cloud-native 5G core network in an online manner automatically, which consists of a Kubernetes-based cloud environment and an AIE for deploying DRL algorithms. The AIE collects network state data from the environment to make an intelligent decision and then executes it in the environment via Kubernetes APIs.