## 2 Methodologies
### Random-Delay-Corrected DRL Framework  
The goal of DRL algorithms with PH-MDP is to find an optimal policy \(\pi^{*}(a_{t}|x_{t})\) to maximize the discounted accumulated reward (\(G_{t}\)). \[G_{t}=r_{t}+\gamma r_{t+1}+\gamma^{2}r_{t+2}+... \tag{4}\]  
where \(\gamma\) is the discount factor used to evaluate the importance of future estimates. As is learned from the transition distribution of PH-MDP, the problem can be transformed to be the standard MDP solved by the state-of-the-art DRL algorithms. The random-delay-corrected DRL framework with a forward model and a novel trajectory sampling method is expected to make the transformation happen. Forward Model. Equation (3) reveals that the current state (\(\hat{s}_{t}\)) used to generate an action \(a_{t}\) at time \(t\) can be predicted using latest-received observation \(s_{c}^{(t)}\) and latest-executed actions (\(a_{Id(s_{c}^{(t)})},...,a_{t-1}\)) whose length is dynamic because of the random delay \(d1\); therefore, we propose a forward model to iteratively predict the next state using ground-truth state \(s\) and action \(a\). The forward model is a neural network weighted by parameters \(\theta\) and is presented by \(g_{\theta}(\cdot)\); therefore, we can obtain  
\[\hat{s}=g_{\theta}(s,a) \tag{5}\]  
Then, the state \(\hat{s}_{t}\) can be predicted by Equation (6) as follows. \[ \begin{split} s^{(0)}&=\ \ \ \ \ \ \ \ \ g_{\theta}(s_{c}^{(t)},a_{Id(s_{c}^{(t)})})\\ s^{(1)}&=\ \ \ \ \ \ \ g_{\theta}(s^{(0)},a_{Id(s_{c}^{(t)} )+1})\\...\\ \hat{s}_{t}&=s^{(t-Id(s_{c}^{(t)}))}=\ \ \ g_{\theta}(s^{(t-Id(s_{c}^{(t)} )-1)},a_{t-1})\end{split}  \tag{6}\]  
The forward model takes the collected state and generated action as inputs and outputs the predicted state so that it can be trained using samples such as \((s_{i},a_{i},s_{i+1})\). The loss function can be presented as the minimum mean square error as shown in Equation (7). \[\mathcal{L}(\theta)=\sum_{i=1}^{N}(g_{\theta}(s_{i},a_{i})-s_{i+1})^{2} \tag{7}\]  
Relationship Between \(\Delta T\) and Delays \(d1\). In the PH-MDP, we assume that the observation delay \(d1\) between the agent and the environment varies randomly in the range \((d_{min},d_{max})\) and we denote \(\Delta T\) as the time interval for state collection. Considering the relationship between \(\Delta T\) and \(d1\), there may exist three interaction scenarios illustrated in Figure 1. With the turn-based interaction pattern, \(\Delta T\) is supposed to be greater than the maximum value of \(d1\), that is \(\Delta T\geq d_{max}\). In this scenario, the PH-MDP is exactly the same as the standard MDP so that the agent can generate an action with the received state, which will not be affected by other actions. As discussed above, the turn-based pattern will not be suitable for real-time network control, especially when the environment is changing over time. The periodicity-based pattern with constant delays is an optional solution but with some constraints. We assume that the number of delayed time steps is \(n\) (\(n\geq 1,n\in Z\)) and then it will meet the constraints in Equation (8). \[ \begin{split} n\Delta T&\leq d_{min}\\ (n+1)\Delta T&\geq d_{max}\end{split}  \tag{8}\]  
Then, we can obtain  
\[ \begin{split}\frac{d_{max}}{n+1}&\leq\Delta T\leq \frac{d_{min}}{n}\\ \Rightarrow\frac{d_{max}}{n+1}&\leq\frac{d_{min}}{n} \end{split}  \tag{9}\]  
As \(n\) is supposed to be an integer that is bigger than one, therefore, we can obtain the constraints of the range of \(d1\) as follows:  
\[d_{max}=\psi_{i=1}(1+\frac{1}{i})d_{min}=\{\frac{2}{1}d_{min},\frac{3}{2}d_{ min},\frac{4}{4}d_{min},...\} \tag{10}\]Equation (10) reveals that not all scenarios can be transformed to be the periodicity-based pattern with constant delays, which may limit the applications of the PH-MDP. Therefore, a periodicity-based pattern with random delays is supposed to be the common interaction method between the agent and the environment. In this scenario, the collected state will be delayed for a maximum \(ceil(\frac{d}{\Delta T})\) number of time steps.