## 2 Methodologies
### Modeling: Partial History-Dependent Markov Decision Process  
A standard MDP is characterized by a tuple with (\(S\), \(A\), \(\mu\), \(p\), \(r\)) where \(S\) is the state space (collected from the environment), \(A\) is the action space (network control decision), \(\mu:S\rightarrow\mathbb{R}\) is the initial state distribution, \(p:S\times A\rightarrow\mathbb{R}\) is the transition distribution, and \(r:S\times A\rightarrow\mathbb{R}\) is the reward function (metrics from the environment). As illustrated in Figure 1, the new decision process may be enhanced from the standard MDP, which is defined as a PH-MDP characterized by a tuple with (\(\mathcal{X}\), \(\mathcal{A}\), \(\overline{\mu}\), \(\overline{p}\), \(\overline{r}\)) in this paper. State space \(\mathcal{X}\). Figure 1b,c shows that the current state \(x_{t}\) is decided by the latest received state \(s_{t-d1_{max}}\) and latest executed actions (also called partial history action buffer) \((a_{t-d1_{max}},...,a_{t-1})\). As \(d1\) may vary randomly within a certain range so that the length of partial history action buffer is dynamic; therefore, we can define the state space \(\mathcal{X}=S\times A^{n}\), where \(n\) is a dynamic variable. Then, we can obtain  
\[x_{t}=(s_{c}^{(t)},a_{Id(s_{c}^{(t)})},a_{Id(s_{c}^{(t)})+1},...,a_{t-1}) \tag{1}\]  
where \(s_{c}^{(t)}\in(s_{t-d1_{max},...,s_{t-1}})\) is the latest received state from the environment before time \(t\), \(Id(*)\) is the time step when a state \(s\) or an action \(a\) is generated (e.g., \(Id(s_{0})=0\), \(Id(a_{1})=1\)), and \((a_{Id(s_{c}^{(t)})+1},...,a_{t-1})\subseteq(a_{t-d1_{max}},...,a_{t-1})\). Action space \(\mathcal{A}=A\). The PH-MDP shares the same action space with standard MDP for network control. Initial state distribution \(\overline{\mu}(x_{0})\) where \(x_{0}=(s_{c}^{(0)},a_{-1})\), then we can obtain  
\[\overline{\mu}(x_{0})=\overline{\mu}(s_{c}^{(0)},a_{-1})=\mu(s_{c}^{(0)}) \delta(a_{-1}-c_{-1}) \tag{2}\]  
where \(s_{c}^{(0)}=s_{-1}\) is a random-chosen state at time \(t=0\) because the collected state \(s_{0}\) is still on the road toward the agent; \(a_{-1}\) is a random-chosen action at time \(t=0\) because the agent has not received any state; \(\delta(*)\) is the Dirac delta distribution and if \(y\sim\delta(\cdot-x)\) then \(y=x\)with probability one. Equation (2) reveals that the initial state \(x_{0}\) in PH-MDP is independent with respect to \(s_{0}\) in the standard MDP and transferred to be \(x_{1}\) by a random-chosen action \(a_{-1}\). Transition distribution \(\overline{\mathbf{p}}(x_{t+1}|x_{t},a_{t})\). The PH-MDP extends the transition distribution of the standard MDP \(p(s_{t-1}|s_{t},a_{t})\) with the help of a partial history action buffer, whose definition is presented as follows:  
\[\overline{\mathbf{p}}(x_{t+1}|x_{t},a_{t}) =\overline{\mathbf{p}}(s_{c}^{(t+1)},a_{Id(s_{c}^{(t+1)})}^{(t+1)},...,a_{t-1}^{(t+1)},a_{t}^{(t+1)}|s_{c}^{(t)},a_{Id(s_{c}^{(t)})}^{(t)},...,a_{t-1 }^{(t)},a_{t})\] \[=\delta(s_{c}^{(t+1)}-s_{c}^{(t)})\prod_{i=1}^{t-Id(s_{c}^{(t)})} \delta(a_{t-i}^{(t+1)}-a_{t-i}^{(t)})\delta(a_{t}^{(t+1)}-a_{t})\] \[+\prod_{i=1}^{Id(s_{c}^{(t+1)})-Id(s_{c}^{(t)})}p(s^{(i)}|s^{(i-1 )},a_{Id(s^{(i-1)})}^{(t)}) \tag{3}\] \[\times\delta(s^{(0)}-s_{c}^{(t)})\delta(s_{c}^{t+1}-s^{(Id(s_{c}^ {(t+1)})-Id(s_{c}^{(t)}))})\] \[\times\prod_{i=1}^{t+1-Id(s_{c}^{(t+1)})}\delta(a_{t-i}^{(t+1)}-a _{t-i}^{(t)})\delta(a_{t}^{(t+1)}-a_{t})\]  
where \(s_{c}^{(t)}\) represents the latest received observation by the agent at time \(t\); the superscript of \(a_{t1}^{(t2)}\) means that the action is one element of partial history action buffer at time \(t2\) and the subscript \(t1\) represents the time when this action is generated; \(a_{t}\in A\) is a newly generated action at time \(t\). Equation (3) reveals that there are two cases on how to record the transition distribution at the agent because of the uncertainty caused by the random delay \(d1\) as shown in Figure 1b,c. * No fresh state is received by the agent at time \(t\).