## 2 Methodologies
### Random-Delay-Corrected DRL Framework
Delay-Corrected Trajectory Sampling. To obtain useful and efficient samples from the PH-MDP with an uncertain number of delayed time steps to train the agent to make the right decisions with delayed observations, we propose a unified delay-corrected trajectory sampling (DCTS) method. We denote \(\mathbb{O}\) and \(\mathbb{R}\) as the possible set of received observations and rewards by the agent, respectively, at time \(t\), which are defined in (11). \[ \begin{split}\mathbb{O}&\subseteq\{s_{t-n_{min}},s_ {t-n_{min}-1},...,s_{t-n_{max}}\}\\ \mathbb{R}&\subseteq\{r_{t-n_{min}},r_{t-n_{min}-1},...,r_{t-n_{max}}\}\end{split}  \tag{11}\]  
where \(n_{min}=ceil(\frac{d_{min}}{\Delta T})\) and \(n_{max}=ceil(\frac{d_{max}}{\Delta T})\). We further define a valid observation set \(\mathbb{O}_{avail}\) to remove outdated observations from \(\mathbb{O}\), which is shown in (12). \[\mathbb{O}_{avail}=\{s|s\in\mathbb{O},Id(s)>Id(s_{c})\} \tag{12}\]  
where \(s_{c}\) is the latest received observation by \(agent\) and \(Id(s)\) reveals the time when observation \(s\) was generated. It means that the outdated observation \(s\) was generated earlier but arrived at the agent later than \(s_{c}\), in which case the outdated observation shall not be used to predict a future state. Similarly, we define an action buffer \(\mathbb{A}\) to record actions generated at each time step and an available action buffer \(\mathbb{A}_{avail}\) to record actions meeting the requirements in (13). \[\mathbb{A}_{avail}=\{a|a\in\mathbb{A},Id(s_{c})\leq Id(a)<ts\} \tag{13}\]  
where \(Id(a)\) is the time when action \(a\) was executed. At the very beginning, the agent does not receive any ground-truth observation from the environment; therefore, the agent initializes a random-generated state \(s_{c}\) with \(Id(s_{c})=-1\) and random-chosen action \(a_{-1}=\epsilon\), where \(\epsilon\in[0,1]\). Over time, the agent may receive one, multiple, none, or outdated observations at each time step. * One observation. There is only one element in \(\mathbb{O}\) and \(\mathbb{O}_{avail}\). This observation \(s\in\mathbb{O}_{avail}\) is then the latest received observation \(s_{c}\) in \(agent\), that is \(s_{c}=s\). Then, with action buffer \(\mathbb{A}\), the agent resets \(\mathbb{A}_{avail}\) that meets Equation (13). This observation \(s\in\mathbb{O}\) and reward \(r\in\mathbb{R}\) are stored in the raw trajectory \(\mathbb{T}\) for training the agent. * Multiple observations. There are multiple elements in \(\mathbb{O}_{avail}\). These observations are later than \(s_{c}\) but arrive at the agent in the same time step. In this case, the newest observation is chosen to be the latest observation, that is \(s_{c}=\arg\max_{s\in\mathbb{O}_{avail}}Id(s)\). Accordingly, \(\mathbb{A}_{avail}\) can be reset. All observations in \(\mathbb{O}\) and all rewards in \(\mathbb{R}\) are stored in the raw trajectory \(\mathbb{T}\) for training the agent. * None observation. There are no observations received by the agent at someone's time step. Therefore, \(s_{c}\) stays the same but \(\mathbb{A}_{avail}\) will be updated by adding executed action at the last time step. Afterward, we can predict the state (\(\hat{s}_{t}\)) on which action \(a_{t}\) is executed with \(s_{c}\) and \(\mathbb{A}_{avail}\), which is based on forward model defined in (6). Hence, action \(a_{t}\) can be determined by the actor policy network \(\pi(a_{t}|\hat{s}_{t})\). In order to better explore the state of the environment, random noise is added to the generated action. Then, \(a_{t}\) will be put into action buffer \(\mathbb{A}\) and raw trajectory \(\mathbb{T}\). Through continuous interactions between \(agent\) and \(env\), a raw trajectory \(\mathbb{T}\) may be as shown in (14).