## 4 Problem Formulation  
We consider the online pickup and delivery problem of an agent under environmental uncertainties. An agent is designed to collect workpieces from \(I(I\in\mathbb{N}_{+})\) specific worksta \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Methods** & \begin{tabular}{c} **Handling of** \\ **Environmental Uncertainty** \\ \end{tabular} & \
\begin{tabular}{c} **Human-** \\ **Interaction** \\ \end{tabular} &
\begin{tabular}{c} **Task** \\ **Sequencing** \\ \end{tabular} \\ \hline Product MDP [46] & – & – & \(\checkmark\) \\ Feb-MDP [22] & real-time way & – & \(\checkmark\) \\ DRL [47] & – & – & \(\checkmark\) \\ Plan-Seq-Learn [48] & – & – & \(\checkmark\) \\ G2 RL [15] & real-time way & – & – \\ Co-HAN [49] & real-time way & \(\checkmark\) & – \\ Ours & at the planning level & \(\checkmark\) & \(\checkmark\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison of different methods for task sequencing. tions and transport them to different destinations. Let \(o_{i}\) be the position of the workstation \(i(1\leq i\leq I)\) and \(g_{i}\) the destination of the workpieces from workstation \(i\). For each workstation \(i\), the agent:  
* Visits \(o_{i}\) and collects workpieces, which is called the collection task
* Delivers the collected workpieces to its destination \(g_{i}\), named the delivery task
* \(T_{i}^{1},T_{i}^{2}\) are the budgets for the collection and delivery tasks, respectively. Therefore, \(I\) workstations define \(2I\) tasks. We denote the task state at time step \(t\) as \(\tau^{t}=\{\tau_{1},\dots,\tau_{i},\dots,\tau_{I},\tau_{I+1},\dots,\tau_{I+i}, \dots,\tau_{2I}\}\), where \(\tau_{i}\), \(\tau_{I+i}\) represent the state of the collection and delivery tasks of workstation \(i\), respectively. Here, \(\forall\tau_{i},\tau_{I+i},\tau_{I},\tau_{I+i}\in\{False,True\}\), where:  
* \(False\) shows a task that is not achieved
* \(True\) represents a task that is achieved. Thus, the initial task state \(\tau^{0}=\{\tau_{1},\dots,\tau_{i},\dots,\tau_{2I}\}=\{False,\dots,False, \dots,False\}\). After \(\tau_{i}\) is achieved at time step \(t\), \(\tau^{t}=\{False,\dots,True,\dots,False\}\). After all tasks are completed, this becomes \(\{True,\dots,True,\dots,True\}\). **Definition 1**. : _If a task is achieved within its given budget, we say that it is online; otherwise, it is offline._  
**Definition 2**. : _If all tasks are online completed, we say that it is a success; otherwise, we consider it as a failure._  
Environmental uncertainties arise from the states of the doors. There are \(N(N\in\mathbb{N}_{+})\) doors in the environment that the agent has to pass through. Each door \(n(1\leq n\leq N)\) can be in one of two states, open or closed. The state of each door is determined by the door control system. Let \(d_{n}^{t}\) define the state of door \(n\) at time step \(t\); then:  
\[d_{n}^{t}= \begin{cases}1,&\text{if the door could open},\\ 0,&\text{otherwise}.\end{cases}  \tag{3}\]  
When the agent needs to pass through a door, it sends an open request to the door control system. However, the request may be denied if the door is not available when the outside is dangerous, such as in a scenario where other vehicles are passing through. Based on the \(2I\) tasks of the agent, we define a set of atomic propositions \(PrP=\cup_{i=1}^{2I}Pr_{i}\) and \(Pr_{i}=\{null_{i},pick_{i},deliver_{i}\}(1\leq i\leq I)\), where:  
* \(null_{i}\): The agent does not accomplish the collection task at workstation \(o_{i}\)
* \(pick_{i}\): The agent successfully completes the online collection task at workstation \(o_{i}\)
* \(deliver_{i}\): The agent successfully completes the online delivery task of workpieces from workstation \(o_{i}\).