## 5 Proposals
### High-Level: FSA-RL Learner  
The high level is a global learner based on FSA. All tasks are first written in formula \(\varphi\), as shown in Equation (4). Afterwards, an FSA \(\mathcal{A}_{\varphi}\) is transformed from formula \(\varphi\) using off-the-shelf tools, where \(\mathcal{A}_{\varphi}=<Q,PrP,\delta,q_{0},F>\). Based on \(\mathcal{A}_{\varphi}\), we construct the high-level learner \(\mathcal{M}_{h}=<S_{h},A_{h},P_{h},\delta_{h},R_{h},q_{0},q_{F},PrP,L>\) where:  
* \(S_{h}=Q\) is the state set
* \(A_{h}=\{move,stay\}\) is the action set
* \(P_{h}:S_{h}\times A_{h}\times S_{h}\rightarrow[0,1]\) is the state transition probability
* \(\delta_{h}:S_{h}\times PrP\times L(S_{l})\to S_{h}\) is the state transition function (with \(S_{l}\) being the low-level state set)
* \(L:S_{l}\to 2^{PrP}\) is the label function
* \(PrP\) is the atomic proposition set
* \(R_{h}\) is the reward function, which is determined by the low-level MDP learner
* \(q_{0}\) is the initial state and \(q_{F}\in F\) is the accept state. Figure 3: Communication between the agent and the door control system. The agent chooses an action \(a_{h}\in A_{h}\) for state transition from \(s_{h}\) to \(s^{{}^{\prime}}_{h}\) with probability \(P(s^{{}^{\prime}}_{h}|s_{h},a_{h})\). We claim that in the case of \(a_{h}=move\), the agent moves to next state \(s^{{}^{\prime}}_{h}(s^{{}^{\prime}}_{h}\
eq s_{h})\) with probability \(p(s^{{}^{\prime}}_{h}|s_{h},a_{h})\) and may remain at \(s_{h}\) with probability \(1-p(s^{{}^{\prime}}_{h}|s_{h},a_{h})\). An example is shown in Figure 4 (left), where the agent chooses to move to \(q^{{}^{\prime}}\) from \(q\). However, it may stay at \(q\) with probability \(1-p(q|q,a_{h})\). The agent stays at \(q\) when \(a_{h}=stay\) such that \(p(q|q,a_{h}=stay)=1\), as shown in Figure 4 (right). In detail, the state transition is defined by \(\delta_{h}(\cdot)\) as in Equation (7):  
\[s^{{}^{\prime}}_{h}=\delta_{h}(s_{h},prp,L(s_{l})) \tag{7}\]  
where \(prp\in PrP_{i}\) is an atomic proposition which represents a task and \(L(s_{l})\) is the label. Note that the action set \(A\) is redefined as  
\[A_{h}=\{pick_{i},deliver_{i},null_{i}\}, \tag{8}\]  
where \(move\) action is replaced by \(\{pick_{i},deliver_{i}\}\) and \(stay\) is replaced by \(null_{i}\). Here, \(\delta_{h}(\cdot)\) is different from the deterministic state transition \(\delta(\cdot)\) in \(\mathcal{A}_{\varphi}\). Given a state \(s_{h}=q\in Q\), let \(neighbours\) be the set of all possible states to which \(q\) may transit. \(\forall q^{{}^{\prime}}\in neighbours\), it has a corresponding atomic proposition \(prp\) that leads the agent to move. As described previously, each \(prp\) represents an online collection task \(\tau_{i}\) or delivery task \(\tau_{l+i}\). Let \(f(\cdot)\) define the mapping from \(prp\) to a position of the corresponding task, defined as follows:  
\[s_{l}=f(prp) \tag{9}\]  
where \(s_{l}\in\{o_{1},\ldots,o_{l},g_{1},\ldots,g_{l}\}\). After the agent chooses a neighbor to visit, it invokes the lower level to obtain a label \(L(s_{l})\), where:  
\[L(s_{l})= \begin{cases}True,&\text{if task is online compeleted},\\ False,&\text{otherwise}.\end{cases}  \tag{10}\]  
If task \(\tau_{l}\) or \(\tau_{l+i}\) is achieved online, that is, if the agent visits \(s_{l}\) within the given budget in the lower level, then \(L(s_{l})\) would return \(True\). Then, the agent transits from \(s_{h}=q\) to \(s^{{}^{\prime}}_{h}=q^{{}^{\prime}}\), written as \(\delta_{h}(q,prp,L(s_{l})=True)\to q^{{}^{\prime}}\). Otherwise, \(L(s_{l})=False\), and \(\delta_{h}(q,prp,L(s_{l})=False)\to q\). Figure 5 shows an example in which an agent chooses to move to \(q^{{}^{\prime}}_{1}\) with \(prp_{i}=pick_{i}\) from state \(q\). Here, \(\tilde{q}\) is the last state and the last transition is \(\delta_{h}(\tilde{q},prp,L(\tilde{s}_{l})=True)\to q\), where \(p\tilde{p}p\) is the chosen atomic proposition at \(\tilde{q}\) and \(\tilde{s_{l}}=f(p\tilde{p}p)\). Note that there is a special case when \(q=q_{0}\).