## 5 Proposals
### Low-Level: MDP Learner
```
1 Initialize \(\{0_{1},\dots,o_{I}\}\) and \(\{g_{1},\dots,g_{I}\}\)
2 Formulate all tasks into scLTL formula \(\varphi\)
3\(\mathcal{A}_{\varphi}=<Q,PrP,\delta,q_{0},F>\leftarrow\) off-the-shelf tools: \(\varphi\)
4 while\(computation\_budget\)do
5 Initialize \(s_{h_{0}}=q_{0}\)
6\(s_{h}=s_{h_{0}}\)
7 while\(s_{h}\
eq q_{F}\)do
8 Select an atomic proposition \(prp\leftarrow\epsilon\)-greedy()
9\(a_{h}=prp\)
10\(s_{l}\gets f(prp)\)
11\(L(s_{l}),x\leftarrow\) low-level learner
12\(s_{h}^{{}^{\prime}}\leftarrow\delta_{h}(s_{h},prp,L(s_{l}))\)
13 Reward \(r=R(s_{h},a_{h},s_{h}^{{}^{\prime}})\)
14\(Q(s_{h},a_{h})\gets Q(s_{h},a_{h})+\alpha(r+\gamma\max_{a_{h}^{{}^{\prime}}} Q(s_{h}^{{}^{\prime}},a_{h}^{{}^{\prime}})-Q(s_{h},a_{h}))\)
15\(\pi(s_{h}^{{}^{\prime}})\gets arg\max_{a_{h}\in A_{h}(s_{h})}Q(s_{h},a_{h})\)
16\(s_{h}\leftarrow s_{h}^{{}^{\prime}}\)
17 end while
18
19 end while
```  
**Algorithm 1**FSA-RL learner
```
1 Given map information \(G=<V,Edge>\)
2 Receive \(s_{l_{0}}\) and \(s_{l_{g}}\) from high-level
3 Budget \(\gets T_{i}^{1}/T_{i}^{2}\) according to \(s_{l_{g}}\)
4\(Z_{t}\leftarrow\) observation
5\(R_{isk}\leftarrow\) stochastic simulations
6 while\(computation\_budget\)do
7\(s_{l}=s_{l_{0}}\)
8 while\(s_{l}\
eq s_{l_{g}}\)do
9 Select an action \(a_{l}\leftarrow\epsilon\)-greedy() \(s_{l}^{{}^{\prime}}\gets p_{l}(s_{l},a_{l},\cdot)\)
10 Reward \(r=R_{l}(s_{l},a_{l},s_{l}^{{}^{\prime}})\)
11\(Q(s_{l},a_{l})\gets Q(s_{l},a_{l})+\alpha(r+\gamma\max_{a_{l}^{{}^{\prime}}} Q(s_{l}^{{}^{\prime}},a_{l}^{{}^{\prime}})-Q(s_{l},a_{l}))\)
12\(\pi(s_{l}^{{}^{\prime}})\gets arg\max_{a_{l}\in A_{l}(s_{l})}Q(s_{l},a_{l})\)
13\(s_{l}\gets s_{l}^{{}^{\prime}}\)
14 end while
15
16 end while
17 Generate optimal path \(x\)
18 if\(x\) does not satisfy the given Budgetthen
19 Return \(L(s_{l})=False\) and \(\emptyset\)
20 end if
21 Return \(L(s_{l})=True\) and \(x\)
```  
**Algorithm 2**Low-level MDP learner  
As described above, the high-level learner is responsible for searching the global task strategy, while the low-level focuses on learning local paths. This hierarchical decomposition effectively transforms a complex global learning problem into several simpler local learning problems. For a single-layer RL learner, the size of the search space is \(|S_{l}|\times 2^{2l}\). In contrast, our hierarchical approach reduces the search space to \(|Q|\) for the high-level learner and \(|S_{l}|\) for low-level learner, where \(|Q|\) denotes the number of automaton states derived from scLTL. As is well-known, learning an optimal policy in a small search space is much easier than in a large search space. By decomposing the complex global learning problem into two levels, high-level for task sequencing and low-level for local path planning, the learning cost is reduced. Moreover, the proposed method reduces redundant learning cost, as it eliminates the need to re-learn a local path from \(s_{l_{0}}\) to \(s_{l_{g}}\) after it has already been successfully learned. In contrast, a single-layer RL learner would need to  
Figure 6: High- and low-level interaction for learning the optimal global path. continue learning until finding an optimal global path, resulting in higher learning costs and slower convergence.