## 3 Materials and Methods
### Automatic Curriculum Learning: Extension of NavACL to NavACL-Q  
In this part, we introduce our improved Automatic Curriculum Learning approach _NavACL-Q_ based on the original study _NavACL_. _NavACL_[20] is an Automatic Curriculum Learning method that specially addresses the challenges of robotic navigation. The idea of _NavACL_ is to autonomously propose tasks of suitable difficulty to reduce overall training cycle and enhance the final performance. To automatically form a curriculum, _NavACL_ uses a neural network \(f_{\pi}\) to estimate the probability of the current policy \(\pi\) solving task \(l\), with \(f_{\pi}^{\,*}(l)=0\) for certain failure and \(f_{\pi}^{\,*}(l)=1\) for certain success. The success probability \(f_{\pi}^{\,*}(l)\) is estimated based on a list of pre-defined task-specific geometric properties \(l\) relevant of map knowledge, i.e., geodesic distance between goal and initial state, agent/goal clearance, relative initial angle and etc., altogether 5 properties. The neural network \(f_{\pi}\) is updated concurrently with the training of the policy \(\pi\). For training, the network is provided with the input of \(l\) from the most recent trained task and a binary label that indicates whether or not the respective task was solved successfully. To determine which curriculum should be posed next, _NavACL_ samples on _easy_, _frontier_, and _random_ tasks with some defined probabilities. _Frontier_ tasks refer to challenging situations and _random_ tasks encourage exploration, while _easy task_ prevents the catastrophic forgetting. Since agent's ability to solve the task changes dynamically during the training, the authors of _NavACL_ used _adaptive filtering_ (AF) as a criterion to evaluate in which category the generated candidate tasks fall into. Specifically, AF proposes a set of candidate tasks characterized by their respective geometric properties \(L_{c}\) are forwarded through the network \(f_{\pi}\). Then a normal distribution is fitted to the estimated success probabilities, which is formally expressed as \(\mu_{f},\sigma_{f}\gets FitNormal(f_{\pi}^{\,*}(L_{c}))\). A task is regarded as _easy_ if \(f_{\pi}^{\,*}(l)>\mu_{f}+\beta\sigma_{f}\) and is classified as _frontier_ when \(\mu_{f}-\eta\sigma_{f}<f_{\pi}^{\,*}(l)<\mu_{f}+\eta\sigma_{f}\). The coefficients \(\eta\) and \(\beta\) are hyper-parameters. Now we illustrate our improvements and extensions of _NavACL_ algorithm to _NavACL-Q_ algorithm. We adapt the 5 geometric properties of the network to our use case as shown in Table 2. Among them, we reduce the number of geometric properties by one, and combine our proposed idea. Rather than fully relying on the map properties, it is more favorable that the input features to success probability network can be more generalizable for tasks in different domains, e.g., navigation tasks, robotic manipulation tasks, games, etc. Figure 3: An Illustration of the designed training arena. It consists of 9 total cells of different sizes and layouts. For instance, the walls and floors feature different colors and patterns, the light sources differ also in each cell. The initial pose of robot, target dolly and the obstacles are also placed with some random settings. For details, please refer to Appendix C.  
In contrast, the current input features (geometric properties) need redesigning to fit other tasks. To cope with this, we propose using the initial Q-value from the critic network, as the learned Q-value should give a good estimate on how well the initial pose is, as shown in Section 3.1. Thus, the Q-value is highly correlated with the success given the initial state. In this work, we append the estimated initial Q-value with other geometric properties together as the input to _NavACL_. Moreover, we spot that AF could result in an undersampling on the _easy_ tasks if \(\mu_{f}+\beta\sigma_{f}>1\). This could happen when either \(\mu_{f}\) approaches \(1\) or \(\sigma_{f}\) is large. The first case indicates that the agent reaches near-final performance and is thus of minor importance.