## 3 Our Approach
### Policy Training with DRL  
The proposed Graph-STNN is performed as the value network, which helps to estimate optimal action through combining NoisyNet DQN [33]. Moreover, a multi-step strategy is used to estimate the value function by taking longer trajectories that contain more observations of multi steps' rewards, which can be represented as:  
\[Q(s_{t},a_{t},\theta,\zeta)=\sum_{k=0}^{m-1}\gamma^{k}r_{t+k}+\gamma^{m}\text{ Q}(s_{t+m},a_{t+m},\theta,\zeta), \tag{21}\]  
where \(Q(s_{t},a_{t},\theta,\zeta)\) represents the value of taking an action \(a_{t}\) in the current state \(s_{t}\); \(\zeta\) represents the Gaussian noise added in the final MLP; \(\gamma\) is the discount factor; \(m\) is the step to future observation, which is an adjustable parameter. \(\theta\) is the learnable parameter, which is soft updated as:  
\[\theta=\theta^{-}+\alpha\frac{\partial L}{\partial\theta}|_{\theta=\theta^{-}}, \tag{22}\]  
where \(L\) is the loss function, \(\alpha\) the soft update coefficient.  
The state transition process in DRL is shown in Figure 5, in which the exploration sequence consists of series of exploration graphs. Besides, each exploration graph represents a state, consisting of the adjacency matrix and the feature matrix. In each step, the robot selects the best frontier waypoint as the local target according to the evaluation of Graph-STNN. By using A* algorithm [34], the robot is able to plan a path and navigate to the target. Then, the exploration graph is refreshed as well as the state is updating to \(s_{t+1}\) with the environment reward \(r_{t}\). The robot repeats this process until finishing exploring the whole environment. Eventually, the goal is to maximize the total rewards obtained after the iteration.  
Since state transition in each episode is sequentially related in our approach, the number of steps to future observation is the total step of the whole exploration progress. Thus, we update the network once after each episode, and the policy network is trained with the loss function as follows:  
\[L=\frac{\sum_{i=1}^{T-1}\frac{1}{2}{\left[\max_{a_{i}}Q(s_{i},a_{i},\theta, \zeta)-\sum_{j=i}^{T-1}\gamma^{j-i}r_{j}\right]}^{2}}{T-1}, \tag{23}\]  
where \(T\) denotes the number of states for the entire training episode.  
Figure 5: Extracting the exploration sequence. The exploration sequence consists of a series of exploration graphs, and each exploration graph represents a state. In the current state \(s_{t}\), if the robot takes the action \(a_{t}\) to the next state \(s_{t+1}\), it will obtain the immediate reward \(r_{t}\).  
The environment reward is used to evaluate the action taken. The larger rewards obtained from the environment, the stronger tendency of the robot to adopt the strategy in the future. Furthermore, the ultimate goal of the robot is to maximize the desired reward of the whole exploration progress. Thus, to guide the selection of target points that can encourage efficient exploration, the reward function is defined as:  
\[R=-\omega_{1}*R_{1}-\omega_{2}*R_{2}-\omega_{3}*R_{3}-\omega_{4}*R_{4}-\omega_{5}* R_{5}+\omega_{6}*R_{6}, \tag{24}\]  
where \(R_{1}=\Delta U\) is the variation of the whole map's uncertainty calculated by Equation (1) from the current state to the next state; \(R_{2}\) and \(R_{3}\) are the robot's angle change and the path length, according to the local path plan result by using A* algorithm; \(R_{4}=e_{\text{next}}\) is the maximum trajectory error calculated by Equation (3) after changing to the next state; \(R_{5}\) is the total number of exploration steps from the beginning to the next state; \(R_{6}\) is the environmental exploration rate (the ratio of the known area to the entire map area); \(\omega_{1}\sim\omega_{6}\) are constants larger than zero.