## 3 Our Approach
### Graph-STNN
#### 3.3.3 State Encoder  
Although the spatiotemporal information can be obtained, we still expect to obtain the global information to effectively accomplish the exploration task. Hence, a state encoder is proposed to further aggregate the global and historical environment information. A similar idea of separating considerations of state and action is recently proposed by [PERSON] et al. [31]. They use two MLPs at the end of the feature aggregation to extract the state value and the state-action value, respectively. Different from them, we explicitly divide the node feature into a state feature and an action feature. Notice that the state feature is independent of the robot's actions. We construct the state encoder based on gated recurrent unit (GRU) [32]. We concatenate the initial feature, the spatial feature, and the temporal feature together as the input of the state encoder. \[\mathbf{h}_{T}=\text{concat}(\mathbf{F},\mathbf{F}_{\text{Spatial}},\mathbf{ F}_{\text{Temporal}}). \tag{14}\]  
We then use a two-layer GRU to extract the state feature. The structure of GRU is shown in Figure 4, and the convolution process of layer \(l\) in the GRU is calculated by:  
\[\mathbf{h}_{T}^{(l)}=\mathbf{z}_{T}^{(l)}\odot\mathbf{h}_{T-1}^{(l)}+(\mathbf{ 1}-\mathbf{z}_{T}^{(l)})\odot\mathbf{h}_{T}^{{}^{\prime}(l)}, \tag{15}\]  
where \(\mathbf{z}_{T}^{(l)}\) is the update gate defined as:\[\mathbf{z}_{T}^{(l)}=\sigma_{g}(\mathbf{W}_{z}^{(l)}\mathbf{h}_{T}^{(l-1)}+\mathbf{ U}_{z}^{(l)}\mathbf{h}_{T-1}^{(l)}), \tag{16}\]  
and \(\mathbf{h}_{T}^{{}^{\prime}}{}^{(l)}\) is the current memory information defined as:  
\[\mathbf{h}_{T}^{{}^{\prime}}{}^{(l)}=\sigma_{h}(\mathbf{W}_{h}^{(l)}\mathbf{h} _{T}^{(l-1)}+\mathbf{r}_{T}^{(l)}\odot\mathbf{U}_{h}^{(l)}\mathbf{h}_{T-1}^{( l)}), \tag{17}\]  
and \(\mathbf{r}_{T}^{(l)}\) is the reset gate defined as:  
\[\mathbf{r}_{T}^{(l)}=\sigma_{g}(\mathbf{W}_{r}^{(l)}\mathbf{h}_{T}^{(l-1)}+ \mathbf{U}_{r}^{(l)}\mathbf{h}_{T-1}^{(l)}), \tag{18}\]  
where \(\odot\) is the element-wise product, \(\mathbf{h}_{T-1}^{(l)}\) the hidden vector passed down from the layer \(l\), \(\mathbf{h}_{T}^{(l)}\) the hidden vector passing to the next layer and the output, \(\mathbf{h}_{T}^{(0)}=\mathbf{h}_{T}\), and \(\mathbf{W}_{z}^{(l)}\mathbf{U}_{z}^{(l)}\mathbf{W}_{r}^{(l)}\mathbf{U}_{r}^{ (l)}\mathbf{W}_{h}^{(l)}\), \(\mathbf{U}_{h}^{(l)}\) the learnable parameters in layer \(l\) of GRU. Three components work together and construct the state feature. The update gate passes the information from the previous step as well as the valuable information from the current input to the future. The reset gate determines the part of the previous message to be retained based on the current input and the output of the previous step. The current memory information consists of the current input information and the part of the previous information to be retained as determined by the reset gate. Similar to the spatial encoder, we also use an additional attention layer to further aggregate the two layers' output to the final vector, which can be represented by:  
\[\mathbf{F}_{\text{State}}{=}\alpha_{1}*\mathbf{h}_{T}^{(1)}+\alpha_{2}* \mathbf{h}_{T}^{(2)}, \tag{19}\]  
where  
\[\alpha_{l}=\frac{\exp(\mathbf{W}_{2}\mathbf{h}_{T}^{(l)}+\mathbf{b}_{2})}{ \sum_{l=1}^{2}\exp(\mathbf{W}_{2}\mathbf{h}_{T}^{(l)}+\mathbf{b}_{2})}, \tag{20}\]  
\(\mathbf{h}_{T}\) and \(\mathbf{H}_{T-1}=[\mathbf{h}_{T-1}^{(1)},\mathbf{h}_{T-1}^{(2)}]\) are the input of our state encoder in current time \(T\); \(\mathbf{h}_{T}^{(l)}\) is the output of the layer \(l\) of GRU at \(T\), \(\mathbf{W}_{2}\) and \(\mathbf{b}_{2}\) are the learnable parameters of the attention mechanism. To reduce the risk of overfitting and improve the exploration ability, we perform dropout to each node feature, concatenated by state feature, spatial features, and temporal feature. The dropout rate decreases gradually with the number of exploration steps.