## 1 Introduction  
The prior information about the environment (usually a map) is always helpful for robots performing specific tasks, e.g., service, security, rescue, etc. To explore the unknown environments, the manually remote operation of the mobile robots to gather information is probably not a good option due to the possible poor communication link [1]. On the contrary, an autonomous scheme by selecting a target point to navigate in the unknown region is more popular. By repetitively selecting and navigating to the candidate point, the robot would cover the whole region [2, 3]. However, this method requires expert human experience to choose the suitable point and thus is less adaptable to different situations. Moreover, with the increasing dimensionality of the state space and the action space, the computational burden of the multi-objective optimization problem will increase rapidly. Besides, these methods do not take the localization uncertainty into account during the exploration process, which may eventually lead to an inaccurate map [4]. There are also some deep learning-based methods. Typically, neural networks are used to map the sensor data or the global occupancy maps built by Simultaneous Localization and Mapping (SLAM) into the actions of robots [5, 6, 7]. However, such frameworks require significant computational resources and may take a longer time to converge during the training phase, even fail to converge. Some works aim to combine the advantages of human experts and deep learning. For instance, the deep learning methods blend the learning scheme with the traditional planning methods helps to obtain the optional target point [8; 9; 10]. The common problem in learning-based methods is poor generalizability in new unknown surroundings [8]. Recently, [PERSON] et al. [4] firstly represented the environment as a graph and proposed a new method to autonomously navigate around the undetected area based on graph neural networks (GNNs). Compared with raw data from sensors or the real-time map, the graph reduces the computational effort and better captures the structural information about the environment. In addition, due to the dynamically changed exploration process, the hyperparameters of end-to-end deep learning are hard to tune. More recently, [PERSON] et al. [8] further designed a generalized exploration framework combining the GNNs with deep reinforcement learning (DRL) to learn a more robust policy. GNNs are used to enable network reasoning about the surroundings, and the extracted feature is then used to help DRL policy determine. Although GNNs combined with DRL show great potential for solving the autonomous exploration problem for mobile robots, there are still some problems. The above-mentioned works did not consider obstacle avoidance, which may cause serious damage. Additionally, the characteristics of the continuous state transition and the incrementally constructed exploration graph are underutilized. Moreover, to promote the effectiveness of the exploration process, the information of the interaction structure and the historical trajectory of the mobile robot are supposed to be fully utilized. To address the limitations of the traditional methods, the learning-based methods, and the GCN-based methods, we propose a novel approach combined with graph neural networks and DRL with the aim of improving the effectiveness, robustness, and generalizability. The main contributions of this paper are as follows: First, more general cases with obstacle avoidance are considered by embedding the obstacle information into the exploration graph. Second, to the best of our knowledge, the spatiotemporal information is the first time to introduce to the autonomous exploration problem, which helps improve the efficiency of exploration by learning the potential graph representation on incremental graphs.