— Graph-based scene model has been receiving in- creasing attention as a ﬂexible and descriptive scene model for visual robot self-localization. In a typical self-localization application, objects, object features, and object relationship in an environment map are described respectively by nodes, node features, and edges in a scene graph, which are then matched against a query scene graph by a graph matching engine. However, its overhead for computation, storage, and communication, is proportional to the number and feature dimensionality of graph nodes, and can be signiﬁcant in large-scale applications. In this study, we observe that graph- convolutional neural network (GCN) has a potential to become an efﬁcient tool to train and predict with a graph matching engine. However, it is non-trivial to translate a given visual feature to a proper graph feature that contributes to good self- localization performance. To address this issue, we introduce a new knowledge transfer (KT) framework, which introduces an arbitrary self-localization model as a teacher to train the stu- dent, GCN-based self-localization system. Our KT framework enables lightweight storage/communication by using compact teacher’s output signals as training data. Results on RobotCar datasets show that the proposed method outperforms existing comparing methods as well as the teacher self-localization system. I. I NTRODUCTION Graph-based scene model has been receiving increasing attention as a ﬂexible and descriptive scene model for visual robot self-localization. In a typical self-localization application, objects, object features, and object relatonship in an environment map are described respectively by nodes, node features, and edges in a scene graph, which are then matched against a query scene graph by a graph matching engine. Such a scene graph model is sufﬁciently general and applicable to various types of scene data. For example, in [1], an input scene is segmented into semantic segments which serve as graph nodes and are connected with their neighbors via graph edges. An alternative representative example is to model a view-sequence as a scene graph whose nodes are image frames and edges connect successive image frames [2]. In our experiments, we also focus on such a view- sequence -based scene graph representation (Fig. 1). This paper is concerned with the scalability of a graph- based representation to large-scale applications, such as long- term map-learning [3]. Firstly, the storage cost for a scene graph is proportional to the number and dimensionality of graph nodes (i.e., #graphs, #nodes per graph, dimensionality of node feature), and grows rapidly with the environment Our work has been supported in part by JSPS KAKENHI Grant-in-Aid for Scientiﬁc Research (C) 17K00361, and (C) 20K12008. The authors are with Graduate School of Engineering, University of Fukui, Japan. ftakedakoji00, tanakakanji g@gmail.com Fig. 1. In this study, a general framework of GCN-based self-localization is proposed and veriﬁed in a speciﬁc application of view-sequence -based scene graphs. The bottom panel illustrates nodes and time/attribute edges of a scene graph by circles and horizontal/vertical line-segments. size. Moreover, the computational cost for the graph match- ing engine is proportional to the graph size and often requires approximations such as dimension-reduction to reach rea- sonable computation speed. Therefore, a new framework is desired for improving efﬁciency without sacriﬁcing accuracy of a scene graph -based self-localization system. In this study, we observe that graph-convolutional neural network (GCN) [4] has a potential to become an efﬁcient tool to train and predict with a graph matching engine. GCN is recently developed, one of most widely used graph neural networks. In GCN, a graph-convolutional operation is introduced to produce graph features, which are then passed to a graph-summarizing operation to produce higher- order graph features. GCN has been successfully applied to various kinds of graph data applications, including chemical reactivity [5] and web-scale recommender systems [6]. GCN training and prediction process is computationally efﬁcient, whose complexity is in the order of O(m+n)where mand nare #edges and #nodes. From the perspective of visual robot self-localization, a non-trivial issue is how a robot can translate a given visual feature to a proper graph feature that contributes to good self- localization performance. This is a non-trivial issue because typical visuial features are originally designed for visual self- localization task and can often be deteriorated when they are directly used as graph features. We tackle this issue by introducing a novel knowledge transfer (KT) framework, which introduces an arbitrary self-localization model as a teacher to train the student, GCN-based self-localization system. Our KT strategy is inspired by the standard KT framework of knowledge distillation [7]. Our feature learning strategy is derived from the ﬁeld of multi-media information retrieval (MMIR) [8]. Our contributions in this paper are summarized as follows. (1) We ﬁrst study the use of GCN for augmenting self- localization performance, while suppressing computation, storage and communication cost. (2) We formulate the ver- satile framework for feature learning by introducing a novel teacher-to-student knowledge transfer model. (3) Results on RobotCar datasets show that the proposed method clearly outperforms the existing comparison methods as well as the teacher self-localization system. II. R ELATED WORK Visual robot self-localization is one of most important issues in mobile robotics and has been studied in many different contexts, including multi-hypothesis pose tracking [9], map matching [10], image retrieval [11], view-sequence matching [3], etc. Our self-localization scenario is most closely related to the view-sequence matching scenario, which takes a short-term live view-sequence as query and searches for corresponding part in the map view-sequence. Unlike many existing works, the proposed approach for- mulates self-localization as a classiﬁcation problem, which (1) Partitions the robot workspace into place classes, (2) Trains a visual place classiﬁer from a class-speciﬁc training set, and (3) Predicts the place class for a given query image with the pre-trained classiﬁer. In this line of researches, it is straightforward to train a deep convolutional neural network (DCN), as a visual place classiﬁer, as demonstrated in our previous study [12]. More recently, in [13], DCN has been successfully used for visual place classiﬁcation in an alternative context of 3D point cloud -based self-localization with the scan-context image representation. However, the current study is different from these existing studies in the following two aspects. (1) We focus on the problem of graph- based view-sequence representation, which can deal with interactions between image frames. (2) We further address knowledge transfer from a teacher self-localization model to the student GCN-based self-localization system. Graph neural network (GNN) has attracted recent interest in pattern recognition community as a ﬂexible and efﬁcient model for pattern recognition and machine learning. GCN is a most widely used GNN, which generalizes the traditional convolution to data of graph structures. In previous studies, GCN has been successful in applications where traditional DCN are very inefﬁcient or not applicable [5], [6], [14]. On the other hand, in this study, we revisit the traditional important application of visual robot self-localization and aim to augment and outperform existing solutions.III. A PPROACH A. System Overview We formulate self-localization as a classiﬁcation prob- lem [12], which consists of three distinct stages: (1) Place partitioning that aims to partition the robot workspace into a collection of place classes; (2) Mapping (i.e., training) that takes a visual experience with ground-truth viewpoint information [12] collected in the workspace as training data and trains a visual place classiﬁer; (3) Self-localization (i.e., testing) that takes a query graph representing a short-term live view-sequence with length T, and predicts the place class. We suppose that the original training data is no longer available at the test stage, but only the trained classiﬁer is available, to save the overall long-term storage cost. We do not rely on any post-veriﬁcation stage, such as RANSAC post-veriﬁcation [15], in order to reduce the over- all computational burden. Nevertheless, experimental results will show that the proposed framework is surprisingly robust against outliers in measurements. We use a standard grid-based place partitioning method to deﬁne place classes [12]. First, a 2D regular grid is imposed on the robot workspace (i.e., moving plane). Then, each grid cell is viewed as a place class. It should be noted that the place partitioning could be improved by introducing state-of- the-art place partitioning techniques, as we also demonstrated in our recent study [16]. B. Graph Matching Engine Figure 2 depicts the pipeline of the graph matching engine. It consists of three modules: 1)A scene graph descriptor that translates an input length Tview-sequence to a scene graph. 2)A knowledge transfer module that transfers knowledge from a teacher self-localization model to the student GCN-based self-localization system. 3)A supervised learning module that takes length Tview- sequences as training samples, and trains a classiﬁer that predicts the place class given a query sample. These three modules are detailed in the subsequent subsec- tions III-C, III-D, and III-E. We follow the supervised learning procedure to train the scene graph classiﬁer. In the mapping stage, a collection of overlapping sub-sequences with the same length Tis sam- pled from the visual experience, and divided into place-class- speciﬁc training sets according to the available viewpoint information as well as the pre-deﬁned place partitioning. We emphasize that all the training set can be thrown away once after training the GCN classiﬁer. Since our framework uses overlapping sub-sequences as training data, the total data size as well as the number of graph nodes can be signiﬁcantly larger than the original training view-sequence. Nevertheless, the training data has no impact on the storage overhead after compressing the training data into a GCN classiﬁer. a b c Fig. 2. Pipeline of graph matching engine. (a) Scene graph descriptor. (b) Knowledge transfer from teacher self-localization model. (c) Supervised learning of GCN-based self-localization network. C. Scene Graph Descriptor We build the domain invariance by controlling the length and intervals of map/query (i.e., training/testing) view- sequences (Fig. 3). Firstly, the same length Tis assumed for all training/testing view-sequences, to build invariance across different domains. Moreover, these Tframes are selected so that travel distances between successive frames are approximately equal to a pre-set value, 2[m], to build invariance against the vehicle’s ego-motion speed. We will experimentally show that the above strategy contributes to good self-localization performance. It should be noted that the GCN theory is not restricted to such homogeneous graphs (with the same size and shape), and extending this approach to deal with heterogeneous graphs is a future direction of research. We build a collection of Kdifferent image feature ex- tractors F1,,FK, by combining several image processing techniques, such as NetVLAD [17], Canny operation [18], depth regression [19], and semantic segmentation [20], as shown in Section III-D. Then, each graph node n= (t;k;fk[t]) represents an attribute feature vector fk[t]extracted by k-th extractor from t-th image frame. On the other hand, two types of graph edges, time edge and attribute edge, are employed in our approach (Fig. 3). A time edge e= (t;t+1;k)connects two graph nodes with successive time indexes t;t+1with the same attribute index k. An attribute edge e= (t;k1;k2) connects two graph nodes with different attribute indexes k1;k2with the same time index t. Fig. 3. Time and attribute edges. D. Knowledge Transfer We now discuss how a robot can translate input view- images to graph features that are required by GCN train- ing/testing. A naive and intuitive way is to use visual features that are originally designed for visual self-localization tasks, directly as graph node features. Designing visual features has been a dominant topic in recent self-localization literature [17], [21], [22]. Many works have proposed compact yet discriminative visual features. Recent examples include autoencoder-based method [21], GAN-based method [22], CNN-based method [17]. In particular, NetVLAD [17] is recently developed and one of most widely used visual features in computer vision and robotics, which is used also in our experiments as a comparing method. A concern is that typical visual features are not optimized for graph convolutions. In theory, their good performance in the original applications does not guarantee their good performance in GCN-based self-localization. In fact, results in our experiments will show that the self-localization perfor- mance is deteriorated when they are directly used as graph node features in a GCN-based self-localization. To address this, we propose to use class-speciﬁc proba- bility distribution vector (PDV) output by the teacher self- localization model as the training data. This strategy is sim- ilar in concept with the standard KT approach of knowledge distillation [7]. It should be noted that the PDV representation ensures the versatile applicability to a broad range of teacher output signals, including tf-idf score in bag-of-words image retrieval [23], RANSAC scores in a post-veriﬁcation stage [24], as well as mean average intersection-over-union in an object matching system [25]. We convert a node image Ito a graph feature vector f=M(Y(I)) (1) by using a teacher self-localization system Yand an image- to-feature translator M. The conversion procedure is as fol- lows: (1) Input the node image Ito a teacher self-localization system Y, and obtain the output PDV signal o=Y(I)from the teacher system. (2) Map the output PDV oto a graph feature vector f=M(o). This strategy is similar in concept with our previous study in [26], where outputs from a teacher self-localization system are used as visual features for a student self-localization system. The above two functions Y;Mare detailed in the following. We build four teacher systems Y1,Y2,Y3, and Y4(Fig. 4), by combining four different image ﬁlters Zi(I) (i2[1;4])with a single nearest-neighbor (NN) [27] -based VLAD matching engine Yo, in the form: Yi=Yo(Zi(I)): (2) An NN matching engine represents each place class by a collection of VLAD descriptors, each of which has been extracted from each image in the class-speciﬁc training set. Then, it computes the image-to-class distance (i.e., dissimilarity) as the distance from a given query VLAD vector to the nearest-neighbor in the class-speciﬁc VLAD vectors. The image ﬁlters are implemented as follows. Z1 is a simple identity mapping function (Fig. 4 “raw image”). Z2is Canny image ﬁlter that converts an input image to a gradient-emphasized image (Fig. 4 “canny”). Z3is depth image regressor that is trained in an unsupervised manner and aims to predict a depth image from a monocular image (Fig. 4 “depth”). Z4is semantic segmentation ﬁlter [20] that converts an input image to a semantic label image, whose pixel color represents the pixel-wise class label that is deﬁned in the original color palette in [20] (Fig. 4 “semantic”). Fig. 4. Images and feature vectors generated by individual image ﬁlters and image-to-feature translators. We consider and investigate the following four types of mapping functions M1(),M2(),M3(), and M4().M1is an identity mapping that can be used only when Z=Z1 (Fig. 4 “NetVLAD vector”). M2is a class-speciﬁc distance value vector, in which each c-th place class for the vector is assigned with the L2 norm from the query feature to its nearest neighbor feature in the class (Fig. 4 “match distance”). M3uses a ranking function, in which each c-th place class for a given PDV is assigned with a rank value by sorting the elements in a PDV in ascending order of the probability scores, and then the vector of rank values is used as the node feature vector (Fig. 4 “rank”). This ranking based representation is inspired by the recent ﬁndings in the ﬁeld of MMIR [8], in which rank values are used for information fusion across different domains (e.g., individual users’ domains of MMIR). M4is different from M3only in that inverse-rank values are used in place of the rank values (Fig. 4 “inverse rank”). This strategy can be viewed as an application of inverse rank fusion methods [28] to our graph feature fusion task. E. Self-localization The training stage of the self-localization system basically follows the standard training procedure for GCN in [4]. A Graph is represented as G= (V;E)where Vis the set of nodes, and Eis the set of edges. Let vi2Vdenote a node and ei j= (vi;vj)2Edenote edge pointing from vjtovi. We deﬁne all graphs as undirected graph, i.e., a special case of directed graphs where there is a pair of edges with inverse directions if two nodes are connected. The neighborhood of a node vis deﬁned as N(v) =fu2Vj (u;v)2Eg. Each node vhas a feature vector h2RD. The representation of a node vis generated by aggregating its own features hvand features hu(u2N(v)) of neighboring nodes N(v)that are connected to vvia edges, in the following steps. First, each node receives features from the neighbors N(v), which are then summarized via SUM operation. Then, these summarized features are passed to single layer fully connected neural network, which is followed by non-linear ReLU transformation, in the form: hnew i=ReLU (W(å u2N(vi)[vihu)); (3) where Wis weight matrix W2RDD′to apply linear transformation. DandD′respectively are the dimensions of feature vector before and after applying this operation. The operation at l-th GCN layer is generalized in the form: h(l) i=ReLU (W(l1)(å u2N(vi)[vih(l1) u )) (4) This operation is applied to all nodes. In this way, all node features are updated. This operation is repeated Ltimes, where Lis the number of layers and set L=2in this study. Finally, features of all nodes are summarized by an averaging operation, and then passed to an fully-connected (FC) softmax operation: p=Softmax (1 jVjå u2VhK u): (5) Kis the number of ﬁnal GCN layers, and huis the feature at the node uoutput by the ﬁnal GCN layer. Our imple- mentation is based on the Deep Graph Library with Pytorch backend in [4]. IV. E VALUATION We evaluated the proposed algorithm on RobotCar dataset [29]. Table I shows characteristics of the dataset. For the grid-based place partitioning in III-A, we used a 1417 grid with the resolution of 0.1 degree in latitude and longitude (approximately 110[m] 70[m] resolution). As a result, the number of training images per place class becomes 80-90 on average. A place class is eliminated from the training/testing set if the number of images belonging to the class equals to or lower than 6. Every image is cropped by a 1080 800 region to eliminate those regions that are occluded by the vehicle itself (i.e., 100 pixels from each size, and 180 pixels from the bottom). The length of a map/query view-sequence is set T=10. The intervals between succes- sive frames in travel distance are set approximately 2[m]. Those sequences that span adjacent places are removed from training/testing set. Self-localization performance in terms of top-1 accuracy is shown in Tab. II. For comparing method, we use NN matching of NetVLAD descriptor [17], adapting the imple- mentation in [30]. For the dissimilarity measure, the image- to-class distance deﬁned in III-D is reused. The number of GCN layers is 2. The feature dimension- ality of GCN layers are set as C, 256, 256 and Cfor a size Cclass set. For the node summarization, SUM and ReLU operations are used. The number of epochs, batch size, and learning rate are set as 5, 32, and 0.001. The GCN operation works in CPU using Intel(R) Xeon(R) GOLC 6130 CPU @ 2.10GHz. The self-localization performance is measured byTABLE I DATASET CHARACTERISTICS dataset ID weather #images detour roadworks 15-08-28-09-50-22 (A) sun 31,855   15-10-30-13-52-14 (B) overcast 48,196   15-11-10-10-32-52 (C) overcast 29,350  ◦ 15-11-12-13-27-51 (D) clouds 41,472 ◦ ◦ 15-11-13-10-28-08 (E) overcast 42,968   TABLE II TOP-1ACCURACY A B C D E A 92.3 88.5 80.8 88.4 B 92.4 97.3 87.3 97.6 C 91.5 94.8 90.9 95.7 D 88.2 88.0 93.2 91.6 E 92.7 97.4 99.2 94.4 TABLE III PERFORMANCE COMPARISON Method Average Top-1 accuracy ɹ Ours 92.1 NetVLAD 87.6 Ours w/o edge 86.7 Ours w/o attribute edge 91.6 Ours w/o time edge 91.3 Ours w/o attribute node/edge 86.9 TABLE IV RESULTS FOR DIFFERENT CHOICES OF IMAGE FILTERS . A B C D E A92.3 92.8 93.2 83.188.5 90.0 89.9 85.080.8 82.0 83.3 79.788.4 90.2 91.5 82.2 B92.4 92.3 91.1 86.997.3 98.5 96.7 95.286.6 85.0 84.2 81.697.8 97.6 97.6 95.5 C91.5 91.2 91.7 83.694.8 94.9 94.7 91.790.9 89.8 90.7 88.795.7 95.8 95.8 94.0 D88.2 87.6 87.2 78.088.0 87.7 87.3 87.193.2 87.6 93.2 92.491.6 91.9 92.6 88.4 E92.7 93.3 94.5 83.997.4 97.4 96.9 93.399.2 99.0 99.2 97.394.4 93.1 94.1 91.1 top-1 accuracy. The timing result for training was 170 [sec] for 5 epochs and 31,835 training samples. The time cost for prediction is 15.5 [msec] per query graph. It can be said that the proposed method is computationally very efﬁcient and realtime capable. Table IV shows results for the proposed method with dif- ferent choices of the image ﬁlter Zias well as the comparing method. From top to bottom, each line corresponds to the image ﬁlters Z1,Z2,Z3, and Z4. By comparing the different combinations of image ﬁlters, the combination of Z1andZ4 yielded the best performance. It can be seen that the proposed method outperforms the comparing method for almost all settings considered here. As an ablation study, we performed two experiments. In the ﬁrst case, we modiﬁed the scene graphs by removing all the edges, then train and test the GCN. In the second case, we modiﬁed the graph topology by removing one of the two types of edges, either time or attribute edge. Table III shows results for the ablation study. It is apparent that graph with both time and attribute edges work signiﬁcantly better for almost all cases. Speciﬁcally, the use of time edges often contributed to improve the robustness against partial occlusions against illumination changes between training and test domains. In consequence, the proposed GCN framework was successfully shown to solve the variety of problems by integrating the available cues from different image ﬁlters, as well as time and spatial graphs. V. C ONCLUSIONS In this paper, we ﬁrst explore the use of GCN for augment- ing visual robot self-localization systems, and improve the self-localization performance without sacriﬁcing efﬁciency in computation, storage and communication. A novel versatile KT framework is introduced for transferring knowledge from a teacher self-localization model to integrate the available cues from different image ﬁlters as well as time and spatial contextual information. Results on RobotCar datasets show that the proposed method clearly outperforms the existing comparing methods as well as the teacher self-localization system. In the scope of this paper, we worked with view- sequence -based scene graph representaion, but other scene graph representations may be used such as attribute grammar -based scene graphs [31]. Another direction is to consider more general heterogeneous scene graphs to deal with map/query scene graphs with variable sizes and shapes.