— Deep convolutional neural network (DCN) has become a common approach in visual robot self-localization. In a typical self-localization system, a DCN is trained as a visual place classiﬁer from past visual experiences in the target environment. However, its classiﬁcation performance can be deteriorated when it is tested in a different domain (e.g., times of day, weathers, seasons), due to domain shifts. Therefore, an efﬁcient domain-adaptation (DA) approach to suppress per- domain DA cost would be desired. In this study, we address this issue with a novel “domain-shift localization (DSL)” technique that diagnosis the DCN classiﬁer with the goal of localizing which region of the robot workspace is signiﬁcantly affected by domain-shifts. In our approach, the DSL task is formulated as a fault-diagnosis (FD) problem, in which the deterioration of DCN-based self-localization for a given query image is viewed as an indicator of domain-shifts at the imaged region. In our contributions, we address the following non-trivial issues: (1) We address a subimage-level ﬁne-grained DSL task given a typical coarse image-level DCN classiﬁer, in which the target DCN system is queried with a region-of-interest (RoI) masked synthesized query image to diagnosis the RoI region; (2) We extend the DSL task to a relevance feedback (RF) framework, to perform a further query and return improved diagnosis results; and (3) We implement the proposed framework on 3D point cloud imagery -based self-localization and experimentally demon- strate the effectiveness of the proposed algorithm. I. I NTRODUCTION Deep convolutional neural network (DCN) has become a common approach in visual robot self-localization. In a typical self-localization system, a DCN is trained as a visual place classiﬁer from past visual experiences in the target environment [1]. However, its classiﬁcation performance can be deteriorated when it is tested in a different domain (e.g., times of day, weathers, seasons), due to environmental and optical effects, such as occlusions, dynamic objects, and confusing features. A possible method to maintain the self- localization performance is introducing a domain adaptation (DA) module that collects a new visual experience over the entire workspace and trains a new DCN classiﬁer. However, such a naive DA method requires signiﬁcant per-domain DA cost for training data collection and retraining, which increases in proportion to the environment size. Thus, an alternative efﬁcient DA approach would be desired. We address this issue with a novel “domain-shift localiza- tion (DSL)” technique that diagnosis the DCN classiﬁer with the goal of localizing which region of the robot workspace Our work has been supported in part by JSPS KAKENHI Grant-in-Aid for Scientiﬁc Research (C) 17K00361, and (C) 20K12008. K. Tanaka is with Faculty of Engineering, University of Fukui, Japan. tnkknj@u-fukui.ac.jp (a) robot workspace and viewpoints query image/scene mapped image/scene (b) raw images and corresponding scenes (c) chain of synthetic queries Fig. 1. Given a visual experience along a viewpoint trajectory (a), we aim to diagnosis a deep self-localization network with the goal of localizing which region of the robot workspace is signiﬁcantly affected by domain shifts. Given raw images (b), effective synthetic images are generated and queried for subimage-level domain-shift localization (c). is signiﬁcantly affected by domain-shifts (Fig. 1). Such a DSL technique enables a DA module to focus the available resources on the domain-shifted region, rather than the entire workspace, which leads to signiﬁcant reduction in the per- domain DA cost for data collection and retraining. In our approach, the DSL task is formulated as a fault-diagnosis (FD) problem as in our previous study [2], in which the deterioration of the DCN classiﬁer for a given query image is viewed as an indicator of domain-shifts at the imaged region. Figure 1 shows an overview of the DSL task for a typical self-localization system based on 3D point cloud imagery. In our contributions, we tackle several non-trivial issues: (1) We address the subimage-level ﬁne-grained DSL. This is a non-trivial task because a typical DCN place classi- Fig. 2. DSL example of 3D point cloud imagery. Left: live image. Middle: map image. Right: localized domain-shifts. original PCs cropped subPCs Fig. 3. PC preprocessing ﬁer provides only coarse image-level localization cues. To address this, we introduce a data augmentation method to generate synthesized query images that are effective for subimage-level DSL. (2) We further consider an efﬁcient relevance feedback (RF) framework, derived from the ﬁeld of content-based image retrieval (CBIR) [3], to reduce the total number of queries and accelerate the diagnosis process. In the RF framework, each query result is used as a cue to perform a further query and return improved diagnosis results. (3) We implement the proposed framework on a 3D point cloud imagery -based self-localization (Fig. 2) and experimentally demonstrate the effectiveness of the proposed algorithm using publicly available NCLT dataset. II. A PPROACH This section presents the FD-based DSL approach. We ﬁrst introduce our experimental system of the deep self- localization network to clarify our DSL task (Section II- A), and formulate the DSL problem (Section II-B). Then, we describe the proposed DSL method for subimage-level diagnosis (Section II-C). Further, we extend it to a relevance feedback framework to accelerate the diagnosis process (Section II-D). Figure 4 illustrates the overview of the DSL systems. A. Experimental System Although our framework is sufﬁciently general and appli- cable to a broad class of DCN systems, our experiments will focus on 3D point cloud imagery -based DCN place classi- ﬁer. The experimental system uses an on-board LIDAR as the sole sensor system (Fig. 3), which is available in the NCLT dataset [4]. Similar with the recently developed scan context technique [1], a 2D grid (spatial resolution: 0.05m 0.05m) is imposed on the horizontal plane and datapoints that belong toeach grid cell are represented by their maximum normalized truncated height (MNTH). That is, height values of the datapoints are truncated in range [0.5, 1.5], normalized to the range [0, 1], and then max pooled. However, instead of the polar-coordinate, the bird’s eye view image coordinate is used as the coordinate for the per-cell MHNT data, to make the image-to-subimage translation (Section II-B) more straightforward. Figure 3 shows an example of original and preprocessed PCs. The DCN place classiﬁer aims to classify a given query image into one of predeﬁned place classes. It is trained via unsupervised and supervised learning stages. First, the unsu- pervised learning stage aims to partition the robot workspace into place classes. Although the problem of place partitioning (PP) is signiﬁcantly ill-posed and an on-going research topic as addressed in our recent paper [5], in this work, we use a simple standard PP method that partitions the workspace into grid cells (resolution: 40m 40m) and deﬁnes each grid cell as a place class. Next, the supervised learning stage aims to ﬁne-tune a pretrained DCN with a place-speciﬁc train image set. In implementation, we use one speciﬁc season in the NCLT dataset [4] as a train set, and as a result, we obtain around 100 place classes and 258 train images per class on average. For the train set, each image is annotated with its corresponding place label, by using GPS information that is also available in the NCLT dataset. For the ﬁne-tuning, SGD optimizer (learning rate: 104, momentum: 0.9) is used. B. Problem Formulation The DSL task is formulated as follows. Suppose a DCN is pre-trained in the procedure described in II-A, as a place classiﬁer C[]that maps a given query image xto a class- speciﬁc probability density vector (PDV) C[x]. Suppose we are given a sequence of live images (x1;;xL)along the viewpoint trajectory. Suppose we are given the ground- truth place class t[xj]for each live image xj(j2[1;L]), from an external positioning system such as structure-from- motion [6] or SLAM [7]. Then, a passive DSL task can be formulated as follows: Given a live image xwith its ground- truth place class t[x], generate an effective set of synthesized query images X=f(x) (1) from x, and diagnosis the output PDV C[x′]for each query input x′(x′2X)to estimate the likelihood Y[x′](p)of each pixel pin the imaged region being affected by domain-shifts: Y[x′] =g(x′;t[x′];C[x′]): (2) Then, an active DSL task can be formulated as follows: Given a sequence of previous queries and diagnosis results q[x′ 1];:::;q[x′ i1], where q[x′] = (x′;t[x′];C[x′];Y[x′]), select an effective next i-th query x′ i=h(q[x′ 1];:::;q[x′ i1]);x′ i2f(x1)[  [ f(xL) (3) among all the possible synthesized images to perform a fur- ther query and return improved results. To design the above generator function f, diagnosis function g, and selection Fig. 4. System integration of deep self-localization network, diagnosis, generation, and selection functions. function his not a trivial issue and is the main issue of this study. In this work, the PDV C[x]is used in two ways. First, it is translated to an inverse rank (IR) list of place classes in the descending order of place-speciﬁc probability, which can be viewed as a main output of the self-localization module. Second, the entropy E[C[x]]is computed from the PDV, and then its inverse is used as an evaluation of conﬁdence of the estimate. This conﬁdence value is used for causal analysis of the self-localization performance deterioration. That is, when the conﬁdence value is lower than a pre-set threshold 1.0, our diagnosis system simply ignores and does not reﬂect the PDV to the diagnosis result. C. Subimage-level Diagnosis Realizing a ﬁne-grained subimage-level DSL from a coarse image-level DCN is not a trivial issue. In our ap- proach, we propose to use a masked image as a synthesized image. This strategy is inspired by image masking approach [8] in the ﬁeld of image saliency, where the target DCN system is queried with an RoI masked query image to analyze saliency of the RoI region. In our approach, we create a synthesized image by replacing the height value with zero at every pixel outside the RoI. Although simple, this strategy Fig. 5. The 24 masks used for image synthesis. frequently provides a realistic synthesized image (e.g., Fig. 1(c)), as the original input MNTH image is already truncated and normalized to the range of [0, 1]. A possible direction of future research is to replace the simple binary mask with a more advanced fuzzy mask, such as adversarial erasing, to improve the quality of the subimage-level DSL. In a subimage-level diagnosis framework, one pixel may belong to multiple overlapping RoI regions, and thus receive multiple different IR values from such overlapping RoIs. Here, we fuse such multiple IRs into a single IR value, with a similar information fusion method as in our previous work in [2], which is theoretically supported by multi-modal information retrieval in [9]. That is, we start with a set of multiple diagnosis results in the form of multiple PDF vectors. Then, we average the IR r1 cfor each class cover all the related diagnosis results, and output the averaged IR (åcr1 c)1as the ﬁnal pixel-level diagnosis result. D. Relevance Feedback One possible way to improve the diagnosis performance is to increase the number of synthesized images queried. However, this naive strategy comes with high computational cost, which is proportional to the cost for DCN evaluation and the number of synthesized images queried. To accelerate the diagnosis process, we here consider an active diagnosis task with the RF framework, derived from the ﬁeld of CBIR [3]. However, a key difference of our RF task is that the original train image set is no longer accessible but only the trained DCN is available, which makes our RF task more challenging. The active diagnosis framework is formulated as follows. For simplicity, we discretize the mask space and obtain M=24RoI mask candidates, which are described in Fig. 5. Suppose we are given a set of Nlive images. Thus, we have MNRoI candidates in total. Our goal is to intelligently query a small portion (e.g., 10 percent) of effective candidates, instead of exhaustively querying for all possible RoI candidates. We implemented three different strategies for such an RoI sampling task. One is “random” strategy that samples a new RoI randomly and independently of the previous samples. The second strategy is “nearest-neighbor (NN)” strategy that is basically the same as the “random” strategy but if the previous query is “successful”, it samples a new RoI from the same image as the previous query, rather than from all the input images. Here, the condition for “successful” query is that its rank value should be equal or lower than a pre- learned threshold (1.0). The third strategy is “spatial NN” strategy that is basically same as the NN strategy, but if the previous query is “successful”, it prioritizes the non-visited RoI candidates by spatial nearness jpnewpprevj1of each RoI’s center pnewto the previous RoI’s center pprev, and samples a non-visited RoI with the nearest spatial location pnew=argmin pnewjpnewpprevj1. III. E XPERIMENTAL EVALUATION We evaluated the proposed methods using NCLT cross- season dataset [4]. The NCLT dataset is a large-scale, long- term autonomy dataset for robotics research collected at the University of Michigan’s North Campus by a Segway vehicle robotic platform for 15 months, 27 sessions, in both indoor and outdoor. During the navigation, the vehicle encounters several different environment changes, includ- ing car parking, falling leaves, building construction, and furniture movement. These environmental changes can be main causes of self-localization performance deterioration. In the current experiments, we used three pairings of query (i.e., test) and map (i.e., train) seasons from the NCLT dataset, (2012/03/31, 2012/01/22), (2012/08/04, 2012/03/31), and (2012/11/17, 2012/08/04), and for each dataset, a size 1,500 image set is randomly sampled. Given a wide-range 70m radius view provided by the 3D point cloud imagery, we crop the image into 12.8m 12.8m image region with spatial resolution of 0.05m, which is centered at the robot viewpoint, and whose x-axis is aligned with the heading direction of the robot. Figure 6 shows example diagnosis results. As shown in Fig. 1(c), the proposed active diagnosis framework generates an effective chain of queries based on the previous query responses. The locations of each synthesized image in the sequence with respect to the bird’s eye view coordinate is visualized in Fig. 1(a). Effectiveness of such an active diag- nosis technique will be shown in the following quantitative evaluation. For quantitative performance evaluation, we prepare a ground truth diagnosis result for each test image. The ground truth is in the same data structure as the diagnosis result image, but the pixel value is determined by using the ﬁne- grained GPS and RANSAC based point cloud registration and precise pixel-wise pairwise 3D point cloud compari- son, based on RANSAC scoring using subPCs truncated in height [-2, 2]. Suppose we are given a pre-deﬁned hyper- parameters T hand Height . First, each grid cell (spatial resolution: 0.05m 0.05m) in the ground-truth data is labeled as “domain-shifted” if height difference between the live and map images at that grid cell is greater than the preset Height value. Then, an image region in the ground-truth data is labeled as “domain-shifted” if the ratio of grid cells with “domain-shifted” labels within the region exceeds the threshold T h. For the performance index, we used pixel-wise precision- recall. This index is computed as follows. First, every query image is discretized into a regular grid with the spatial resolution of 0.3m. Then, all the cells in the image are sorted Fig. 6. Examples of DSL results (left: query, middle: map, right: pixel-level diagnosis result). in the descending order of the likelihood of domain-shifts. Then, 51-point interpolated mean-Average-Precision (mAP) is computed over the top-50% recall points. Figure 7 shows the performance results for a speciﬁc (default) setting of the hyper-parameters, Height andT h. For performance comparison, we implemented and tested six different strategies: “FULL”, “RANDOM”, “RANK >1”, “RANK >A VG”, “RANK >1 CHAIN”, and “RANK >1 CHAIN SPATIAL”. The “FULL” method is the subimage- level DSL method that utilizes all the available RoI can- didates in exhaustive manner. The “RANDOM” method randomly samples and uses 10% of RoI candidates. The other four methods also samples and uses 10% of RoI candidates, but in an intelligent manner within the relevance feedback framework explained in Section II-D. Among them, the “RANK >1” method samples the next RoI from the same image if the current query is successful, otherwise randomly. The “RANK >A VG” method is similar with “RANK >1”, but 0.16 0.18 0.2 0.22 0.24 0.26 0.28 0.3 0.32 0.34 0 20 40 60 80 100 120 140mAP (top-50%, 51-points interpolated) height (h): h/256*4-2[m]FULL RANDOM RANK>1 RANK>AVG RANK>1 CHAIN RANK>1 CHAIN SPATIAL(a) 0.16 0.18 0.2 0.22 0.24 0.26 0.28 0.3 0.32 0.34 50 60 70 80 90 100mAP (top-50%, 51-points interpolated) threshold (th): th [%]FULL RANDOM RANK>1 RANK>AVG RANK>1 CHAIN RANK>1 CHAIN SPATIAL (b) Fig. 7. Performance results for the default setting ( T h: 80, Height : 64). the deﬁnition of “successful” is different, that is, average of the rank values of all the queries is used as the threshold instead of the default deﬁnition (1). The “RANK >1 CHAIN” method is similar with “RANK >1”, but the next query is sampled from the same image as long as the query response is “successful”. The “RANK >1 CHAIN SPATIAL” is similar with “RANK >1 CHAIN”, but the NN method is used to prefer the spatial nearest queries, as explained in II-D. As can be seen, the proposed active diagnosis method “RANK >1 CHAIN SPATIAL” or “RANK >1 CHAIN” out- performs almost all the other methods, except for the exhaus- tive “FULL” strategy, and it is competitive to the exhaustive strategy despite the fact that the proposed method is ten times more efﬁcient in terms of the number of synthesized images queried. On the other hand, the “RANDOM” strategy often fails to generate effective samples, which led to the worst performance result. On the other hand, the proposed method was successful in generating an effective chain of queries that yield much higher performance. IV. C ONCLUSIONS In this study, we addressed the novel problem of domain- shift localization by fault-diagnosing a deep self-localization network. The key idea is that the deterioration of DCN- based self-localization for a given query image is viewed as an indicator of domain-shifts at the imaged region. Inour contributions, several non-trivial issues were addressed: (1) We addressed a subimage-level ﬁne-grained DSL task given only an typical coarse image-level DCN classiﬁer. (2) We extended the DSL task to a relevance feedback framework, in which each query response is used as a cue to make the next query more effective. (3) We implemented the proposed framework on 3D point cloud imagery -based self-localization and experimentally demonstrated the effec- tiveness of the proposed algorithm.