— A typical assumption in state-of-the-art self- localization models is that an annotated training dataset is available in the target workspace. However, this does not always hold when a robot travels in a general open-world. This study introduces a novel training scheme for open-world distributed robot systems. In our scheme, a robot (“student”) can ask the other robots it encounters at unfamiliar places (“teachers”) for guidance. Speciﬁcally, a pseudo-training dataset is reconstructed from the teacher model and thereafter used for continual learning of the student model. Unlike typical knowl- edge transfer schemes, our scheme introduces only minimal assumptions on the teacher model, such that it can handle various types of open-set teachers, including uncooperative, un- trainable (e.g., image retrieval engines), and blackbox teachers (i.e., data privacy). Rather than relying on the availability of private data of teachers as in existing methods, we propose to exploit an assumption that holds universally in self-localization tasks: “The teacher model is a self-localization system” and to reuse the self-localization system of a teacher as a sole accessible communication channel. We particularly focus on designing an excellent student/questioner whose interactions with teachers can yield effective question-and-answer sequences that can be used as pseudo-training datasets for the student self-localization model. When applied to a generic recursive knowledge distillation scenario, our approach exhibited stable and consistent performance improvement. I. I NTRODUCTION Self-localization, that is, the problem of classifying a view image into predeﬁned classes, is a fundamental problem in visual robot navigation and has important applications, including scene understanding, map building, and path plan- ning. Most existing solutions, ranging from image retrieval engines [1] to ConvNet image classiﬁers [2], aim to con- struct high-quality self-localization models using annotated training datasets for supervision. In a familiar workplace, this assumption actually holds, since pseudo-ground-truth viewpoints or place-labels can be reconstructed from a visual experience via structure-from-motion [3]. Several state-of- the-art techniques can achieve excellent performances in such supervised settings, as discussed in [4]. However, this is not the case in an unfamiliar workspace where no supervision is available. Therefore, the problem of self-localization remains largely unresolved. In this study, teacher-to-student knowledge transfer (KT) or distillation (KD) for general open-world and distributed robot systems such as open-world distributed robot localiza- tion (OWDL) is considered as an alternative training setup. Our work has been supported in part by JSPS KAKENHI Grant-in-Aid for Scientiﬁc Research (C) 20K12008 and 23K11270. ∗K. Tsukahara, K. Tanaka, and D. Iwata are with Robotics Coarse, Department of Engineering, University of Fukui, Japan. {ha200973, tnkknj, ha200213 }@u-fukui.ac.jp Fig. 1. Our goal is to design an excellent questioner (i.e., student) that can obtain a sequence of question xtand answer ytpairs via interactions with a blackbox teacher y=f(x)such that the obtained samples {(xt,yt)}T t=1 can then be used for supervised learning or distillation of the student self- localization model. This is relevant to the model inversion problem, except that we are targeting generic self-localization models. We observe that when humans travel around the open world, they often ask the people they meet in unfamiliar places for guidance. Therefore, we propose a similar knowledge- transfer scheme in which a student robot can consider other robots encountered in unfamiliar places as potential teachers and ask them to transfer knowledge about those places. Vari- ous types of teacher robots may exist. Some of them may be cooperative, whereas others may not. Some of them may be trainable (e.g., differentiable neural networks [2]), whereas others may not (e.g., image retrieval engines [1]). Some may have a known architecture, whereas others may have a blackbox architecture (i.e., data privacy). Therefore, we propose introducing minimal assumptions regarding potential teacher robots. This scenario is most relevant to the replay scheme [5] in the continual-learning (CL) ﬁeld in which the model is supposed to be the primary source of knowledge to be transferred, and we wish to have no or minimal dependence on the availability of other training data or metadata. We are interested in CL, because there is always a risk of catastrophic forgetting [6], and because each time the student robot learns domain-/class-/vocabulary-speciﬁc knowledge from a new teacher, it risks forgetting the knowledge learned from previous teachers. The replay scheme can be applied to any architecture; therefore, we observe that it is more suitable for blackbox teachers than other typical CL methods, such as regularization [7] or dynamic architecture [8]. However, naive replay schemes assume the direct memorization of parts of the training sample set [9] or the availability of teacher-speciﬁc sample generators [10], which signiﬁcantly reduces the scope of applicability. To address this issue, we focus on the problem of gener- ating annotated training samples from a pre-trained teacher self-localization model (Fig. 1). The problem of generating training samples from a given model (“Model Inversion” in Fig. 1) can be considered as an inverse problem of supervised learning (“Supervised Learning” in Fig. 1) [11] and is related to recent studies on model inversion (MI) [12] and dataset reconstruction (DR) [13]. However, these techniques focus on image generation and similar generative tasks and cannot be directly applied to self-localization tasks. Inter-robot com- munication channels (that could be used for KT) are usually limited [14], which prohibits the sending/reception of non- lightweight data, such as image data [12], [13]. Furthermore, teacher robots can be of various types, as aforementioned. Rather than relying on the availability of private data of teachers as in existing MI/DR methods, we propose to exploit an assumption that holds universally in self-localization tasks: “The teacher model is a self-localization system,” and to reuse the self-localization system as a sole accessible teacher-side communication channel. Starting from a non- data-free scheme for benchmarking, we consider several KT schemes, those based on random walks in the input query space and those based on Entropy-based criteria, as well as a mixup of these schemes, and show that different schemes have different advantages and disadvantages. When applied to a generic recursive KD scenario [15] (Fig. 3), our approach exhibited stable and consistent performance improvement. The main contributions of this work are: (1) We tackle the challenging problem of training a self-localization model in an unknown workspace where no annotated training dataset is available. (2) We introduce a practical teacher-to- student data-free knowledge transfer (DFKT) framework that introduces only a minimal assumption: “The teacher model is a self-localization system,” and reuses the teacher’s available self-localization model as a sole accessible teacher-side com- munication channel; (3) We ﬁnd that the proposed scheme consistently achieves stable performance improvements in a realistic recursive KD setup. II. O PEN-WORLD DISTRIBUTED ROBOT LOCALIZATION (OWDL) We begin with reviewing the well-known supervised learn- ing (SL) setting of a single robot self-localization model y=f(x)that aims to classify an input scene (embedding) xinto one yof predeﬁned place-classes C(y∈C), via supervised learning using a training set in the form of {(x,y)}. The experimental scenario follows the multi-teacher multi-student KT scenario using the NCLT dataset [16], derived from our previous work [4]. The NCLT dataset contains long-term navigation data from a Segway robot with an onboard monocular front-facing camera navigating a university campus across 27 seasonal domains. The datasetcan provide a pairing of the view image ( xin Fig. 1) annotated with the ground-truth place class ( yin Fig. 1) for each viewpoint on the robot’s viewpoint trajectory for each of the 27 domains/sessions. The deﬁnition of the place-class setCis assumed to be provided in advance and is consistent across robots. It is noteworthy 1) that unlike common well- deﬁned class deﬁnitions, such as country/region/postal codes, deﬁning a place-class in a robot-centric coordinate system is itself an open issue [17], and 2) that students do not necessarily have access to meta-information such as the number of teachers, performance of individual teachers, number and ID of unseen unfamiliar place-classes, relative pose of teachers, and physical means of communication. Nevertheless, for simplicity, in this study, we do not explore these issues. We adopt simple grid-based place partitioning, and partition the the workspace into a 10×10grid of 100 place-classes in the bird’s eye coordinate system. We then describe a non-data-free KT [5] as a natural extension of the above setting. The basic idea is to maintain a portion of the training sample set used for SL as part of the model rather than discarding it after SL [9]. When the robot encounters a new teacher, the training samples contained in the teacher and student models are merged and used as training samples to train a new version of the student. We now introduce a DFKT [18] as an extension of the above KT. The DFKT differs from the above KT only in that it generates (pseudo) training samples from the teacher and previous student models, via student-teacher-interactions (Section III), rather than assuming that training samples are included in the teacher/student model. We speciﬁcally consider a typical communication proto- col (Fig. 2) in which a student sends a questioner (pro- gram code) gto a teacher, and the teacher replies with samples {(xt,yt)}T t=1. Speciﬁcally, the questioner has T′ (≥T)question-and-answer sessions with the teacher. At each question-and-answer step t′(∈[1,T′]), the questioner asks the next question xt′and the teacher returns the cor- responding answers (e.g., yt′). Except for the initial step t′=1, the questioner is allowed to access the question-and- answer history (x1,y1)···(xt′,yt′). After the sequence of T′ questioner-teacher-interactions is ﬁnished, the teacher sends to the student a collection of selected samples {(xt,yt)}T t=1, which is then used to train the student model via normal distillation process. Therefore, it is reasonable to measure the cost of KT based on the number T(≤T′)of samples sent back from the teachers to the students. Our goal is to improve the trade-off between self- localization performance (top-1 accuracy) and KT cost T (teacher-to-student communication cost). A. Self-localization Model Although our framework is sufﬁciently general and ap- plicable to general black-box teachers, we build on our previous work [19] that is categorized as L2-based KD in a recent survey [20] and develop a speciﬁc experimental system using graph neural embedding [21] and semantic scene graph [22] (Fig. 3). The embedding model is based Fig. 2. A typical communication protocol: (1) The student robot sends the questioner (program code) gto the teacher robot. (2) The questioner g repeats the question and answer session with the teacher. (3) The student robot receives the selected samples {(xt,yt)}T t=1via the KT channel. In this work, we aim to obtain a small number of good samples. on scene graphs, which have proven their effectiveness in the ﬁeld of visual self-localization [23], [24]. It is trained as a scene graph classiﬁer that maps the input scene to the class-speciﬁc probability map. The same graph convolutional neural network (GCN) architecture as in [21] is used. A scene graph is a discriminative scene model that combines the advantages of pose-invariant local, condition-invariant global, and their hybrid part descriptors (i.e., the graph nodes), with the additional advantage of discriminatively describing the relationships (i.e., the graph edges) between these scene parts. Speciﬁcally, our model comprises [22] (1) a scene graph generator that generates a scene graph from a scene and (2) a scene graph embedding that maps the scene graph to a class-speciﬁc probability map. The steps to scene graph generation are as follows: First, semantic labels are assigned to pixels using DeepLab v3+ [25], pretrained on Cityscapes dataset. Then, regions smaller than 1,000 pixels (for 616 ×808 image) are regarded as noise and removed. Subsequently, connected regions with the same semantic labels are identiﬁed using a ﬂood-ﬁll algorithm [26], and each is assigned a unique region ID. Next, each region is connected to each of its neighboring regions by an edge. Finally, an image scene graph with image region nodes is obtained. Speciﬁcally, each node is described by a feature vector as follows. The semantic labels output by a semantic segmentation network [25] were re-categorized into seven different semantic category IDs: “sky,” “tree,” “building,” “pole,” “road,” “trafﬁc sign,” and “the others” which respectively correspond to the labels {“sky”},{“vegetation” },{“building” },{“pole” },{“road,” “sidewalk” },{“trafﬁc-light,” “trafﬁc-sign” }, and {“person,” “rider,” “car,” “truck,” “bus,” “train,” “motorcycle,” “bicycle,” “wall,” “fence,” “terrain” }in the original label space. The location of the region center was quantized by a 3 ×3 regular grid into nine “bearing” category IDs. The region size was quantized into three “size” category IDs: “large (larger than 150 K pixels),” “medium (50 K-150 K pixels),” and “small (smaller than 50 K pixels)”. Finally, these semantic, bearing and size category IDs are combined to obtain a (7 ×9×3=) 189-dim 1-hot vector as the node descriptor. Ablation studies have proven that the proposed spatial features contribute to Fig. 3. Graph neural embedding, which is used by teachers/students as a visual embedding model in the evaluation experiment of this work. To address the information lost in dimension reduction, in [22], the appear- ance/spatial image information is proposed to be mapped to two different features, absolute (node) and relative (edge) features, which complement each other as shown in “proposed”. signiﬁcant performance improvements. The scene graph embedding is trained as follows. The graph convolution operation takes node viin the graph and processes it in the following manner. First, it receives messages from nodes connected by the edge. The collected messages are then summed via the SUM function. The result is passed through a single-layer fully connected neural net- work followed by a nonlinear transformation for conversion into a new feature vector. In this study, we used the rectiﬁed linear unit (ReLU) operation as the nonlinear transformation. The process was applied to all the nodes in the graph in each iteration, yielding a new graph that had the same shape as the original graph but updated node features. The iterative process was repeated Ltimes, where Lrepresents the ID of the last GCN layer. After the graph node information obtained in this manner was averaged, the probability value vector of the prediction for the graph was obtained by applying the fully connected layer and the softmax function. For implementation, we used the deep graph library [27] on the Pytorch backend. III. D ATA-FREEKNOWLEDGE TRANSFER SCHEMES (DFKT) This section discusses schemes for effective interaction between the questioner (i.e., the student’s substitute) with teachers, such that the resulting question-and-answer se- quence makes effective training samples for DFKT. As explained in Section II, the goal is to achieve a good trade-off between KT cost and self-localization performance. A. Replay Scheme First, we introduce the replay scheme [5], a well-known baseline CL scheme that is useful for benchmarking but cannot deal with blackbox teachers because it is not data- free. The basic idea is to maintain a portion of the training sample set used to train the model as part of the model rather than discarding it after training [9]. Such training samples serve as knowledge transferred from the teacher to the student and can naturally be used as training samples for supervised learning or KD. This approach is non-data- free and therefore has the best ability to avoid catastrophic forgetting among all the approaches used in our experiments. On the other hand, an obvious limitation of this approach is the assumption that it assumes accessibility to the teacher’s training samples. B. Reciprocal Rank (RR) Scheme The simplest possible data-free scheme is to use a random sample as a question x. In contrast to the expensive replay samples introduced in III-A, such random samples do not rely on the a priori knowledge of the teacher model and are therefore data-free. However, we experimentally found that this simple random scheme performed poorly. Regardless of whether it is used alone or in combination with the replay scheme, the performance of the simple random scheme is so poor that the entire framework fails. Therefore, we introduce a reciprocal rank (RR) scheme as a simple extension that satisﬁes the data-free requirements. This approach is motivated by the fact that typical visual embeddings serve as ranking functions. In our previous study [21], it was proven that the model output can be well approximated as a class-speciﬁc RR vector called a reciprocal rank feature (RRF), for a model equivalent to the visual embedding of Section II-A. Moreover, we will show that sampling xfrom the RRF space instead of from the entire input space yields samples with acceptable performance. This RRF vector is already low-dimensional but is well approximated by an even lower-dimensional k-hot RRF (k=10). Note that this k-hot RRF can be computed efﬁciently by performing a selection algorithm on an N-dimensional noise vector. This compressed version of the RR scheme was experimentally proven to perform as well as the simple RR scheme described above. For more details of RRF, please see [21]. C. Entropy Scheme Next, we introduce an Entropy-based scheme, where En- tropy is calculated from the class-speciﬁc probability map P(c|x)predicted for each class cby the model for a certain input signal x: E(x) =−∑ cP(c|x)log[ P(c|x)] , (1) and its inverse is used as a score of the likelihood that x belongs to a seen class. In a recent study [28], the Entropy measure is employed in self-localization for an alternative task of discovering unseen place-classes. In contrast, we propose using Entropy as a measure to determine whether the teacher model is familiar with an input signal. Speciﬁcally, the Entropy scheme differs from the RR scheme in the following ways: Instead of generating TRR samples, a much larger number T′(≫T)of RR samples {(xt,yt)}T′ t=1 are generated and TKT samples with highest scores are selected among the T′samples. The class imbalance problem is a serious issue in the implementation of this scheme. Naively sampled RR inputsoften belong to the popular classes. Consequently, the sam- pling efﬁciency of unpopular classes is poor and may lead to severe class imbalance. To avoid this issue, we introduce undersampling of the samples of popular classes. Notably, the Entropy value calculation used by this scheme implicitly relies on an assumption that we have access to class-speciﬁc probability map P(1|xt)···P(C|xt)of teacher output. Note that while this assumption holds true for neural network-type self-localization models, it does not necessarily hold for all self-localization models. For example, in a bag- of-words image retrieval engine-type teacher self-localization model (e.g., [29]), a class probability map may not be available, but only class-speciﬁc rank values or relevance scores, meaning that the RR scheme works but the Entropy scheme does not. In our future work, we would like to expand the Entropy scheme so that it can be applied to diverse types of teacher models. D. Mixup Scheme The disadvantage of replay schemes, that is, their pro- hibitively high sample maintenance costs, can be com- pensated for by combining them with other schemes that maintain samples more cheaply. Therefore, we introduce the mixup scheme as the fourth scheme, which combines the replay scheme with other schemes (RR or Entropy). Speciﬁcally, only a small number R(≪T) of samples per class (e.g., R=1) from the replay scheme is assumed to be available, and by mixing up those few replay samples with (T−R)KT samples provided by other schemes, the required number Tof samples is generated. IV. E XPERIMENTS We evaluated the proposed DFKT scheme (Section III) experimentally using the generic scenario called OWDL (Section II). A. Settings In this experiment, a recursive distillation scenario was assumed in which students and teachers were trained in a supervised manner and the student encountered up to three i=1,2,3teachers sequentially. At the initialization stage i=0, students and teachers are trained using supervised learning in a subset of the place-classes of size ≤100. This subset is referred to as the “classes-in-charge” for students or teachers. By default, the number of the classes-in-charge of each student/teacher was ten, and the classes were randomly selected from among the 100 place-classes. Therefore, the classes-in-charge of the student and teachers may partially overlap. Note that students and teachers nonetheless experi- enced independent seasonal domains. The sequential 27 sessions recorded over two years of NCLT were divided into one test session: “2012/08/04,” one visual embedding training session: “2012/04/29” and 25 sessions for training student/teacher self-localization model: “03/25,” “03/31,” “04/05,” “05/11,” “05/26,” “06/15,” “08/20,” “09/28,” “10/28,” “11/04,” “11/16,” “11/17,” “12/01,” “01/08,” “01/10,” “01/15,” “01/22,” “02/02,” 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 1 10 100Entropy RR Replay Mixup: Replay(1)+Entropy(a) Performance results 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 1 10 100ReplayReplay i=0 i=1 i=2 i=3 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 1 10 100RRRR i=0 i=1 i=2 i=3 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 1 10 100EntropyEntropy i=0 i=1 i=2 i=3 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 1 10 100Mixup: Replay(1)+EntropyMixup: Replay(1)+Entropy i=0 i=1 i=2 i=3 (b) Effects of catastrophic forgetting Fig. 4. Top-1 accuracy performance vs. KT cost ( T). “02/04,” “02/05,” “02/12,” “02/18,” “02/19,” “02/23,” and “03/17” (“MONTH/DAY” is used as a session’s name for brevity), for which session IDs s=0,···, 24 are respectively assigned. To investigate generalization performance on different training datasets, we considered six different sessions s=0, ···, 5 as candidates of the student training dataset. Accord- ingly, each i-th teacher’s training session ( i∈[1,2,3]) is determined as {(6i+s)mod 25}. Note that the student is trained by supervised learning only in the initialization stage i=0, and in subsequent stages i=1,2,3, the student is trained via DFKT from i-th teacher it encountered. Among the class-speciﬁc samples generated from the independent test dataset (“2012/08/04”), those belonging to the union of classes-in-charge of students and teachers are used as the test set. For example, in the case where the number of classes-in-charge per teacher/student model is 10 and classes-in-charge do not overlap between the four different models, a student at the initialization stage ( i=0) who has not encountered any teacher yet has experienced only 10 classes out of 40, and can only expect top-1 accuracy performance of about 25% at most. For the mixup scheme, the number Rof mixed replay samples is set R=1by default. At each KT stage ( i=1,2,3), the student robot encounters a new i-th teacher, then a two-step KT is performed. First, for each class in the new teacher’s classes-in-charge, TKT samples are generated using a scheme described in Section III. Second, for each class that is not the new teacher’s classes-in-charge and is the classes-in-charge of the previous version student denoted as (i−1)-th student, T′RR samples are generated and added to the KT sample set. Note that for the second student-to-student KT step, we can assume no KT cost (communication cost) will be incurred, and thus the number T′of such cost-free samples is set in large quantities (T′=100 samples per class). To demonstrate that the self-localization performance doesnot depend on the particular GCN architecture in Section II- A, we performed knowledge distillation [30] from the GCN to a multilayer perceptron (MLP) with 4,096-dimensional hidden layer, and the MLP was used as an alternative to GCN throughout the experiment. In preliminary experiments, it was proven that this MLP achieves performance comparable to GCN. The main difference between GCN and MLP is the architecture-speciﬁc internal signals, but this is not a serious issue since the DFKT schemes considered in this work do not rely on internal signals. A merit of using MLP in place of GCN is that signal analysis is easy and suitable for experimental evaluations. Since the training dataset gen- erated by the scheme described above will suffer from class imbalance, a balanced pseudo-dataset is temporarily created by oversampling less popular classes and is used to train the MLP. For KD from teacher MLP to student MLP, we again used the method of Hinton et al. [30] because it is the best-known KD scheme. In this standard distillation scheme, the loss function consists of distillation loss and student loss, with soft labels for the former and pseudo labels for the latter as training signals. Note that the soft labels computation assumes the availability of the teacher’s output layer probability map. To be independent of this assumption, the KD algorithm could be replaced with a more general recent KD algorithm, which is out of the scope of this study, and even if such replacement is performed, it is expected that the performance superiority and inferiority between the schemes will be maintained. B. Results First, we investigated the basic performance. Recall that the student encountered up to three i=1,2,3teachers sequentially. The student at the initialization stage i=0and each i-th teacher are trained with a rich annotated training set of the classes-in-charge (i.e., 100 samples per class). At each step i=1,2,3, the student encounters a new teacher and conducts a teacher-to-student KT using the questioner as a mediator. Details of KT and distillation follows the procedures in Sections II and IV-A. Four schemes were used to select query x: RR, replay, Entropy, and mixup. Figure 4 (a) shows the performance results. The results show that all four schemes can achieve high performance when the number of samples per class in- creases. (1) Note that the replay scheme is not affected by catastrophic forgetting under a sufﬁciently large number of samples, so it can achieve performance equivalent to supervised learning. However, this scheme is not data-free and is therefore used in this work only as a reference for benchmarking. (2) Although the RR scheme is a data-free scheme and a quite simple scheme, it was able to achieve surprisingly good performance. This may be partly due to the use of a good quality visual embedding by the graph convolution scene graph classiﬁer (Section II-A), leaving investigation into various embedding methods for future consideration. Another potential factor is that RRF was a good approximation of visual embedding. As also shown in the literature, RRF is a compact feature vector, and even random RR features could be a good approximation of visual embeddings at high frequency. (3) The Entropy scheme is a data-free scheme and has always marked the same or better performance than the RR scheme. In particular, when the number of samples per class is small, the performance is signiﬁcantly higher than the RR scheme, and it is clear that this scheme has the ability to generate a small number of elite samples. As the number of samples per class increased, we were able to achieve performance approaching that of the (non-data-free) replay scheme. However, as discussed in III-C, a potential concern with this scheme is that it relies on the availability of class-speciﬁc probability maps, which is a stronger assumption than the availability of class-speciﬁc rank values (i.e., assumed by the RR scheme). (4) The mixup scheme can achieve versatility and good cost performance. It relies on strong assumptions about keeping training samples as part of the model, but is more practical in that it keeps the number of training samples constant. As expected, this scheme achieved performance approaching that of the replay scheme. However, since this scheme is not strictly data-free, its range of applications is narrower than that of the replay scheme or the RR scheme. In conclusion, the performance was highest in the order of replay scheme, mixup scheme, Entropy scheme, and RR scheme, but the versatility was in the opposite order. We compared the ability to deal with catastrophic for- getting between different schemes. Catastrophic forgetting [31], where learning new knowledge causes one to forget what was previously learned, is a serious challenge in CL. In our case, we expect that a student model at the stage i=3who has learned from all teachers will be most affected by catastrophic forgetting. A simple way to measure the magnitude of this effect is to compare the class-speciﬁc test performance between classes learned from the new teacher i=3and classes learned from past teachers/student i= 0,1,2. Figure 4 (b) shows the results.It can be seen that the performance of the replay scheme is hardly affected by the value of i, meaning that the performance is hardly affected by catastrophic forgetting. As expected, the remaining schemes performed worse than the replay scheme, and among them, the mixup scheme, Entropy scheme, and RR scheme had the best performance in order. It is also noteworthy that the mixup scheme had relatively little performance drop compared with the replay scheme. This is because the small number Rof superior replay samples mitigated the effects of catastrophic forgetting.