Modality-Agnostic Topology Aware Localization

1 Introduction . Self-localization or localizing objects are primary tasks in navigation and surveillance systems . This problem has been the subject of many studies in the machine learning community . It has been addressed in several areas such as visual odometry [ Engel et al. , 2017 , Brahmbhatt et al. , 2018 ] , visual simultaneous mapping and localization ( VSLAM ) [ Davison et al. , 2007 , Mur-Artal et al. , 2015 ] , self-localization [ Arth et al. , 2011 , Sattler et al. , 2019 , Sarlin et al. , 2021 ] , etc. , and many approaches have been proposed . Recent localization methods achieve decimeter precision in the positioning of a moving camera in an indoor environment by leveraging the advances in neural networks [ Brahmbhatt et al. , 2018 , Kendall et al. , 2015 ] . However , such techniques are highly entangled with the modality of data in use ; thus , applying such specialized algorithms to other modalities is often not possible . For example , the existing visual odometry and VSLAM techniques that rely on visual features or camera projection models are incompatible with different sensory systems like radio frequency ( RF ) or audio signals . Still , all instantiate the same problem . In contrast to existing solutions tailored for a particular modality , we formulate the localization problem in terms of low-dimensional manifold learning to represent the input samples in their intrinsic space and transport them to a given topological map by inferring correspondences between intrinsic space and map . This approach allows a broader and more generalizable solution that can be used with data from different modalities . ∗Qualcomm AI Research is an initiative of Qualcomm Technologies , Inc. 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Here , by localization , we aim to pinpoint the location of a single observer at each time instance on a topological map by analyzing the collected sensory data . Unlike VSLAM , our method does not build a map ; we assume that it is given as a prior . Nevertheless , the topological map can have different forms . For example , it can be an approximate sketch of the 2D floor plan or a precise 2D CAD draft . While the observer moves in the environment and visits different locations , the measured signal such as video frames , depth images , or WiFi channel state information ( CSI ) encode the 2D location of the observer as well as the geometry of the space and other modality-specific factors , depending on the type of the sensory system in its high-dimensional ambient space . However , the intrinsic manifold that encodes the positional information ( coordinates ) lies only either in a 2D or 3D space ( 3D in case the altitude of the observer varies ) . Hence , a nonlinear dimensionality-reduction method that maps the input samples into their intrinsic 2D/3D embedding can yield a solution to the localization problem . Incorporating domain knowledge ( e.g . camera projection models for visual sensing or wave propagation equations for RF sensing ) may reduce the task of finding the mapping to a parametric regression problem ; however , the obtained solution would again be modality-specific . Instead , we build on our intuition that the transformation from high-dimensional input to the intrinsic space can be modeled as a parametric mapping , learnable through a neural network between data manifold and the 2D coordinates . In addition , we observe that the temporal data only changes gradually as the observer moves ; for instance , consecutive images in a video stream are similar . This observation implies the data reside in a smooth , locally connected manifold . We further postulate that an embedded latent manifold of much lower dimensionality represents the locations of the observer in the environment . Based on this , we revisit the localization problem and reformulate it as a learning task of the embedding space of data . Since we are given the room/zone labels and a topological map yet not the correspondences between the embedding space and the map , we adopt optimal transportation ( OT ) to estimate them . The proposed method does not rely on modality-specific priors , instead it builds on correlation between the observer location and the measured signal . Since we make no assumption on the transformation , the proposed method is applicable to a large spectrum of different sensor modalities . Fig . 1 illustrates the proposed approach . 2 Related Work . Localization has been the subject of many studies in robotics , computer vision , and RF sensing . As such , classical VSLAM methods [ Davison et al. , 2007 , Mur-Artal et al. , 2015 ] mainly build a 3D point cloud map of the scene while estimating the motion of the camera in the video data . Modern VSLAM approaches are usually composed of several carefully engineered modules , such as tracking , mapping , re-localization , loop closure , graph optimization , key-frame selection , or post-processing by bundle adjustment [ Henriques and Vedaldi , 2018 ] . Many of these modules make strong assumptions on the visual features or a camera projection model . Even recent efforts to replace some of these modules with learnable architectures borrow those assumptions in their methodology [ Chen et al. , 2020 ] or the method requires a strong supervision like known position and orientation [ Kendall et al. , 2015 ] , full 3D models [ Brachmann and Rother , 2018 , Sarlin et al. , 2021 ] , or image pairs [ Toker et al. , 2021 ] . Employing visual features and camera projection models can also be seen in visual odometry [ Engel et al. , 2017 ] . While these advances report decimeter-level accuracy in visual localization , they are not applicable to other sensory systems such as RF or audio signals . Their success demands major engineering efforts in transferring similar ideas to a new data modality . Manifold learning is being used extensively in fields like computer vision , natural language processing , and data mining . In manifold learning , the data points are assumed to lie on a smooth manifold M⊂ Rn in n-dimensional measured ambient space , while they are sampled from a distribution , near or on lower-dimensional sub-manifold N ⊂ Rs , where n > s. The minimum number of variables needed to describe such a distribution is known as the intrinsic dimension . The task of manifold learning is to find a smooth map Φ : M→N . If the data has intrinsic dimensionality of m , according to the Whitney Embedding Theorem [ Lee , 2013 ] , we know thatM can be embedded smoothly into s = 2m , using a homeomorphism Φ . However , it is often impossible to obtain an isometric embedding directly in the intrinsic space [ McQueen et al. , 2016 ] . As far as the smooth embedding preserves the topology ofM , this might be sufficient for many dimension reduction purposes but when preserving the geometry of embedding is desirable , finding an isometric embedding is required . Several previous works in non-linear dimension reduction [ Tenenbaum et al. , 2000 , Verma , 2012 , Weinberger and Saul , 2006 , McQueen et al. , 2016 ] have been driven by the desire to find a smooth embedding of low dimension that is isometric . Since almost all embedding algorithms suffer from distortion , there are several works that propose a different technique for mitigating this issue by filtering out the border samples [ Rosman et al. , 2010 ] , introducing a Riemannian relaxation [ McQueen et al. , 2016 ] , or setting some constraints on the embedding [ Agrawal et al. , 2021 ] . Depending on the amount of imposed distortion in the intrinsic embedding , the performance of a downstream task that is applied on the representation of the sample in the embedding space would degrade . In localization problems , when there is no ground truth positions available , mapping from a distorted intrinsic space into the target map can be difficult and prone to a high error . Optimal transportation has received considerable interest from the topology community [ Peyré et al. , 2019 ] . When the data is associated with geometrical properties , optimal transport metrics ( also called Wasserstein distance or Earth Mover distance ) measure the spatial variations between probability distributions of source and target domains . Correspondence matching is one of the successful applications of optimal transport [ Mémoli , 2011 , Peyré et al. , 2016 , Solomon et al. , 2016 ] . Given a transport cost function , the Wasserstein distance computes the optimal transportation plan between two measures . Recent progress on efficient computation of the optimal transport by introducing entropy regularization and Sinkhorn ’ s matrix scaling algorithms reduced the computational cost of optimal transport several orders of magnitude than of the original transport solver [ Cuturi , 2013 ] . As shown in [ Genevay et al. , 2018 ] , computing the OT loss and its gradient can be tractable by using Sinkhorn fixed point iterations . Finding the transformation for representing the data points on a given topological 2D map requires knowing a set of correspondences in the localization problem . In case the correspondences are available , learning a transformation is straightforward . However , in an unsupervised setting , estimating such a transformation can be very difficult . In what we propose , the correspondences ( i.e . coupling matrix ) between the 2D embedding and the target topological map is learned by using the OT algorithm . We find a coupling matrix ( also called transport plan ) representing the correspondences between two domains by minimizing the cost of transportation . In the next section , we explain how the transportation cost is parameterized by the representation of a neural network in the intrinsic embedding of data . 3 Embedding learning and transportation to the intrinsic space . In this work , we formulate the localization problem in the context of manifold learning and optimal transportation . Our contributions are as follows : 1 . We propose a model that jointly learns the intrinsic embedding and its transportation into a topological map of the environment in a weakly-supervised style by using gradient descent optimization . Such a joint optimization mitigates the distortion of the intrinsic embedding as the model constrains it to resemble the topology of the target map . 2 . The proposed localization method does not make any assumptions about the data modality in use . In that sense , it is modality-agnostic and can be applied to a large family of sensory systems . From the system setup point of view , our method can extend to both active and passive positioning problems . 3.1 Problem and theoretical motivations . Let Ωs ∈ Rn be the ambient space of some measured signal and Ωh ∈ Rm with m ⊂ { 2 , 3 } be its intrinsic space . We are interested in finding the representations in Ωh of the discrete samples Xs = { xsi } Ns i=1 ∈ Ωs . We assume the measured samples Xs lie on a smooth ( Riemannian ) manifold in Ωs ; thus , the manifold is locally connected . This assumption intuitively holds since the data is a temporal sequence that does not change much between two consecutive instances in time . We also assume that the topological map Ωt ∈ Rm that represents the geometry of the environment is known , yet no correspondences between these two spaces are available . This map can be in the form of a 2D sketch or a floor plan of a building . Localizing the observer in Ωt requires finding a mapping between Ωs and Ωt . Using manifold learning techniques , one can estimate the embedding in Rm and consequently seek a transformation to map the embedding into Ωt . When we do not have access to correspondences between the embedding and target domains , finding an isometric embedding that preserves the pairwise distances between samples is desirable as the transformation to the target would be less complex . Hence , methods like Isomap [ Tenenbaum et al. , 2000 ] which preserve the local and global distances , have an advantage . However , due to the non-convexity of Ωs , it is often impossible to obtain an isometric embedding directly in the intrinsic space [ McQueen et al. , 2016 ] . Since walls usually partition interior spaces , there may not be a direct path between any two arbitrary points on the floor plan . Therefore , due to the non-convexity of the sample set , usually , the embedding suffers from a severe distortion that a simple isometric transformation is not sufficient for alignment with the target domain . As an alternative , we propose finding a map between the input and the intrinsic spaces that both preserves the pairwise distances between samples and aligns Ωh and Ωt . We formulate the localization problem as follows . Consider Φ : Ωs → Ωh be a smooth map between input and intrinsic embedding . The map Φ can be implemented by a neural network e.g . an MLP . We define ( Ds , Dh ) ∈ RNs×Ns as distance matrices between samples in Ωs and Ωh , respectively . So the entries of Dh can be computed like d2ij = ‖Φ ( xi ) − Φ ( xj ) ‖ 2 . For now , let ’ s assume we have access to the geodesic distance-matrix Ds , which contains all pairwise geodesic distances of Xs on the input manifold . Later , in section 3.3 , we explain how Ds can be approximated . It is worth mentioning that training the Φ by minimizing the ‖Ds −Dh‖2 and using gradient descent optimization leads to a parametric approximation of the MDS algorithm [ Pai et al. , 2019 ] . However , this formulation is ill-posed when Xs is a non-convex set since comparing the geodesic distances with Euclidean distances by using ‖·‖2 is only valid inside a convex region . For solving the correspondence problem , the Gromov-Wasserstein discrepancy between two distance matrices was used [ Peyré et al. , 2016 ] . There , the coupling between two sets of distances is found by a regularized optimal transport . In this work , our proposed model learns both the map Φ and simultaneously by estimating a coupling matrix ( T ∈ RNs×Nt ) . Training the model therefore amounts to minimizing the loss min Φ , T L ( Ds , Dh ) + µ ∑ ij Tij · Cij with Cij = ‖Φ ( xi ) − yj‖2 , ( 1 ) where the first term L ( . , . ) is a dissimilarity measure between two distance matrices and the second term is the Sinkhorn distance between the samples in the embedding and the target topological map . The scalar µ is a contribution weight . The C ∈ RNs×Nt and T ∈ RNs×Nt are the cost of transportation and the coupling matrix . Here , Y = { yj } Ntj=1 ∈ Ωt is defined by dense sampling from the target domain . The embedding representation can have any arbitrary rotation and scale with respect to their true positions on the floor plan . Here , by using zone/room labels , we constrain the transportation to seek for a valid assignment . To do so , we penalize the transportation when the zone labels of input xi and the target yi are different . This can be done simply by multiplying the cost Cij by a constant factor αij > 1 when the labels are different and αij = 1 if they have a same label . Since the distance matrices , Ds and Dh , are in two different domains , instead of using L = ‖·‖2 , We propose to use the Kullback-Leibler divergence ( KL ) for L [ Cuturi , 2013 ] . Fig . 2 shows the block diagram of the proposed model .

Summary: This paper presents a new method for localization while being agnostic to input measurement signal's modality. Assuming a map is given, localization inside the map is formulated as a deep manifold learning problem, without needing the supervision of ground truth positions for each input measurement. The method utilizes differentiable optimal transport to solve the correspondence challenge in this formulation. The method is tested on several simulated image datasets and one real-world WiFi dataset.