Modality-Agnostic Topology Aware Localization
Authors: Farhad Ghazvinian Zanjani
Ilia Karmanov
Hanno Ackermann
Daniel Dijkman
Simone Merlin
Max Welling
Fatih Porikli
Venue: NeurIPS 2021 Poster (2021)
Keywords: Manifold learning
optimal transportation
indoor localization

Abstract: This work presents a data-driven approach for the indoor localization of an observer on a 2D topological map of the environment. State-of-the-art techniques may yield accurate estimates only when they are tailor-made for a specific data modality like camera-based system that prevents their applicability to broader domains. Here, we establish a modality-agnostic framework (called OT-Isomap) and formulate the localization problem in the context of parametric manifold learning while leveraging optimal transportation. This framework allows jointly learning a low-dimensional embedding as well as correspondences with a topological map. We examine the generalizability of the proposed algorithm by applying it to data from diverse modalities such as image sequences and radio frequency signals. The experimental results demonstrate decimeter-level accuracy for localization using different sensory inputs.

Modality-Agnostie Topology Aware Localization
Farhad G. Zanjani
Ilia Karmanov
Hanno Ackermaan
Daniel Dijkman
Simone Merlin
Max Welling
Fatih Porikli
Qualcomm Al Research"
(fzanjani, iikarmano, haaaaannnaaakeen hackerma, jkmjn smerlin, mww.lligggfforriklleag.. wtw.:/giiiaaalcom..ooomo
Abstract
This work presents data-driven approach for the indoor localization of an observer on 2D topological map the environment. State-of-the-an techniques may yield accurate estimates only when they are tailor-made for specific data modality like camera-based system that prevents their applicability to broader domains. Here, establish a modality-agnostic framework (called OT-Isomap) and formulate the localization problem in the context of parametric manifold learning while leveraging optimal transportation. This framework allows jointly learning low- dimensional embedding as well as correspondences with a topological map. We examine the generalizability of the proposed algorithm by applying data from diverse modalities such as image sequences and radio frequency signals. The experimental results demonstrate decimeter-level accuracy for localization using different sensory inputs.
Introduction
Self-localization localizing objects are primary tasks in navigation and surveillance systems. This problem has been the subject of many studies in the machine learning community. It has been addressed in several areas such as visual odometry (Engel 2017 Brahmbhatt al. [2018), visual simultaneous mapping and localization (VSLAM) Davison et al. 2007 Mur-Artal Mur-Arrrr et al. [013). self-locallzation (Arth al. 2011 Sattler et have been proposed.
2019
Sarlin et al. 2021 etc., and many approaches
Recent localization methods achieve decimeter precision in the positioning of moving camera in an indoor environment by leveraging the advances in neural networks [Brahmbbatt et al. 2018 Kendall et 2015). However, such techniques are highly entangled with the modality of data in use; thus, applying such specialized algorithms to other modalities is often not possible. For example, the existing visual odometry and VSLAM techniques that rely on visual features or camera projection models are incompatible with different sensory systems like radio frequency (RF) or audio signals. Still, all instantiate the same problem. contrast existing solutions tailored for a particular modality, we formulate the localization problem in terms of low-dimensional manifold learning to represent the input samples their intrinsic space and transport them given topological map by inferring correspondences between intrinsic space and map. This approach allows broader and more generalizable solution that can used with data from different modalities.
QQalcomm AI Research is initiative of Qualcomm Technologies. Inc.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Ω
Qt
optimal transport
dep
(1)
(2)
(3)
(4)
Figure Schematic of the proposed modality-aggnsstc localization method; (1) different input modalities, for instance, radio frequency (RF) signals or video sequences; (2) the input signals are assumed to change gradually, thus are on smooth manifold in input space Ω2 (3) the positions are encoded in an intrinsic space S2, Please refer to Sec. [.3] for details about the mapping; (4) correspondences between O4, and the positions on the floormap is estimated by means of optimal transportation. Please refer to Sec. 3.2 for details.
Here, by localization, we aim to pinpoint the location of a single observer at each time instance on a topological map by analyzing the collected sensory data. Unlike VSLAM, our method does not build a map: we assume that it is given as prior. Nevertheless, the topological map can have different forms. For example, it can be approximate sketch of the 2D floor plan precise 2D CAD draft. While the observer moves the environment and visits different locations, the measured signal such video frames, depth images, WiFi channel state information (CSI) encode the 2D location of the observer as well as the geometry of the space and other modality-specific factors, depending on the type of the sensory system in its high-dimensional ambient space. However, the intrinsic manifold that encodes the positional information (coordinates) lies only either in a 2D or 3D space (3D in case the altitude of the observer varies). Hence, nonlinear dimensiionlliy--eduutioo maps the input samples into their intrinsic 2D/3D embedding can yield a solution to the localization problem. Incorporating domain knowledge (e.g. camera projection models for visual sensing or wave propagation equations for RF sensing) may reduce the task of finding the mapping to a parametric regression problem; however, the obtained solution would again be modality-speciffc. Instead, build on our intuition that the transformation from high-dimensional input to the intrinsic space can be modeled as a parametric mapping, learnable through neural network between data manifold and the 2D coordinates.
method that
In addition, observe that the temporal data only changes gradually as the observer moves; for instance, consecutive images in a video stream are similar. This observation implies the data reside in smooth, locally connected manifold. We further postulate that embedded latent manifold of much lower dimensionality represents the locations of the observer in the environment. Based on this, we revisit the localization problem and reformulate it as a learning task of the embedding space of data. Since we are given the room/zone labels and a topological map yet not the correspondences between the embedding space and the map, we adopt optimal transportation (OT) to estimate them.
The proposed method does not rely on modality-specific priors, instead it builds on correlation between the observer location and the measured signal. Since we make no assumption the transformation, the proposed method is applicable to large spectrum of different sensor modalities. Fig. [ illustrttess the proposed approach.
2 Related Work
Localization has been the subject of many studies in robotics, computer vision, and RF sensing. As such, classical VSLAM methods [Davison et al., Mur-Araa et al. 2015) mainly build a 3D point cloud map of the scene while estimating the motion approaches are usually composed of several carefully engineered modules, such as tracking, mapping, re-localization, loop closure, graph optimization, key-frame selection, or poot-proc by bundle adjustment [Henriques and Vedaldi, 2018). Many of these modules make strong assumptions on the
camera in the video data. Modern VSLAM

---

visual features or camera projection model. Even recent efforts to replace some of these modules with learnable architectures borrow those assumptions in their methodology [Chen et al. 2020 or the method requires a strong supervision like known position and orientation [Kendall et al.) 2015] full
3D models [Brachmann and Rother 2018 [aarlin et al. 2021], or image pairs [Toker et al., 2021]. Employing visual features and camera projection models can also be seen in visual odometry EEnel itt al., 2017). While these advances report decimeter-level accuracy in visual localization, they are not applicable to other sensory systems such as RF or audio signals. Their success demands major engineering efforts in transferring similar ideas to a new data modality.
Manifold learning is being used extensively in fields like computer vision, natural language process- ing, and data mining. In manifold learning, the data points are assumed to lie on a smooth manifold M c R" in n-dimensional measured ambient space, while they are sampled from distribution, near or on lower-dimensional aub-manifold A C IK" where n > 8. The minimum number of variables needed to describe such a distribution is known as the intrinsic dimension. The task of manifold learning is to find smooth map & M -> N. the data has intrinsic dimensionality of m, accord- ing to the Whitney Embedding Theorem [Lee] 2013). we know that M can be embedded smoothly into 5 = 2m, using a homeomorphism 4. However, it is often impossible to obtain an isometric embedding directly in the intrinsic space [McQueen et al. 2016). As far the smooth embedding preserves the topology of M. this might be sufficient for many dimension reduction purposes but when preserving the geometry of embedding is desirable, finding an isometric embedding is required. Several previous works in non-linest dimension reduction [Tenenbaum et al., 2000 Verma 2012] Weinberger and Saul 2006 McQueen et al. 2016 have been driven by the desire to find smooth embedding of low dimension that is isometric. Since almost all embedding algorithms suffer from several works that propose different technique for mitigating this issue by filtering out the border samples [Rosman et al.] 2010]. introducing a Riemannian relaxation [McQueen] ct al. 2016], or setting some constraints on the embedding (Agrawal et al.) [022]] Depending on the amount of imposed distortion in the intrinsic embedding, the performance of a downstream task that is applied on the representation of the sample in the embedding space would degrade. In localization problems, when there is no ground truth positions available, mapping from a distorted intrinsic space into the target map can be difficult and prone to a high error.
distortion, there
Optimal transportation has received considerable interest from the topology community (Peyré et al., 2019). When the data is associated with geometrical properties, optimal transport metrics (also called Wasserstein distance or Earth Mover distance) measure the spatial variations between proba- bility distributions of source and target domains. Correspondence matching is one of the successful applications of optimal transport [Mémoli] 2011 Peyré et al. [016)] Solomon et al. 2016). Given a transport cost function, the Wasserstein distance computes the optimal transportation plan between two measures. Recent progress on efficient computation of the optimal transport by introducing entropy regularization and Sinkhorn's 's matrix scaling algorithms reduced the computational cost of optimal transport several orders of magnitude than of the original transport solver [Cuturi. 2013). As shown in [Genevay et al., [2018). computing the OT loss and its gradient can be tractable by using Sinkhorn fixed point iterations.
Finding the transformation for representing the data points on a given topological 2D map requires knowing a set of correspondences in the localization problem. In case the correspondences are avail- able, learning a transformation is straightforward. However, in an unsupervised setting, estimating such a transformation can be very difficult.
In what we propose, the correspondences (i.e. coupling matrix) between the 2D embedding and the target topological map is learned by using the OT algorithm. We find a coupling matrix (also called transport plan) representing the correspondences between two domains by minimizing the cost of transportation. In the next section, we explain how the transportation cost is parameterized by the representation of neural network in the intrinsic embedding of data.
3 Embedding learning and transportation to the intrinsic space
In this work, we formulate the localization problem in the context of manifold learning and optimal transportation. Our contributions are as follows:
1. We propose a model that jointly learns the intrinsic embedding and its transportation into a topologiaal map of the environment in weakly style by using gradient descent
3

---

optimization. Such joint optimization mitigates the distortion of the intrinsic embedding as the model constrains it to resemble the topology of the target map.
2. The proposed localization method does not make any assumptions about the data modality in use. In that sense, it is modality-agnostte and can be applied to large family of sensory systems. From the system setup point of view, our method can extend to both active and passive positioning problems.
(1)(1) Σ Gy
Cu -
Tysooe
Sinkhom ltr.
( (MLP))
Vi2
Painwise dist.
Spatio-lemporal analysis
Pgg
D2
Geodesic compulation
(q.(1) KL(D,, D
Vince
VC)) (Com:Net)
Sequential data
KNN-graph
Exok
KNN comp.
Pushforward
Figure 2: Block diagram of the method. A set of prototype vectors and their KNN are computed by spatio-temporal analysis of the sequential input data. KNN graph is constructed by using the prototype vectors its nodes (V) and the KNN indices (E) the edges (i.e. connectivities). By estimating the pushforward at the location of each node on the data manifold, the small distances between neighboring nodes in the 2D intrinsic space approximated. The distance between non- adjacent nodes is approximated by shortest path through the graph. The 2D intrinsic representation is learned by preserving the pairwise distances and jointly finding the correspondences with given target floor plan (52z).
3.1 Problem and theoretical motivations
Let Qg E RR" be the ambient space of some measured signal and Q2 E Rm with m C (2,3) be its intrinsic space. We are interested in finding the representations in 24 of the discrete samples X. = {a:]
Zg We assume the measured samples X, lie on smooth (Riemannian) manifold in 2z; thus, the manifold is locally connected. This assumption intuitively holds since the data is temporal sequence that does not change much between two consecutive instances in time. We also assume that the topological map 22 E mhmh that represents the geometry of the environment known, yet no correspondences between these two spaces are available. This map can be in the form of a 2D sketch or floor plan of building. Localizing the observer in S2 requires finding a mapping between Ω2 and Ωt.
Using manifold learning techniques, one can estimate the embedding in Ih and consequently seek a transformation to map the embedding into Ω4 When we do not have access to correspondences between the embedding and target domains, finding an isometric embedding that preserves the pairwise distances between samples is desirable as the transformation to the target would be less complex. Hence, methods like Isomap [Tenenbaum et al. 2000] which preserve the local and global distances, have an advantage. However, due to the non con-exnyy of Ω2 it is often impossible to obtain an isometric embedding directly in the intrinsic space [MeQueen et al., (016). Since walls usually partition interior spaces, there may not be a direct path between any two arbitrary points on the floor plan. Therefore, due to the non-convexity the sample set, usually, the embedding suffers from a severe distortion that simple isometric transformation is not sufficient for alignment with the target domain. As an alternative, we propose finding a map between the input and the intrinsic spaces that both preserves the pairwise distances between samples and aligns O, and Ω..

---

We formulate the localization problem as follows. Consider +b : Ω2 + S2p, be a smooth map between input and intrinsic embedding. The map & can be implemented by a neural network e.g. an MLP. We define (Dx, Da) E RNN,N, as distance matrices between samples in S2 and Ωr, respectively. So the For now, let's assume we have access to the geodesic listance-matie matrix D., which contains all pairwise geodesic distances of X, on the input manifold. Later, in section 3.3] explain how D, be approximated. It is worth mentioning that training the & by minimizing the , - D2 and using gradient descent optimization leads to parametric approximation of the MDS algorithm (Pai et al. 2019]. However, this formulation ill-posed when X, is a non-convex set since comparing the geodesic distances with Euclidean distances by using II-11 is only valid inside a convex region.
entries of Dr, can be computed like d' [ ((x,))) ?
For solving the correspondence problem, the Gromov- Wasserstein discrepancy between two distance matrices was used (Peyré al. 2016). There, the coupling between sets of distances is found by regularized optimal transport. In this work, our proposed model learns both the map & and simultancously by estimating coupling matrix (I' E RN..N.). Training the model therefore amounts to minimizing the loss
min T(D DR) 11 To TTy.Cy Cy with C. = ([)(),) リy $T
if
where the first term .)...) is dissimilarity measure between two distance matrices and the second term is the Sinkhorn distance between the samples in the embedding and the target topological map. The scalar yu is contribution weight. The C NexxNt and T E Nex are the cost of transportation and the coupling matrix. Here, Y {リ5 1 24 defined by dense sampling from the target domain, The embedding representation can have any arbitrary rotation and scale with respect to their true positions on the floor plan. Here, by using zone/room labels, we constrain the transportation to seek for a valid assignment. To do so, we penalize the transportation when the zone labels of input I; and the target Vi are different. This can be done simply by multiplying the cost Cjj by a constant factor ai > 1 when the labels are different and aij = 1 if they have a same label. Since the distance matrices, D, and Dr. are in two different domains, instead of using L = II-11?, We propose to use the Kullback-Leiler divergence (KL) for [Cuturi 2013). Fig. shows the block diagram of the proposed model.
3.2 Regularized Transport with Differentiable Sinkhorn Distance
One major advantage of regularizing the optimal transport problem is that it becomes solvable efficiently using Sinkhorn's 's algorithm [Genevay et al. 2018). In computing the entropy constraint of the Sinkhorn distance as proposed in [Cuturi 2013], we are looking for coupling matrix that satisfies
T(C.p.g) = argmin (T, C) H(T). H
(2)
TE>(p.4)
Here, p and are probability distributions of samples in source Ω, and target Ωr domains and 7(p, q) is their joint probability. The C E N is cost matrix for transporting mass between two
domains. In Eq. 2). the term H(T) = Σ T log(T)) is the entropy of coupling T. The solution of is Cuturi 2013::
T(C,p. .) d Kdiag(b),
where K = - E 14, Nox is the so-called Gibbs kernel associated to C' and (a, E IR, N. x
R." N, can be computed using the Sinkhorn- Knopp iterative algorithm:
P Kb
and
b
(4)
KTaa
where T denotes the transpose of a matrix and the division is element-wise When there is no prior knowledge about the location of the observer in the environment, a uniform distribution can be assigned to p and q. As proposed in [Genevay al. 2018 using the Sinkhorn iteration is an efficient way for differentiating Eq. 2 By running Eq. (4) for M iteration, and putting the obtained (a, b) in #f
Eq. (3), we can compute the derivative 2K and consequently ae. Hence, Eq. (1 is differentiable. As the cost matrix C' depends on & the gradient can be back-propagane to optimize the parameters of
5

---

the neural network @. The pseudo code in Algorithm describes the procedure of jointly learning the embedding and the transportation.
The value of the regularization coefficient (2) in Eq. controls the distribution of T. A high entropy the coupling matrix 7 means uniform distribution over its entries that makes it smooth assignment matrix. In contrast, a low entropy means the I'' makes a hard assignment. Regularizing the Eq. 2 with high entropy makes the optimization stable [Genevay et al., 2018], the cost of a higher error in finding the correct correspondences. When training the model, we apply a linear annealing schedule the weight of the entropy regularization from an initial high value to a final small value. This range of À is a
of the model and is fixed in all our experiments.
Algorithm 1: Jointly learning the embedding and the transportation plan.
Input :X, {z.) N, E 2z Y { Ω4 and D2 E RNNN,, Output: model parameters T and coupling matrix T N.NN Initialize: $ T, and À = {1 (mazz)
while 1 < Maxar do
Dij = [$(x;) - {1;))1
Ct ais (z,, y
// constants aij 2
T= Sinkhorn Sinkhorn(C, (C,)
// Eq. (3) and Eq. (4)
L DKL(D, ((++1) -- ((()) 76
(T,C);
// Eq. (1)
// is learning rate
end
3.3 Computing the geodesic distance-matrix
The objective function in Eq. ( requires cre-ctnn the distance matrix D, that represents the pair-wise distances in the training set Xg. Since the inputs X, are distributed on a non-convex manifold in input space, the pairwise geodesic distances between non-neighboring samples should be measured. For an efficient computation of the geodesic distances, three steps are taken: (1) Reducing the size of X, by finding set of representative samples (also called prototype vectors or landmarks) and computing their k-nearest neighbors; (2) estimating the push-forward metric for estimating the Euclidean distances between neighboring prototypes in the embedding; (3) estimating the pairwise geodesic distances between non-neighbor prototypes, using shortest path algorithm; In the following, we explain each step.
3.3.1 Finding set of prototypes and their KNN
Similar to landmark Isomap De Silva and Tenenbaum, 2003 instead of computing all pairwise distances in a large training of X,, only the pairwise distances in of N, representative samples- - called prototype vectors - are computed. Finding a set of prototypes by using farthest-point sampling (Pai et al., 2019] and consequently measuring the Euclidean distance between them in the nigh-diimnsional input space have some drawbacks. Even for the 2D manifolds in R&, such as surfaces with holes or self-intersection, finding the KNN, by using Euclidean distance is susceptible to short-circuiting on the manifold.
Instead, we propose learn metric space for computing both the prototype vectors and their neighbor indices by training a neural network on the triplet sampled data (Hadsell et al.) 2006). Each triplet set contains two samples that temporally close, and third one that is distant. We apply an upper bound to the maximal temporal distance. The network learns to produce similar feature vectors for samples, based on their temporal vicinity by minimizing its triplet margin loss:
L x = max d(h (000,,,) (?)) - A() + +a) where ha = (I:)).
(5)
In E5. the symbol Ve denotes the function of the neural network, d measures the Euclidean distances, and (hę, hp. hr) E R², < n, are the output vectors of the network, produced from the vth set of anchor (x:)), positive (x²) and negative (x.)) instances and the scalar a is constant margin.
After convergence, the data samples (X.) are partitioned into N, clusters by applying standard the centroid vectors of each
k-means clustering on the obtained features set h. As prototypes,

---

class. Furthermore, by measuring the pairwise Euclidean distances between the feature set, the K nearest neighbors of each prototype are found.
3.3.2 Approximating the distances in embedding
For estimating the small distances between neighbor samples in the embedding, measuring the Fuclidean distance in the input space not informative [Dsilva et al., 2015]. For more reliable estimation, the unimportant sources of variability need to be suppressed. As in (Dsilva et al. 2015). using the Mahalanobis distance is representative distance along the local principal direction of data manifold and can be used an approximation of small Euclidean distances in the embedding:
[x(x)) (x,))l 1/, - =; (C* (x,))) [x, - mi),
where C(x)) is the measured local covariance matrix of data at the location of sample H₁, and denotes the Moore-Penross Penrose pseudoinverse. For computing Eq. (6). we estimate the covariance matrix of samples each cluster. By using Eq. (6) the pairwise distances between neighboring prototype vectors in the embedding space be approximated. When the distances small enough, such an estimation is similar to the pushforward in differential geometry that an approximation to the differential of map (0T) between tangent spaces of two manifolds.
After computing all pairwise distances between each prototype and KNN, we create a KNN-graph similar to the procedure in Isomap (De Silva and Tenenbaum, 2003) and the distances between non-neighboring samples are estimated by using the Dijkstra's shortest path algorithm [Dijkstra et 1959]. Then, the geodesic matrix D. in the embedding space is known and Eq. ( can be evaluated for training the model.
Experiments
We applied the proposed model three different data modalities. For details about the implementa- tion, hyper-parameter and training please refer to the supplementary material.
4.1 Synthetic data 2D maze environment
To validate approach without the added complexities of data, first setup experiment with simple 2D Maze environment" [Henriques and Vedaldi 2018 Parisotto and Salakhutdinov 2017]. In this simulation, an agent walks through maze from starting point to the end point. The starting and end points are chosen randomly on the outer walls the maze (see Fig 33. The solution path between these points is considered choosing several different starting and end points. A square window, centered at the location agent is used for sampling image patch from the environment. collection of sampled patches is used as the training set. The model predicts the position of observer with 0.7 pixel error for the maze environment of size 24 24 pixels. More inference results for number of trajectories are shown in the supplementary (see Fig. 2).
trajectory. We create many trajectories by randomly
PEESS
Figure 3: Synthetic maze environment size 24 24 pixels; (left) An example of path and sampling window; (middle) extracted patches for the shown path; (right) an example of the ground truth (in green) and the predicted (in red) trajectories crarrrttrrrss
https//ggihubocccm////lllils/nnn (GNU General Public License v3.0)
7

---

Figure 4:
An example of the iGibson environment; (left) the guadruple-vieee at the location of observer and the constructed panorama image; (middle) ground truth trajectory; (right) prediction
Table 1: First row: indexes of the 15 scenes of iGibson (Rs, Merom-O, Merom-1, Pomaria-O, Pomaria-1, Pomaria-2, Ihlen-O, Ihlen-1, Benevolence-O, Benevolence-l, Benevolence-2, Beechwood- 0, Beechwood-l, Wainscott-O, Wainscott- Second row: Localization errors (in meters).
Environments
index
5
6
7
8
10
12
etror
0.62
1.05
0.73
0.64
0.69
0.83
1,02
1.03
0.73
0.62
0.74
0.71
0.98
4.2
Camera-based localization
In case of using the image modality for localization by learning the data manifold, independent of camera orientation, the input data requires to contain 360 degree image sequences of several indoor spaces. To the best of our knowledge, such dataset is not publicly available. However, take advantage cxisting indoor 3D scan dataset and by navigating a camera the 3D scans create videos. With this aim, setup experiment by using the iGibson dataset (Shen et al. 2020]. The dataset consists of indoor 3D scans of 15 buildings that can be explored interactively [ et al.. 2020). The scans created from real homes and offices. The iGibson allows of trajectories each environment that simulates the navigation camera. We create several image sequences by navigating the camera through all rooms/zones of each environments. We leave out some of the image sequences for the test set. For sampling from the data manifold, each location of agent, we capture four orthogonal-viens images by rotating the camera with multiple of 90% with the up-direction being the rotation axis. These four images are used for constructing panorama image.
create set
robot agent, equipped with a
The panorama sequences and the bird-eye view of floor plan are the inputs to the model. Visiting the same location in the environment through several trajectories with different azimuth angles of the camera introduces a column- circular shift in the panorama images. To make the spatio-temporal analysis of data to be invariant with respect this rolling effect, we normalize the panorama images. We know that cyclic rotations of panorama images are equivalent to an addition of linear phase in the Fourier domain (Pajdla and Hlavac. 1999. Sturzl and Mallot, 2006). Thus, by computing the FFT of each panorama image and then estimating and removing the linear phase in the horizontal direction and transforming it back into image domain, we normalize each panorama image. Fig. shows an example of four images, taken at the location of observer, ground-trun -tuth trajectory, and its prediction. Table reporrs the localization errors for different environments in the dataset. More qualitative results have been shown in the supplementary.
4.3 Passive WiFi localization
Localization of moving target in the pervasive WiFi signal has received some attention of the machine learning community [Fan al., 2020). Positioning through WiFi can be useful in the domain of home, enterprise and industrial automation and robotics. It is not affected by poor light conditions and can thus work in the dark, and can work across walls. Also, a device without video-camera is more likely to be adopted by user since it largely mitigates issues of privacy related to video images. Our experimental setup is designed to mimic a real-world deployment and thus we only use three receivers and one transmitter for large multi-room environment where most areas lack

---

Figure 5: WiFi localization: (left) floor plan topologiaa map; (middle) ground ground-truth positions of observer, (right) inference. The points color encodes the x-y locations of person for the ground-true -trut and their correspondences with the prediction.
(+)
Figure 6: Example of perturbation in the environment; (a) shows reference environment that has been used for the training: (b-d) show panoramas which correspond 70%, 50% and 30% of the objects identical with those in (a).
line-of-sight. We consider passive positioning, i.e. the only source information are the reflections of the electromagnetic waves from the body of the moving target. We collected a data by using four commercial IEEE 802. 11 access points (AP), operating in the 5GHz band, and deployed them in the environment of size 14 x 20 meters in building. The environment contains three rooms, two long aisles and large lab (see Fig. 5) We use three receivers with eight antennas each, and transmitter with single antenna. Each receiver collects the Channel State Information (CSI) at periodic intervals. The CSI represents the channel between the transmitter antenna and each of its receiver antennas, across 208 frequency tones that span the transmission bandwidth. Hence, the CSI is represented as multidimensional tensor of complex numbers of dimension 8x1x208 per each WiFi packet. Here, we only use the magnitude of CSI signal. We recorded about five minutes of CSI data from all three receivers, while person freely walks through different locations in the environment. For ground-truth positions, we used several 3D cameras, installed in the environment. The algorithm achieved average error of 1.2 meters for locating the person on the 2D floor plan image of the environment. Fig. visualizes the predicted locations the person in the building.
4.4 Ablation Studies
We did an ablation study on the robustness of the camera-based localization by altering the visual features and object instances in the environment. This has been done by randomly exchanging both texture and appearance of objects in the scene. Fig. [olhows several examples of such variations with respect to the original environment that has been used for training. By measuring the accuracy the algorithm achieves on the altered scenes in a validation run, we are able to determine how susceptible the algorithm is when it is tested on the perturbed images. To estimate mean and standard deviation, we repeated this test ten times with ten differently randomized scenes. The results can be seen in Table 2

---

Table 2: Mean and standard deviation of localization error (cm) under different amount of changes in the environment.
Amount of perturbation on the test set
Rate
0%
10%
20%
30%
50%
70% 64.07 + 0.26
Error (mean, std)
61.9
61.87 + 0.09
62.2 + 0.09
62.82 + 0.17
63.77 + 0.14
5
Discussion
The localization problem includes a large group of systems and methods that by using a sensory system and a computational algorithm aims find the location of observer on a given topological map. Depending on the type of sensory system, several approaches have been presented in the literature. However, the existing solutions, even the data-driven methods, mostly work for a specific modality. This is mainly because these algorithms are built based on some strong assumptions about the characteristic of input data like image features or sensory system like the projection model of the camera. These assumptions can easily be violated if the algorithm applies on different data modality like RF signals or vice versa. In this work, we formulated the localization problem in the context of manifold learning by finding the 2D intrinsic space of data that leads to more generic solution for large family of localization problems. Finding an isometric embedding for the realistic noisy data under limited amount of samples, is highly erroneous and the computed embedding usually suffers from a severe distortion. Such a distortion imposes difficulty in finding a precise transformation to the topologiaal map in an weakly-supervvin and the transformation to the target topological map jointly by gradient descent optimization. Since the exact position of target on the topological map is unknown for the model, employed optimal transport for finding the correspondence (i.e. coupling) between the intrinsic embedding and the target geometry. Our experiments on both synthetic and realistic data with different modalities show the effectiveness of the proposed method.
style. In this work, proposed to learn the embedding
Broader Impact
In a positive prospect, we believe that our model contributes to further the development of less -omain--pecific framework for the indoor localization problem and can inspire future research works. This can drive many applications of localization in other less developed domains than vision-base systems.
One of the limitations of our proposed method is that we assume the main source of temporal variations in measured signal is due to the displacement of a single agent/person in the environment. However, this assumption can be violated when the motion of other objects/people in the environment causes a large variation in the signal. Such uninformative variations with respect to the location of the observer can deteriorate finding the correct KNN samples on the data manifold. While minimizing the triplet loss, high dependency between the spatial and temporal information of the observer was assumed. This issue may be less important for the camera- baseed localization and in general for active localization problem (i.e. when the observer carries device that is being used for measurements like RF signal of a cell phone or a camera). In passive localization cases, like localization of the person in WiFi medium by the reflected signal from the surface of the body, motions of other people cause a catastrophic impact on the manifold learning as then the measured signal contains superposition of more than one sample on the data manifold.
Depending on the data modality in use, the measured signal can be almost identical for two different locations in the environment. For example, the high visual similarity of two rooms of a hotel can easily cause mistakes for many image-based even confuse human. Although this may not be the for WiFi localization, it can suffer as well when the geometry of the reflectors in the environment be very symmetrical with respect to a virtual line between the transmitter and receiver.
localization algorithms. The high visual similarity can
Acknowledgments and Disclosure of Funding
This work was funded by Qualcomm Technologies, Inc
10

---

References
A. Agrawal, A. Ali, and Boyd. Minimum-ddistorioon embedding. arXiv preprint arXiv:2103.02559, 2021.
C. Arth, M. Klopschitz, G. Reitmayr, and Schmalstieg, Real-time seff-localization from panoramic images on mobile devices. In 20/ 10th ieee international symposium mixed and augmented reality, pages 37-46 IEEE, 2011.
E. Brachmann and C. Rother. Learning less is more-6d camera localization via 3d surface regression. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4654-4662, 2018.
S. Brahmbhatt, J. Gu, K. K.m, J. Hays, and J. Kautz. Geometry-aware learning of maps for camera localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2616-2625, 2018.
C. Chen, B. Wang. C. X. Lu, N. Trigoni, and A. Markham. survey on deep learning for localization and mapping: Towards the age of spatial machine intelligence. arXiv preprint arXiv:2006.12567, 2020.
M. Cuturi. Lightspeed computation of optimal transportation distances. Advances in Neural Informa- tion Processing Systems, 26, 2013.
A. Davison, I. D. Reid, N. D. Molton, and O. Stasse. Monoslam: Real-time single camera slam. IEEE transactions on pattern analysis and machine intelligence, 1956:1015- 2007.
V. De Silva and J. B. Tenenbaum. Global versus local methods in nonlinear dimensionality reduction. Advances in neural information processing systems, pages 721-728, 2003.
E. W. Dijkstra et al. A note on two problems in connexion with graphs. Numerische mathematik, 1 (1):269 271, 1959.
C. J. Dsilva, R. Talmon, C. W. Gear, R. R. Coifman, and I. G. Kevrekidis. Data-driven reduction for multiscale stochastic dynamical systems. arXiv preprint arXiv:1501.05195. 2015.
J. Engel, V. Koltun, and D. Cremers. Direct sparse odometry. IEEE transactions on pattern analysis and machine intelligence, 40(3):611-625, 2017.
L. Fan, T. Li, Y. Yuan, and D. Katabi. In-home daily-life captioning using radio signals. In European Conference on Computer Vision (ECCV), 2020.
A. Genevay, G. Peyré, and M. Cuturi. Learning generative models with sinkhorn divergences. In International Conference on Artificial Intelligence and Statisties, pages
PMLR, 2018.
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR ( volume 2, pages 1735-1742. IEEE, 2006.
J. F. Henriques and A. Vedaldi. Mapnet: An allocentric spatial memory for mapping environments. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2018.
A. Kendall, M. Grimes, and R. Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In Proceedings of the IEEE international conference on computer vision, pages 2015.
M. Lee. Smooth manifolds. Introduction to Smooth Manifolds, pages 1-31. Springer, 2013.
J. McQueen, M. Meila, and D. Joncas. Nearly isometric embedding by relaxation. Advances in Neural Information Processing Systems, 29:263 2016.
Mémoli. Graserewasserstein distances and the metric approach to object matching. Foundations of computational mathematics, 11(4))4174877, 201

---

R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos. Orb-slam: versatile and accurate monocular slam system. IEEE transactions on robotics, 31(5):114-1116, 2015.
G. Pai, R. Talmon, A. Bronstein, and R. Kimmel. Dimal: Deep isometric manifold learning using sparse geodesic sampling. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 819- 828.. IEEE, 2019.
Pajdla and V. Hlavac. Zero phase representation of panoramic images for image based localization. In International Conference Computer Analysis of Images and Patterns, pages 550-557. Springer, 1999.
E. Parisotto and R. Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. arXiv preprint arXiv: 1XX2v:7702.0866 2017.
G. Peyré, M. Cuturi, and Solomon, Gromov-wasserstiin averaging of kernel and distance matrices. In International Conference on Machine Learning, pages 2664 2672. PMLR, 2016.
G. Peyré, M. Cuturi, al. Computational optimal transport: With applications data science. Foundations and Trends@ in Machine Learning, 11(5-6):355-66 007-335556667--00 2019.
G. Rosman, M. M. Bronstein, A. M. Bronstein, and R. Kimmel. Nonlinear dimensionality reduction by topologically constrained isometric embedding. International Journal of Computer Vision, 89 (1):56-68, 2010.
P.-E. Sarlin, A. Unagar, M. Larsson, H. Germain, C. Toft, V. Larsson, M. Pollefeys, V. Lepetit, L. Hammarstrand, Kahl, et al. Back to the feature: Learning robust camera localization from pixels to pose. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3247-3252 2021.
T. Sattler, Q Zhou, M. Pollefeys, and L. Leal-Taixe. Understanding the limitations of cnn-based absolute camera pose regression. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 3302-3312, 2019.
B. Shen, F. Xia, C. Li, R. Martín-Martín L. Fan, G. Wang, S. Buch, C. D' D'Arpino, S. Srivastava, L. P. Tchapmi, K. Vainio, L. Fei-Fei, and S. Savarese. igibson, simulation environment for interactive tasks in large realistic scenes. arXiv preprint arXiv:20/2.029244 2020.
J. Solomon, G. Peyré, V. G. Kim, and S. Sra. Entropic metric alignment for correspondence problems. ACM Transactions on Graphics (TOG), 35(4):1-13, 2016.
W. Sturzl and H. A. Mallot. Efficient visual homing based on fourier transformed panoramic images. Robotics and Autonomous Systems, 54(4):300-313, 2006.
J. B. Tenenbaum, V. De Silva, and J. C. Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319-2555 2000.
A. Toker, Q. Zhou, M. Maximov, and L. Taixé. Coming down to earth: Satellite-to-stre view synthesis for geo-localization. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 6488-6497, 2021.
N. Verma. Distance preserving embeddings for general n-dimensional manifolds. In Conference on Learning Theory, pages 32-1. JMLR Workshop and Conference Proceedings, 2012.
K. Q. Weinberger and L.. K. Saul. Unsupervised learning of image manifolds by semidefinite programming. International journal of computer vision, 70(1):77-90, 2006.
F. Xia, W. B. Shen, C. Li. P. Kasimbeg, M. E. Tchapmi, A. Toshev, R. Martín-Martin and S. Savarese. Interactive gibson benchmark: A benchmark for interactive navigation cluttered environments. IEEE Robotics and Automation Letters. 5(2):713-720, 2020.
12


Reviews:
[{"review_id": "zVD1Bq8g-MS", "reviewer": "Reviewer_ynTh", "summary": "The reviewed paper considers the problem of 2D localization of an observer on a given topological map. A data-driven approach is proposed to solve the problem. While the topological map of the environment is required, no prior information is needed about the nature of the input signal. Moreover, no supervision (in terms of pairs of signal measurement and observer's location) is needed.\n\nThe work formulates the localization problem as the problem of learning a low-dimensional manifold, representing the input in their intrinsic space and then transport them to the given topological map by inferring correspondences between the two spaces.\nThe work is built on two intuitions:\n1. that the transformation from high-dimensional input to the intrinsic space can be modeled as a mapping that can be learned through a neural network;\n2. temporal data is subject to gradual changes as the observer moves, suggesting that the data reside on a smooth, locally connected manifold;\n\nResults from experiments on three different environments (and different sensory inputs) support the claims.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed solution to the 2D localization problem appears to be technically sound and well documented.\nThe document is clearly written and organized. \nThough the amount of information and references is large, the document leaves a reader familiar with the subject with enough pointers to reproduce the work.\n\nA few minor typos:\n- Line 162: \"Euc**l**idean\"\n- Line 186: \"Re**q**ularizing\"\n- Line 252: Missing reference to a figure is shown as \"??\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should consider a more extensive description of the limitations of the proposed approach. For example, in the experiments reported in the document, the only source of disturbance/signal was the observer and its motion itself. What happens when other people/objects occupy/walk the environment at the same time? Obviously, in these scenarios, solutions to localization involving cameras have a clear advantage, but it would be useful to the reader to get a sense of what kind of problems can arise in those scenarios.\n\nNo clear presence of potential negative social impact in this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "GmHM7qEWc8", "reviewer": "Reviewer_TceL", "summary": "This paper presents a new method for localization while being agnostic to input measurement signal's modality. Assuming a map is given, localization inside the map is formulated as a deep manifold learning problem, without needing the supervision of ground truth positions for each input measurement. The method utilizes differentiable optimal transport to solve the correspondence challenge in this formulation. The method is tested on several simulated image datasets and one real-world WiFi dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strength*:    \n1. The problem formulation based on manifold learning seems novel. And the use of optimal transport is an interesting approach.   \n2. The method allows modal-agnostic localization, due to the use of deep networks (similar to PoseNet).\n3. The method is tested on three different types of experiments, showing satisfactory results.\n\n*Weakness*:   \nMajor:   \n1. Lack of many important technical details:   \n    1.1. Some important symbols are not explained, which makes it very difficult to understand what has been done. For example, what is $y_j$? And how is $\\Omega_t$ sampled (sparely sampled on the trajectory only, or randomly and densely covering the testing environment)? What is V, E in Figure 2?  \n    1.2. The method did not mention at all how the map $\\Phi$ can handle variations in $\\Omega_s$ caused by camera rotations, which is a key question on whether the map can localize the camera when it takes two photos at the same position but with different orientations.     \n    1.3. How are the positive and negative samples in the triplets generated? What's the influence of the hyperparameters in this module (3.3.1) on the whole method?\n    1.4. There is no experiment/ablation study to justify the need/importance for equation (6) instead of Euclidean distance in a local area.\n2. Lack of discussions of the failure cases. Would this method fail in some cases? For example, the experiments never showed the trajectory in symmetric environments/maps. If $\\Omega_t$ is a square or rectangle, how can this method handle the rotational ambiguities? How does it know that an image is taken at the top-left corner instead of the top-right corner (because the optimal transport cost could be the same in this case)?\n3. Lack of comparisons to baseline methods, and related work discussions on other weakly supervised training of the localization function $\\Phi$ (essentially a PoseNet [Kendall 2015]).\n4. Lack of real-world image-based localization experiments.\n\nMinor:   \n5. Section 3.2 may be shortened since many equations are NOT proposed by this work, but are well-known in the community already. This would save space to include the above missing important details.    \n6. A topological map has a special meaning in SLAM and localization (no global metric information), which is different from the one used in this paper. It is suggested to change the term to a metric map, because it seems that you do need to make metric measurements (even if it could be noisy) to construct $\\Omega_t$.    \n7. Is it stable to train the deep localization network with the differentiable Sinkhorn? I wish to see some training losses.   \n8. What if you use L2 instead of KL for L(.,.)? Any ablation study? Do you have any reference to support your claim that \"choosing Eucidean distances for L is ill-posed\"?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I like the raw idea of the paper, and the problem being addressed is an important one in robotics. However, the lack of details and the quality of the presentation make it very difficult for me to recommend this paper for acceptance.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "1gEPzakgp3C", "reviewer": "Reviewer_w8Mg", "summary": "This paper presents a novel generic localization framework that is agnostic to the data modalities and doesn't require correspondence labels. The paper demonstrates the effectiveness of their framework through experiments on three different environments, each of which has a different data modality. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nS1 The idea of designing a generic localization framework is interesting and can inspire future research works. \n \nS2 The design of the proposed framework is novel and clever.\n\nS3 The paper is well written and easy to follow. The theoretical motivation part in Section 3.1 is interesting and inspiring.\n\nS4 The experiments cover three different data modalities, which is great. The experiment results are also very strong. \n\n\nComments\n\nC1 I wonder how the proposed framework performs in buildings where there are a lot of similar places. For example, in a hotel, many rooms are identical. If we look at the corridors of a hotel, an apartment, or a dormitory, there could be a lot of visually similar locations. \n\nC2 In Sections 4.2 and 4.3, do we assume that the moving distances between any two consecutive observations are identical and preknown? This seems to be true for the maze environment. I'm not sure if the proposed framework relies on this assumption.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an algorithm for 2D topological localization that is largely agnostic to the type of observation data. Formulated as a deep manifold learning problem, the framework transforms the (potentially) high-dimensional observation into a low-dimensional embedding. The method then uses optimal transport to identify the correspondence between this representation and that of the topological map. Importantly, this correspondence is learned without ground-truth correspondences. The paper presents experimental results using simulated camera observations as well as radio frequency (WiFi) observations.

Localization has been researched in the robotics community for several decades and is considered by many to be a problem that is all but solved. As the paper notes, traditional methods employ measurement models that are specific to the particular sensing modality(ies). To some extent, these models provide an abstraction of the specific sensor, and the underlying localization algorithms tend to be more sensor-agnostic. The capacity for the end-to-end framework to be truly independent of the particular sensor, however, is compelling. The way in which the method employs manifold learning and optimal transport is interesting and is technically sound. The experiments demonstrate the algorithm's ability to generalize to very different sensing modalities, however experiments involving real rather than simulated images would provide more compelling evidence. The reviewers raised a few concerns regarding the need for a clearer discussion of the algorithm's limitations and the omission of various technical details, which the authors largely clarified in their response. These issues should be addressed and the authors are encouraged to consider the possibility of including an experimental evaluation on real-world imagery.
Decision: Accept (Poster)