Representation-Constrained Autoencoders and an Application to Wireless Positioning

1 INTRODUCTION . Autoencoders ( AEs ) are single- or multi-layer neural networks that are widely used for dimensionalityreduction tasks ( Hinton & Salakhutdinov , 2006 ; van der Maaten et al. , 2009 ; Goodfellow et al. , 2016 ; Baldi , 2012 ) . AEs learn low-dimensional representations ( also known as embeddings ) of a given high-dimensional dataset and have been shown to accurately preserve spatial relationships in both high- and low-dimensional space for a broad range of synthetic and real-world datasets ( van der Maaten et al. , 2009 ) . With the success of deep neural networks , AEs are also gaining increased attention for unsupervised learning tasks ( Geng et al. , 2015 ) . Notable application examples of AEs include : learning word embeddings ( Mikolov et al. , 2013 ) , unsupervised pretraining of deep networks to improve robustness ( Erhan et al. , 2010 ) , imputation of missing data ( Abdella & Marwala , 2005 ; Qiu et al. , 2018 ) , image compression ( Theis et al. , 2017 ) , extracting features that are robust to data corruption ( Vincent et al. , 2008 ) , and generative models ( Bengio et al. , 2013 ) . In all of these applications , AEs are typically trained in an unsupervised manner , i.e. , no labels are used , potential side information on the training data is ignored , or application-specific representation structure is not imposed during training . AEs that impose structural constraints on the latent variables include sparse AEs ( Hinton et al. , 2006 ; Makhzani & Frey , 2013 ) and variational AEs ( Doersch , 2016 ) . Sparse AEs enforce sparsity on the representations , which enables one to learn embeddings with low effective dimensionality caused by the sparsity constraints . Variational AEs are able to learn a latent variable ( representation ) drawn from a distribution that represents the high-dimensional input ( Doersch , 2016 ) ; such AEs have been shown to be able to generate complex data , such as handwritten digits ( Kingma & Welling , 2013 ) , faces ( Kulkarni et al. , 2015 ) , house numbers ( Kingma et al. , 2014 ) , or physical scene models ( Kulkarni et al. , 2015 ) . 1.1 AUTOENCODERS WITH REPRESENTATION CONSTRAINTS . Certain applications provide additional constraints or side information that can be imposed on the low-dimensional representations . Such side information may stem either from the dataset itself or from the application ( e.g. , from the way data was collected ) . One example arises when data is acquired over time . In such a scenario it may be natural to enforce constraints between representations by exploiting the fact that for temporally correlated datapoints , the associated low-dimensional representations should exhibit a certain degree of similarity . Another example arises in situations in which a subset of the representations are known a-priori , e.g. , when a subset of training data is annotated . The obtained annotation information can then be translated into representation constraints , which leads to semi-supervised training of an AEs . A concrete application example in which representation constraints are important is positioning of users in wireless communication systems using channel charting ( CC ) ( Studer et al. , 2018 ) . CC measures high-dimensional channel state information ( CSI ) of users transmitting data to a wireless access point or cell tower . By collecting CSI over time , one can train an AE for which the lowdimensional representations reflect relative user positions . While the method in ( Studer et al. , 2018 ) enables logical ( or relative ) localization without access to global navigation satellite systems ( GNSS ) and without expensive measurement campaigns ( Liu et al. , 2007 ; Xu et al. , 2016 ) , valuable side information should not be ignored when available . For example , users can only move with finite velocity . One could include this information when training an AE to ensure that temporally correlated datapoints are nearby in the representation space . In addition , one could imagine that certain points in space with known location ( e.g. , a coffee shop ) can be associated with measured CSI ; this helps to pin down a small set of spatial locations in the representation space , which may enable absolute positioning in space . Put simply , by enforcing constraints between representations , one can hope to improve the efficacy of AEs and enable interpretability of the learned low-dimensional representations . 1.2 CONTRIBUTIONS . This paper investigates four distinct representation constraints for AEs and provides a framework for including these during training . We propose constraints on pairs of representations in which either the absolute or relative distance among ( a subset of ) representations is enforced . We formulate these constraints as nonconvex regularizers , which can easily be included into existing deep-learning frameworks . We provide synthetic experiments to demonstrate that only a small amount of representation constraints are required to ( often significantly ) improve the representation quality of AEs . As a byproduct of our framework , we show that one of the proposed constraint types enables one to learn a parametric version of Sammon ’ s mapping ( Sammon , 1969 ) that avoids the need of costly out-of-sample extensions . We highlight the efficacy of representation constraints for the application of CC-based positioning in wireless systems . In an application context , we demonstrate that by combining partially-annotated datapoints with temporal constraints that arise from the acquisition process , the positioning performance of channel charting ( Studer et al. , 2018 ) can be improved significantly . 1.3 RELEVANT PRIOR ART . Autoencoders have been shown to consistently outperform other dimensionality-reduction algorithms on real-world datasets ( van der Maaten et al. , 2009 ) . Despite their success , AEs do not aim at preserving geometric properties on the representations ( such as distances between points ) , which is in stark contrast to , e.g. , Sammon ’ s mapping ( Sammon , 1969 ) or multidimensional scaling ( Davison , 1991 ) . Variations on classical AEs , such as denoising AEs , convolutional AEs , sparse AEs , or variational AEs , have been shown to further improve performance of the learned representations . van der Maaten ( 2009 ) proposes the use of AEs to learn a parametric mapping between highdimensional datapoints and low-dimensional representations by enforcing structure obtained via Student-t stochastic neighborhood embedding ( t-SNE ) . None of these methods incorporate potential side information that stems from the analyzed or the application at hand . We will extend AEs with representation constraints that further improve performance in certain applications . We note that our methods can be used in combination with existing techniques . Metric learning deals with the design of suitable measures of similarity ( Kulis et al. , 2013 ; Xing et al. , 2002 ; Weinberger & Saul , 2009 ; Baghshah & Shouraki , 2009 ; Hoi et al. , 2010 ) . The use of side information for metric learning has shown to ( often significantly ) improve performance ( Xing et al. , 2002 ; Kumar & Kummamuru , 2008 ) . Common constraints enforce that “ similar ” points lie close together in latent space ( Xing et al. , 2002 ) . The methods we propose resemble those of ( Klein et al. , 2002 ; Kumar & Kummamuru , 2008 ; Friedman , 1994 ; Schultz & Joachims , 2004 ; Baghshah & Shouraki , 2009 ) with the key difference that we are not interested in learning a metric but rather a representation that preserves structure provided by the application and dataset . Related to applications of interest , machine learning has slowly found its way into wireless communication systems . Past research has largely focused on replacing detection or decoding with deep neural networks ( O ’ Shea & Hoydis , 2017 ) . More recently , the efficacy of deep learning has been explored for wireless positioning ( Chen et al. , 2017 ; Vieira et al. , 2017 ; Arnold et al. , 2018 ) . All of these methods require extensive measurement campaigns in which one has to annotate CSI measurements with exact position information obtained from GNSS , such as the global positioning system ( GPS ) . To avoid the drawback of such supervised methods , channel charting ( CC ) ( Studer et al. , 2018 ) uses dimensionality reduction techniques to extract relative positions of users without the necessity of costly measurement campaigns . CC exploits the fact that CSI is high-dimensional , but it strongly depends on a user ’ s position , which is low-dimensional . Dimensionality reduction ( e.g. , via Sammon ’ s mapping or AEs ) applied to CSI measurements can be used to learn a so-called channel chart in which nearby points represent nearby positions in true space—exact position information is not available . However , wireless positioning provides a unique set of representation constraints originating from the acquisition process , which have been ignored in prior work . We propose to include such constraints into AEs which enables significantly improved CC performance and paves the way for exact positioning via CC . 2 REPRESENTATION-CONSTRAINED AUTOENCODERS . We briefly summarize the basics of AEs , introduce representation constraints , and then show a range of synthetic results that demonstrate the efficacy of representation constraints . 2.1 AUTOENCODERS IN A NUTSHELL . AEs take a high-dimensional dataset consisting ofN datapoints ( vectors ) xn ∈ RD , n = 1 , . . . , N , of dimensionD , and learn two functions : the encoder fe : RD → RD′ and the decoder fd : RD → RD′ . The encoder maps datapoints onto low-dimensional representations yn ∈ RD ′ , n = 1 , . . . , N , where D′ D is the dimension of the representations , and the decoder maps representations back to datapoints , i.e. , we have yn = f e ( xn ) and xn = fd ( yn ) , n = 1 , . . . , N. The encoder and decoder functions of AEs are implemented as multilayer ( shallow or deep ) feedforward neural networks that are trained to minimize the mean-square error ( MSE ) between the input and the output of the network . Specifically , one seeks to minimize the following loss function : L ( We , Wd ) = 1 N N∑ n=1 ‖xn − fd ( fe ( xn ) ) ‖2 . ( 1 ) Here , the parameters to be learned from the dataset { xn } Nn=1 are the weights and bias terms contained in the setsWe andWd that define the neural network forming the encoder and decoder , respectively . All norms used in this paper are ` 2-norms . While training of AEs is notoriously difficult , numerous strategies have been proposed in the past that improve the quality of training and reduce the probability of getting stuck in local minima ; see , e.g. , ( van der Maaten et al. , 2009 ; Kingma & Welling , 2013 ) , for the details . The D′-dimensional output of the encoder fe is typically of lower dimension than the intrinsic dimension of the manifold embedding the inputs xi in D dimensions . Hence , we have that xn ≈ fd ( fe ( xn ) ) , n = 1 , . . . , N , unless the dataset { xn } Nn=1 was D′-dimensional and we were able to learn the underlying structure . Nevertheless , AEs often find low-dimensional representations { yn } Nn=1 with small loss that capture the intrinsic dimensionality of the input datavectors .

Summary: The paper propose to learn autoencoders which incorporate pairwise constraints while learning the representation. Such constraints are motivated from the available side information in a given application. Inducing application-specific structure while training autoencoders allows to learn embeddings with better neighborhood preserving properties. In wireless positioning application, the paper proposes fixed absolute/relative distance and maximum absolute/relative distance constraints. Experiments on synthetic and real-word datasets show improved performance with the proposed approach.