Representation Learning Through Latent Canonicalizations

1 INTRODUCTION . Most state-of-the-art visual recognition models rely on supervised learning using a large set of manually annotated data ( Krizhevsky et al. , 2012 ; He et al. , 2015 ; He et al. , 2017 ; Redmon & Farhadi , 2018 ) . As recognition task complexity increases , so does the number of potential real world variations in visual appearance and hence the size of the example set needed for sufficient test time generalization . Unfortunately , large labeled data sets are expensive to acquire ( Deng et al. , 2009 ; Zhou et al. , 2017 ) , and may even be infeasible for applications with evolving data distributions . Often a large portion of the variance within a collection of data is due to task-agnostic factors of variation . For example , the appearance of a street scene will change substantially based on the time of day , weather pattern , and number of traffic lanes , regardless of whether cars or pedestrians are present . Ideally , the ability to recognize cars and pedestrians would not require labeled examples of street scenes for all combinations of times of day , weather conditions , and geographic locations . Rather it should be sufficient to observe examples from each factor independently and generalize to unseen combinations . However , often the in-domain labeled data available may not even linearly cover all factors of variation . This calls for methods that encourage such sample efficiency by focusing on the individual complexities of the factors of variation , as opposed to their product . Prior approaches to learning representations which isolate factors of variation in the data have typically regularized the representation itself , with the aim of learning “ disentangled ” representations ( Kulkarni et al. , 2015 ; Chen et al. , 2016 ; Higgins et al. , 2017a ; b ; Burgess et al. , 2018 ; Kim & Mnih , 2018 ) . Alternatively , one might regularize the way the representation can be manipulated rather than the representational structure itself . Here , we take such an approach by introducing latent canonicalization , in which we constrain the representation such that individual factors can be clamped to an arbitrary , but fixed value ( “ canonicalized ” ) by a simple linear transformation of the representation . These canonicalizers can be applied independently or composed together to canonicalize multiple factors . We assume access to a large collection of source domain data with meta-labels specifying the factors of variation within an image . This may be available from meta-data , attribute labels , or from hyperparameters used for generation of simulated imagery . Latent canonicalizers are optimized by a pixel loss over pairs of ground-truth canonicalized examples and reconstructions of images with various factors of variation whose representation has been passed through the relevant latent canonicalizers . By requiring the ability for manipulation of the latent space according to factors of variation , latent canonicalization encourages our representation to linearize such factors . We evaluate our approach on its ability to learn general representations after observing only a subset of all potential combinations of factors of variation . We first consider the simple dSprites dataset , introduced to study disentangled representations ( Matthey et al. , 2017 ) and show qualitatively that we can effectively canonicalize individual factors of variation . We next consider the more realistic , though still tractable , task of digit recognition on street view house numbers ( SVHN ) ( Netzer et al. , 2011 ) with few labeled examples . Using a digit simulator , we train our representation with latent canonicalization along multiple axes of variation such as font type , background color , etc . and then use the factored representation to enable more efficient few-shot training of digit recognition in the real data . Our method substantially increased performance over standard supervised learning and fine-tuning for equivalent amounts of data , achieving digit recognition performance that was only attainable with 5× as much SVHN labeled training data for the best-performing baseline method . Our experiments offer promising evidence that encoding structure into the latent representation guided by known factors of variation within data can enable more data efficient learning solutions . 2 RELATED WORK . 2.1 DISENTANGLING . A number of studies have sought to learn low-dimensional representations of the world , with many aiming to learn “ disentangled ” representations . In disentangled representations , single latent units independently represent semantically-meaningful factors of variation in the world and can lead to better performance on some tasks ( van Steenkiste et al. , 2019 ) . This problem has been most commonly studied in an unsupervised setting , often by regularizing latent representations to stay close to an isotropic Gaussian ( Higgins et al. , 2017a ; b ; Burgess et al. , 2018 ; Kim & Mnih , 2018 ) . Other popular unsupervised approaches include maximizing the mutual information between the latents and the observations ( Chen et al. , 2016 ) and adversarial approaches ( Donahue et al. , 2016 ; Radford et al. , 2015 ) . When supervision on the sources of variation is available , it is possible to use this in a weak way ( Kulkarni et al. , 2015 ) . Many of these works have explicitly endeavored to learn semantically meaningful representations which are both linearly independent and axis-aligned , such that individual latents correspond to individual factors of variation in the world . However , recent work has questioned the importance of axis-aligned representations , showing that many of these methods rely on implicit supervision and finding little correspondence between this strict definition of disentanglement and learning of downstream tasks ( Rolinek et al. , 2018 ; Locatello et al. , 2019 ) . Further , while axis-alignment is useful for human interpretability , for the purposes of decodability , any arbitrary rotation of these latents would be equally decodable so long as factors are linearly independent ( Locatello et al. , 2019 ) . In this work , we use explicit supervision in a simulated setting to encourage linear , but not necessarily axis-aligned representations . 2.2 SIM2REAL . The setting of near-unlimited simulated data with ground truth labels and scarce real data occurs often in computer vision and robotics . However , the domain gap between simulated and real data reduces generalization capacity . Many approaches have been proposed to overcome this difficulty which are broadly referred to as sim2real approaches . A simple approach to closing the sim2real gap is to train networks with combinations of real and synthetic data ( Varol et al. , 2017 ) . Transfer Learning and Few-shot Learning : Alternatively , synthetic data can be used for pretraining followed by fine-tuning on real data ( Richter et al. , 2016 ) —a form of transfer learning . Often , the reason one uses simulated data is that there is a shortage of labeled real data . In this situation , one may make use of few-shot learning techniques which seek to prevent over-fitting to the few examples by using a metric loss between data tripets ( Koch et al. , 2015 ) , comparing similarity between individual examples ( Vinyals et al. , 2016 ) or between a prototypical example per category and each instance ( Snell et al. , 2017 ) . A separate class of techniques use meta-learning to design a learning solution which is most amenable to learning from few examples ( Finn et al. , 2017 ) . However , these approaches may be sensitive to the ratio of synthetic and real data . Domain adaptation : With access to a large set of unlabeled real examples , domain adaptation techniques can be used to close the sim2real gap . One class of techniques focuses on matching domains at the feature level ( Ganin & Lempitsky , 2015 ; Long et al. , 2015 ; Tzeng et al. , 2017 ) , aiming to learn domain-invariant features than can make models more transferrable ( Zou et al. , 2018 ; Li et al. , 2018a ) . In fact , image-to-image translation focuses on the appearance gap by bridging the appearance gap in the image domain instead of feature space ( Shrivastava et al. , 2017 ; Liu et al. , 2017 ) . Domain adaptation can also be used to learn content gap in addition to appearance gap ( Kar et al. , 2019 ) . Additional structural constraints , such as cycle consistency , can further refine this image domain translation ( Zhu et al. , 2017 ; Hoffman et al. , 2018 ; Mueller et al. , 2018 ) . Finally , image stylization methods can also be adapted for style transfer and sim2real adaptation ( Li et al. , 2018b ) . Domain randomization : ( DR ) offers a surprising alternative ( Tobin et al. , 2017 ; Prakash et al. , 2018 ) by exploiting control of the synthetic data generation pipeline to randomize sources of variation . Random variations will likely be ignored by networks and thus result in invariance to those variations . A particularly interesting instantiation of DR was suggested in Sundermeyer et al . ( 2018 ) for pose estimation . Pose is an example of a factor of variation which could be ambiguous due to occlusions and symmetries . Instead of explicitly regressing for the pose angle , the authors propose an implicit latent representation . This is achieved by an augmented-autoencoder , a form of denoising-autoencoder that addresses all nuisance factors of variation as noise . This idea can be seen as a particular case of our method where all factors are being canonicalized at once instead of learning to canonicalize each individually . Another interesting example is the quotient space approach of ( Mehr et al. , 2018 ) removes pose information for a 3D shape representation by maxpooling encoder features over a sampling of object rotations . It , however , does not consider how to perform canonicalization as a linear transformation in latent space , nor how to compose different canonicalizers . 3 APPROACH . Our goal is to learn representations which capture the generative structure of a dataset by independently representing the underlying factors of variation present in the data . While many previous works have approached this problem by regularizing the representation itself ( Kulkarni et al. , 2015 ; Chen et al. , 2016 ; Higgins et al. , 2017a ; b ; Burgess et al. , 2018 ; Kim & Mnih , 2018 ) , here we take a different approach : rather than directly encourage the representation to be disentangled , we instead encourage the representation to be structured such that individual factors of variation can be manipulated by a simple linear transformation . In other words , we constrain the way that the representation can be manipulated rather than the structure of the representation itself . 3.1 LATENT CANONICALIZATION . In our approach , we augment a standard convolutional denoising autoencoder ( AE ) with latent canonicalizers . A standard AE learns an encoder , Enc , which takes as input a given image , x , and produces a corresponding latent vector , z . At the same time , the latent vector is used as input to a decoder , Dec , which produces an output image , x̂ . Both the encoder and decoder are learned according to the following objective , Lae , which minimizes the difference between the original input image and the reconstructed output image : z = Enc ( x ; θe ) ; x̂ = Dec ( z ; θd ) ; ( 1 ) Lae ( θe , θd ; x ) = min θe , θd ‖x− x̂‖22 . ( 2 ) To encourage noise-robustness , we augment the potential input images following previous work on denoising autoencoders ( Vincent et al. , 2008 ; 2010 ) , noising each raw input image , x , by adding Gaussian noise , blur , and random crops and rotations , leading to our noised input image , x̃ . In this work , we additionally constrain the structure of the learned latent space using a set of latent canonicalization losses . We define a latent canonicalizer as a learned linear transformation , C , which operates on the latent representation , z , in order to transform a given factor of variation ( e.g. , color or scale ) from its original value to an arbitrary , but fixed , canonical value . So that individual factors can be manipulated separately , we learn unique canonicalization matrices , Cj , for each factor of variation , j ∈ [ 1 , K ] , present in the dataset . In order to constrain the latent representation according to canonicalization along one factor , j , our method yields the following basic form : z ( j ) canon = Enc ( x̃ ; θe ) · Cj ; x̂ ( j ) canon = Dec ( zcanon ; θd ) ( 3 ) To supervise the learning of latent canonicalizers , we compare the images generated by canonicalized latents , x̂canon , to ground truth images with the appropriate factors of variation set to their canonical values , xcanon . Canonicalizers can also be composed together to canonicalize multiple factors ( e.g. , z ( j , k ) canon = z · Cj · Ck ) . During training , each image is passed through a random subset of both individual and pairs of canonicalizers . Given Q canonicalization paths for a given image , x , the corresponding canonicalization loss for that single image ( red in Figure 1 ) is written as : Lcanon = 1 Q Q∑ q ‖x̂ ( q ) canon − x ( q ) canon‖22 ( 4 ) Since many outputs are canonicalized , it is possible that the decoder will simply learn to only generate the canonical value of a given factor of variation . To prevent this form of input-independent memorization , we also include a “ bypass ” loss which is equivalent to the standard denoising autoencoder formulation ( green in Figure 1 ) defined in Equation ( 2 ) , thus forcing information about each factor to be captured in the latent vector , z . Finally , we ensure that our representation not only allows for linear manipulation along factors of variation , but does so while capturing the information necessary to train a classifier to solve our end task . To this end , we add a supervised cross-entropy loss , LCE , which optimizes our end task using available labeled data ( cyan in Figure 1 ) : LCE = y log ŷ ( 5 ) Combining equations 2 , 4 , and 5 with loss-scaling factors ( α and β ) gives us our final per-example loss formulation : L = y log ŷ + α‖x̂− x‖22 + β 1 Q Q∑ q ‖x̂ ( q ) canon − x ( q ) canon‖22 ( 6 ) In practice , two canonicalizers are chosen at random for each input batch , Ch and Cj , and the corresponding latent representation z is passed through { Ch , Cj , ChCj , CjCh } generating four unique canonicalized latents : { z ( h ) canon , z ( j ) canon , z ( h , j ) canon , z ( j , h ) canon } . A diagram of the method is shown in Figure 1 , illustrating each canonicalization path , the bypass path , and the classification model . All description thus far has focused on the single image loss for simplicity of presentation . The full model averages the per image loss over mini-batch before making gradient updates . Latent canonicalizer constraints : Our approach relies upon constraining the way a representation can be manipulated . As a result , the specific choice of constraints should have a significant impact on the representations which are ultimately learned . Here , we limit ourselves to only two constraints : the transformations must be linear and canonicalizers must be composable , at least in pairs . If we were to allow non-linear canonicalizers , there would be little incentive for the encoder to learn an easily manipulable embedding . This would only be exacerbated as the non-linear canonicalizer is made more powerful by e.g. , additional depth . By requiring canonicalizers to be composable , we encourage independence as each canonicalizer must be able to be applied without damaging any other . We explore some other potential constraints in Section 4.2.4 .

Summary: This paper proposes a method for unsupervised learning  of data representations that can be manipulated to remove factors of variation via linear transformations. These transformations are called canonicalizations in the paper. The canonicalizations are trained such that images for arbitrary values of the corresponding factor of variation are transformed into images with a fixed, canonical, value for that factor. The paper proposes a model architecture based on a denoising autoencoder, where the canonicalizations are applied to the encoded representation. It also proposes a loss function and sampling scheme for training the model. The paper demonstrates the method on the dSprites dataset, showing that it can effectively learn linear canonicalizations, and that multiple of these canonicalizations can be applied to the same image representation. The paper goes on to test the method on a digit classification task, where the model is trained to learn a representation in a simulator for SVHN data where the transformations to be canonicalized can be controlled, and  used to train a classifier on unseen real data from the SVHN test set.