Leveraging affinity cycle consistency to isolate factors of variation in learned representations

Identifying the dominant factors of variation across a dataset is a central goal of representation learning . Generative approaches lead to descriptions that are rich enough to recreate the data , but often only a partial description is needed to complete downstream tasks or to gain insights about the dataset . In this work , we operate in the setting where limited information is known about the data in the form of groupings , or set membership , and the task is to learn representations which isolate the factors of variation that are common across the groupings . Our key insight is the use of affinity cycle consistency ( ACC ) between the learned embeddings of images belonging to different sets . In contrast to prior work , we demonstrate that ACC can be applied with significantly fewer constraints on the factors of variation , across a remarkably broad range of settings , and without any supervision for half of the data . By curating datasets from Shapes3D , we quantify the effectiveness of ACC through mutual information between the learned representations and the known generative factors . In addition , we demonstrate the applicability of ACC to the tasks of digit style isolation and synthetic-to-real object pose transfer and compare to generative approaches utilizing the same supervision . 1 INTRODUCTION . Isolating desired factors of variation in a dataset requires learning representations that retain information only pertaining to those desired factors while suppressing or being invariant to remaining “ nuisance ” factors . This is a fundamental task in representation learning which is of great practical importance for numerous applications . For example , image retrieval based on certain specific attributes ( e.g . object pose , shape , or color ) requires representations that have effectively isolated those particular factors . In designing approaches for such a task , the possibilities for the structure of the learned representation are inextricably linked to the types of supervision available . As an example , complete supervision of the desired factors of variation provides maximum flexibility in obtaining fully disentangled representations , where there is a simple and interpretable mapping between elements and the factors of the variation ( Bengio et al. , 2013 ) . However , such supervision is unrealistic for most tasks since many common factors of variation in image data , such as 3D pose or lighting , are difficult to annotate at scale in real-world settings . At the other extreme , unsupervised representation learning makes the fewest limiting assumptions about the data but does not allow control over the discovered factors of variation . The challenge is in designing a learning process that best utilizes the supervision that can be realistically obtained in different real-world scenarios . In this paper , we consider weak supervision in the form of set membership ( Kulkarni et al. , 2015 ; Denton & Birodkar , 2017 ) . Specifically , this weak set supervision assumes only that we can curate subsets of training data where only the desired factors of variation to be isolated vary , and the remaining nuisance factors are fixed to same values . We will refer to the factors that vary within a set as the active factors , and those that have fixed and same values as inactive . To illustrate this set supervision , consider the problem of isolating 3D object pose from images belonging to an object category ( say , car images ) . The weak set supervision assumption can be satisfied by simply imaging each object from multiple viewpoints . Note , this would not require consistency or correspondence in viewpoints across object instances , nor any target pose val- ues attached to the images . In practice , collecting multiple views of an object in a static environment is much more reasonable than collecting views of different objects with identical poses . In this paper we propose a novel approach for isolating factors of variation by formulating the problem as one of finding alignment between two sets with some common active factors of variation . Considering the application of synthetic-to-real object pose transfer , Figure 1 illustrates two sample sets of car images where pose is the only active factor in the first set P ( I|d0 ) of synthetic car images and the second set P ( I ) is comprised of both real and synthetic car images . Given these sets , without any other supervision , the aim is to automatically learn the embeddings that can find meaningful correspondences between the points in the two sets . The key idea behind our approach is a novel utilization of cycle consistency . A cycle consistent mapping can be described broadly as some non-trivial mapping that brings an input back to itself , and in our case the mapping is between sets of points in embedding space . We denote our application of cycle consistency as affinity cycle consistency ( ACC ) as it uses a differentiable version of soft nearest neighbors since the correspondence forming the cycle or not known a priori1 . Further , no explicit pairwise correspondence between the input sets is needed ; it is found by the loss . We posit that this process of finding correspondences is crucial to isolating the desired factors of variation : to match across sets , the representations must ignore commonality within a set ( the inactive factors ) and focus on the active factors common to both the sets . For example , ACC-learned embeddings from the two sets of car images in Figure 1 can isolate the object pose factor as that is the common active factor across both the sets . We also show how our ACC model can be generalized to the partial set supervision setting : ACC can learn to isolate factors of variation even when set supervision is provided for only one set , while the second set is virtually unrestricted . This has practical importance as it allows us to integrate unsupervised data during training . In Section 4.3 we show how this process can be applied to isolate 3D pose in real images without ever seeing any supervised real images during training . In the following two sections we cover the related works and formally introduce our ACC method . Given the novelty of our approach for isolating factors of variation , we present a progression of experiments to develop an intuition for the technique as it operates in different scenarios . In Section 4.1 we evaluate ACC in various settings using the synthetic Shapes3D dataset where the latent factor values are known , allowing a quantitative analysis . Later , in Section 4.2 we demonstrate the use of ACC in isolating handwritten digit style from its content ( class id ) . In Section 4.3 , we show how ACC can be applied in its most general form to isolate 3D object pose in real images with a training 1This specific loss has been used previously in Dwibedi et al . ( 2019 ) to align different videos of the same action , and here we show this loss is much more general . We term it affinity cycle consistency as opposed to the prior work ’ s terminology , temporal cycle consistency , to indicate as such . process that combines a collection of set-supervised synthetic data with unsupervised real images . We conclude with a discussion and analysis . 2 RELATED WORK . Disentangled representations . Most approaches toward disentangled respresentations are unsupervised , and are generally based on generative modeling frameworks such as variational autoencoders ( Kingma & Welling , 2014 ) or generative adversarial networks ( Goodfellow et al. , 2014 ) . The VAE is a latent variable model that encourages disentanglement through its isotropic Gaussian prior , which is a factorized distribution . Numerous variations of the VAE have been proposed to further disentanglement , and these include β-VAE ( Higgins et al. , 2017 ) , β-TCVAE ( Chen et al. , 2018 ) , FactorVAE ( Kim & Mnih , 2018 ) , DIP-VAE ( Kumar et al. , 2018 ) , JointVAE ( Dupont , 2018 ) , and ML-VAE ( Bouchacourt et al. , 2018 ) . InfoGAN ( Chen et al. , 2016 ) encourages an interpretable latent representation by maximizing mutual information between the input and a small subset of latent variables . In Hu et al . ( 2018 ) adversarial training is combined with mixing autoencoders . In Locatello et al . ( 2019 ) it is shown that true unsupervised disentanglement is impossible in generative models , and inductive biases or implicit supervision must be exploited . Supervision has been incorporated in different ways . Graphical model structures are integrated into the encoder/decoder of a VAE to allow for partial supervision ( Siddharth et al. , 2017 ) . In Sanchez et al . ( 2020 ) disentanglement without generative modeling is proposed employing similar set supervision , but requires sequential training to learn all the factors of variation so that those varying across a set may be encoded . In contrast to these approaches , ACC produces entangled representations which directly target and isolate factors of variation complementary to those for which the set supervision is known . Cycle consistency . Often , cycle consistency has been used as a constraint for establishing point correspondences on images ( Zhou et al. , 2016 ; Oron et al. , 2016 ) or 3D point clouds ( Yang et al. , 2020 ; Navaneet et al. , 2020 ) . In a different setting , the time window between the image frames of Atari games can be learned using a discrete version of cycle consistency ( Aytar et al. , 2018 ) . In contrast to Aytar et al . ( 2018 ) , we use a discriminative approach and do not recover a disentangled representation . Cycle consistency has also been used in disentangling factors of variation with variational autoencoders using weak supervision in the form of set supervision ( Jha et al. , 2018 ) . The most closely related work to ours is the work on temporal cycle consistency ( TCC ) ( Dwibedi et al. , 2019 ) , where a differential soft nearest-neighbor loss is used to find correspondences across time in multiple videos of the same action . Our approach differs in two key ways . First , we generalize the approach to a broader class of problems where the data does not provide an ordering and is less likely to permit a 1-1 correspondence between sets . Second , and most significantly , we show the fundamental constraint of TCC , that both sets must share common active factors of variation , can be relaxed to allow one unconstrained set ( no inactive factors ) , which allows for incorporating training data with no set supervision . Weak supervision . For recovering semantics , there exists several self-supervision methods ( Chen et al. , 2020 ; Misra & van der Maaten , 2019 ) that rely on elaborate data augmentation and selfsupervision tasks such as jigsaw and rotations . Augmentation can be effective if it is known how factors of variation act on the image space , but this is only true for some fully observable factors such as 2D position and orientation . This restriction similarly applies to models that bake transformations into the architecture , as with spatial transformers ( Hinton et al. , 2011 ) or capsules ( Jaderberg et al. , 2015 ) . Often , we use fully self-supervised methods for recovering semantics necessary for downstream tasks . In this work , we are interested in isolating geometric factors of variation such as pose that are difficult to annotate . In order to do this , we rely on set supervision ( Kulkarni et al. , 2015 ; Mathieu et al. , 2016 ; Cohen & Welling , 2015 ) . In Cohen & Welling ( 2015 ) the latent representations of the training images are optimized limiting view synthesis to objects seen at training time . In practice , getting full supervision with ground truth parameters for geometric entities such as lighting and pose is challenging . On the other hand , one can often capture videos where we fix one or more of these factors of variation , and allow the others to vary . This form of set supervision is a good tradeoff between labor-intensive manual annotation required for fully supervised methods , and fully self-supervised methods . 3D pose aware representations . An important factor of variation for many image tasks is 3D object pose , and not surprisingly there have been attempts to learn representations which encode this property . The SO ( 3 ) -VAE ( Falorsi et al. , 2018 ) places a uniform prior on the 3D rotation group SO ( 3 ) , which allows learning manifold-valued latent variables . Latent representations that are pose- equivariant have been proposed in Worrall et al . ( 2017 ) ; Rhodin et al . ( 2018 ) , and this allows for pose to be directly transformed in the latent space . Generative techniques for pose disentanglement include Kulkarni et al . ( 2015 ) ; Yang et al . ( 2015a ) . In Kulkarni et al . ( 2015 ) a simplistic experimental setting is considered ( fewer factors of variation with synthetic or grayscale images , and disentangling only the 1D azimuth angle ) .

Summary: The paper presents an approach to isolate factors of variation using weak supervision in the form of group labels. The proposed method Affinity Cycle Consistency (ACC)  claims to work with these group labels, which are weaker than the more common, one factor per group type labeling.  An important aspect of this approach is that it does not attempt to disentangle the factors of variation, but only capture (or isolate) them in the latent space.