Leveraging affinity cycle consistency to isolate factors of variation in learned representations
Authors: Kieran A Murphy
Varun Jampani
Srikumar Ramalingam
Ameesh Makadia
Venue:  (2021)

Abstract: Identifying the dominant factors of variation across a dataset is a central goal of representation learning. Generative approaches lead to descriptions that are rich enough to recreate the data, but often only a partial description is needed to complete downstream tasks or to gain insights about the dataset.  In this work, we operate in the setting where limited information is known about the data in the form of groupings, or set membership, and the task is to learn representations which isolate the factors of variation that are common across the groupings.  Our key insight is the use of affinity cycle consistency (ACC) between the learned embeddings of images belonging to different sets. In contrast to prior work, we demonstrate that ACC can be applied with significantly fewer constraints on the factors of variation, across a remarkably broad range of settings, and without any supervision for half of the data. By curating datasets from Shapes3D, we quantify the effectiveness of ACC through mutual information between the learned representations and the known generative factors. In addition, we demonstrate the applicability of ACC to the tasks of digit style isolation and synthetic-to-real object pose transfer and compare to generative approaches utilizing the same supervision.

Under review as a conference paper at ICLR 2021
LEVERAGING AFFINITY CYCLE CONSISTENCY TO ISO- LATE FACTORS OF VARIATION IN LEARNED REPRESEN- TATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Identifying the dominant factors of variation across dataset is central goal of representation learning. Generative approaches lead to descriptions that are rich enough recreate the data, but often only a partial description is needed to com- plete downstream tasks or to gain insights about the dataset. In this work, we oper- ate in the setting where limited information is known about the data in the form of groupings, or set membership, and the task is to learn representations which iso- late the factors of variation that are common across the groupings. Our key insight is the use of affinity cycle consistency (ACC) between the learned embeddings of images belonging to different sets. In contrast to prior work, we demonstrate that ACC can be applied with significantly fewer constraints on the factors of variation, across a remarkably broad range of settings, and without any supervision for half of the data. By curating datasets from Shapes3D, of ACC through mutual information between the learned representations and the known generative factors. In addition, we demonstrate the applicability of ACC to the tasks of digit style isolation and synthetic-to-red object pose transfer and compare to generative approaches utilizing the same supervision.
quantify the effectiveness
INTRODUCTION
Isolating desired factors of variation in dataset requires learning representations that retain infor- mation only pertaining to those desired factors while suppressing or being invariant to remaining "nuisance" factors. This is a fundamental task in representation learning which is of great practi- cal importance for numerous applications. For example, image retrieval based on certain specific attributes (e.g. object pose, shape, color) requires representations that have effectively isolated those particular factors. designing approaches for such a task, the possibilities for the structure of the learned representation inextricably linked to the types of supervision available. As an example, complete supervision of the desired factors of variation provides maximum flexibility in obtaining fully disentangled representations, where there is simple and interpretable mapping be- tween elements and the factors of the variation (Bengio et al., 2013). However, such supervision is unrealistic for most tasks since many common factors of variation in image data, such as 3D pose or lighting, are difficult to annotate at scale in real-world settings. At the other extreme, unsupervised representation learning makes the fewest limiting assumptions about the data but does not allow control over the discovered factors of variation.
The challenge in designing a learning process that best utilizes the supervision that can be realis- tically obtained in different real-world scenarios. In this paper. we consider weak supervision in the form of set membership (Kulkarni et al., 2015; Denton & Birodkar, 2017). Specifically, this weak set supervision assumes only that we can curate subsets of training data where only the desired factors of variation to be isolated vary, and the remaining nuisance factors are fixed to same values. We will refer to the factors that vary within a set as the active factors, and those that have fixed and same values as inactive. To illustrate this set supervision, consider the problem of isolating 3D object pose from images belonging to object category (say, car images). The weak set supervision assump- tion can be satisfied by simply imaging each object from multiple viewpoints. Note, this would not require consistency or correspondence in viewpoints across object instances, nor any target pose val-

---

Under review as a conference paper at ICLR 2021
I = 0
f
-
P(7)da)
P(I)
Figure I: Affinityy cycle consistency (ACC) yields embeddings which isolate factors of variation in dataset P(I). It leverages weak supervision in the form of set membership. such as the set of im- ages P(I(d)) rendered around given synthetic car (top, left). A cycle consistency loss encourages finding correspondence between sets of inputs by extracting common factors that vary within both sets and suppressing factors which do not. We show that ACC isolates nontrivial factors of variation, such as pose in the example above, even when only one of the sets has been grouped. Importantly, this allows the incorporation of data with no supervision at all, such as the images of real cars (bot- tom, left). The learned representations (right, + oc), contain only the isolated factor of variation (contrast the alignment here with the untrained representations shown in the middle, 0).
ues attached to the images. In practice, collecting multiple views of an object a static environment much more reasonable than collecting views of different objects with identical poses.
In this paper we propose a novel approach for isolating factors of variation by formulating the prob- lem one of finding alignment between two sets with some common active factors of variation. Considering the application of synthetic-to-real object pose transfer, Figure illustrates two sample sets of car images where pose is the only active factor in the first set P([[do) of synthetic car im- ages and the second set P(I) is comprised of both real and synthetic car images. Given these sets, without any other supervision, the aim is to automatically learn the embeddings that find mean- ingful correspondences between the points in the two sets. The key idea behind our approach is a novel utilization of cycle consistency. cycle consistent mapping can be described broadly as some non-trivial mapping that brings an input back to itself, and in our case the mapping is between sets of points in embedding space. We denote our application of cycle consistency as affinity cycle con- sistency (ACC) as it uses a differentiable version of soft nearest neighbors since the correspondence forming the cycle or not known a priori'. Further, no explicit pairwise correspondence between the input sets is needed; it is found by the loss. We posit that this process of finding correspondences crucial to isolating the desired factors of variation: to match across sets, the representations must ignore commonality within set (the inactive factors) and focus the active factors common to both the sets. For example, ACC-learned embeddings from the two sets car images in Figure can isolate the object pose factor that is the common active factor across both the sets.
We also show how our ACC model can be generalized to the partial set supervision setting: ACC can learn to isolate factors of variation even when set supervision is provided for only one set, while the second set is virtually unrestricted. This has practical importance as it allows us to integrate unsupervised data during training. In Section 4.3 show how this process can be applied to isolate 3D pose in real images without ever seeing any supervised real images during training.
In the following two sections we cover the related works and formally introduce our ACC method. Given the novelty of our approach for isolating factors of variation, we present progression of ex- periments to develop an intuition for the technique we evaluate ACC in various settings using the synthetic Shapes3D dataset where the latent factor values are known, allowing quantitative analysis. Later, in Section 4.2 we demonstrate the use of ACC in isolating handwritten digit style from its content (class id). In Section 4.3, we show how ACC can applied in its most general form to isolate 3D object pose in real images with a training
operates in different scenarios. In Section 4.1
'This specific loss has been used previously in Dwibedi et al. (2019) to align different videos of the same action, and here we show this loss is much more general. We term it affenity cycle consistency opposed to the prior work's terminology, temporal cycle consistency, indicate such.
2

---

Under review as a conference paper at ICLR 2021
process that combines collection of set-supervised synthetic data with unsupervised real images. We conclude with a discussion and analysis.
2 RELATED WORK
Disentangled representations. Most approaches toward disentangled respresentations are unsu- pervised, and are generally based on generative modeling frameworks such as variational autoen- coders (Kingma & Welling, 2014) or generative adversarial networks (Goodfellow et al., 2014). The VAE is a latent variable model that encourages disentanglement through its isotropic Gaussian prior, which a factorized distribution. Numerous variations of the VAE have been proposed to further disentanglement, and these include 3-VAE (Higgins et al., 2017), 3-TCVAE (Chen et al., 2018), FactorVAE (Kim & Mnih, 2018), DIP-VAE (Kumar ct al., 2018), JointVAE (Dupont, 2018), and ML-VAE (Bouchacourt al., 2018). InfoGAN (Chen et al., 2016) encourages interpretable latent representation by maximizing mutual information between the input and a small subset of latent variables. In Hu et al. (2018) adversarial training is combined with mixing autoencoders. In Locatello et al. (2019) it is shown that true unsupervised disentanglement is impossible in genera- tive models, and inductive biases or implicit supervision must be exploited. Supervision has been incorporated in different ways. Graphical model structures are integrated into the encoder/decoder of a VAE to allow for partial supervision (Siddharth et al., 2017). In Sanchez et al. (2020) disen- tanglement without generative modeling is proposed employing similar set supervision, but requires sequential training to learn all the factors of variation so that those varying across a set may be encoded. In contrast to these approaches, ACC produces entangled representations which directly target and isolate factors of variation complementary to those for which the set supervision is known.
Cycle consistency. Often, cycle consistency has been used as constraint for establishing point correspondences on images (Zhou et al., 2016; Oron et al., 2016) or 3D point clouds (Yang et al., 2020: Navaneet et al., 2020). In different setting, the time window between the image frames of Atari games can be learned using a discrete version of cycle consistency (Aytar et al., 2018). In contrast to Aytar et al. (2018), we use discriminative approach and do not recover disentangled representation. Cycle consistency has also been used in disentangling factors of variation with vari- ational autoencoders using weak supervision in the form of set supervision (Jha et al., 2018). The most closely related work to ours is the work on temporal cycle consistency (TCC) (Dwibedi et al., 2019), where differential soft nearest-neighbor loss is used to find correspondences across time in multiple videos of the same action. Our approach differs in two key ways. First, we generalize the approach to a broader class of problems where the data does not provide an ordering and is less likely to permit a 1-1 correspondence between sets. Second, and most significantly, we show the fundamental constraint of TCC, that both sets must share common active factors of variation, can be relaxed to allow one unconstrained set (no inactive factors), which allows for incorporating training data with no set supervision.
Weak supervision. For recovering semantics, there exists several self-supervision methods (Chen et al., 2020; Misra & van der Maaten, 2019) that rely on elaborate data augmentation and self- supervision tasks such as jigsaw and rotations. Augmentation can be effective if it is known how factors of variation act on the image space, but this is only true for some fully observable factors such 2D position and orientation. This restriction similarly applies models that bake transforma- tions into the architecture, as with spatial transformers (Hinton et al., 2011) or capsules (Jaderberg al., 2015). Often, we use fully self-supervised methods for recovering semantics necessary for downstream tasks. In this work, we are interested in isolating geometric factors of variation such as pose that are difficult to annotate. In order to do this, we rely on set supervision (Kulkarni et al., 2015; Mathieu et al., 2016; Cohen & Welling, 2015). In Cohen & Welling (2015) the latent rep- resentations of the training images are optimized limiting view synthesis objects seen at training time. In practice, getting full supervision with ground truth parameters for geometric entities such as lighting and pose is challenging. On the other hand, one can often capture videos where we fix one more of these factors of variation, and allow the others vary. This form of set supervision is a good tradcoff between labor-intensive manual annotation required for fully supervised methods, and fully self- relf-supervised methods.
3D pose aware representations. An important factor of variation for many image tasks is 3D ob- ject pose, and not surprisingly there have been attempts to learn representations which encode this property. The 50(3)- VAE (Falorsi et al., 2018) places a uniform prior on the 3D rotation group 53(3), which allows learning manifold-vallud latent variables. Latent representations that are pose-
3

---

Under review as a conference paper at ICLR 2021
equivariant have been proposed in Worrall et al. (2017); Rhodin al. (2018), and this allows for pose to be directly transformed in the latent space. Generative techniques for pose disentanglement include Kulkarni et al. (2015); Yang et al. (2015a). In Kulkarni et al. (2015) simplistic exper- imental setting is considered (fewer factors of variation with synthetic or grayscale images, and disentangling only the 1D azimuth angle).
AFFINITY CYCLE CONSISTENCY
Notation: We are given a finite set of n training samples S = {111, Lnn. where v, E X, and X' denote the instance space with dimensions D. Let ((x, w) X + IR be the function that maps the input vector to an embedding in E-dimensional space. We use an affinity cycle consistency loss on the embeddings TR. R., to learn the parameters: w² arg min ( w)).
Fioorsse Shape Soile Orienttion
Ohjechue Booha Staze SolA Dxediaion
Vall
Ocaciiu
WAB Poorr Commalor
Objectass hre
Shape. Scale Crimiation
Shope.
Figure 2: Sets with different active and inactive factors of variation from the Shapes3ID dataset (Burgess Kim, 2018). On the left, we have the same active factor of variation for both sets. In the middle, have overlapping active factors of variation. On the right, we have no known inactive factor of variation for set B. We find ACC to be effective in all these three scenarios.
Mini-batch construction using set-membership: Our goal to learn Φ X R such that the learned representations isolate certain desired factors of variation and are invariant to certain other factors variation. Core to the method is how the data is grouped: the affinity cycle consistency loss operates on sets, and strives to obtain 1-1 correspondence between two sets of points in the embedding space (Dwibedi et al., 2019). In the experiments of this paper, we either leverage natural groupings of images or, in the pursuit of insight, curate images into sets for mini-batch construction.
Let us refer to the two sets of images for particular mini- mini-batch by A C X and C X. In Figure 2, show with several active and inactive factors of variation from the Shapes3I) dataset (Burgess & Kim, 2018). In the most restrictive setting, both sets have identical active and inactivate factors of variation as shown in Figure 2(left); this the case of (Dwibedi al., 2019). ACC also functions in less restrictive scenarios where subset of active factors of variation are shared between sets, as in Figure (mmiddle), to the extreme where one of the sets is completely unconstrained without any inactive factors of variation, as in Figure 2(right).
Let the images in each of these sets be given by A {aI {ai..... ......................... an) and B {b (bi...., ... 6m }. re- spectively. Let us denote the associated embeddings as L = (4....,,,, (n and M = {m....., mm }. where 4, a(a, w) and m, = (b(b; w). Functionally, we parameterize & with the same neural net- work for both A and B. Let d(xx, y) denote distance metric between points in embedding space. The notion of cycle consistency is used in many different contexts, and we use the following definition.
Definition 1 (Cycle consistency) Given
sets of points L (4.....,44 (n) and M (mI..... mm.. ). we say that 4 E L is cycle consistent if 4 = arg migner d(m, where m arg minme d(le, m).
In other words, we say that a point 1, E is cycle consistent its nearest neighbor m E M also has 1; its nearest neighbor in L. In order to cycle consistency our loss function, we will use differentiable formulation of the cycle consistency loss. To get a differentiable loss, we approximate the nearest neighbor with soft nearest neighbor (Goldberger et al., 2004). which has also been used in many applications (Movshovitz- Attias et al., 2017; Rocco et al., 2018; Snell et al., 2017).
Definition 2 (Soft nearest neighbor) Given two sets of points L = (h..... 1n3 and M = (m1,... m, the soft nearest neighbor of l E in the set M is given by ΣØ= a,my,
where dj Σk-re e dl4,813777 and is a temperature parameter.
4

---

Under review as a conference paper at ICLR 2021
We first compute the soft nearest neighbor for 1 L mn EE= aymy. To satisfy the cyclic consistency constraint, 4 should cycle back to itself. In order to enforce this, the nearest neighbor of in should be 4. In the differentiable formulation, we map the problem of cycling back to 4 as classification problem (Dwibedi et al., 2019). Given that we have n points in I, compute logits given by Oj = - d(m,, 12): Vj E (1.....n) n and let á = softmax(o). Let denote the 1-hot encoding where 5j 1 and zero otherwise. Now the affinity cyclic consistency (ACC) constraint is enforced as the cross-entropy loss function Σド=1 -sylog(8) Note that the loss naturally prevents point collapse in embedding space as a trivial solution, because the softmax output compares the cycle back to the affinity of all points in T..
Double augmentation: In practice, real images can have many more active factors of variation than those which we desire to isolate. We introduce simple modification to the above method in order to suppress nuisance factors of variation in images which are easily augmented. Instead of enforcing cycle consistency from an image a E A to its soft nearest neighbor in B and then back to ag we allow the cycling back constraint to start from one augmentation of an image a' and return back to another augmentation of the same image a². By carefully selecting the augmentations d' and a" along certain active factors of variation, we can learn embeddings that are invariant to nuisance active factors of variation. We found the double augmentation to critical our pose transfer experiments involving real and synthetic cars in section 4.3.
4 EXPERIMENTS
We evaluate ACC's ability to isolate desired factors of variation in three settings: Shapes3D (Burgess & Kim, 2018), digit style isolation, and object 3D pose transfer from synthetic to real images.
4.1 SYSTEMATIC EVALUATIONS ON SHAPES3D
We quantitatively analyze ACC with the synthetic Shapes3D (Burgess & Kim, 2018) dataset, where can freely control the active and inactive factors of variation in the two sets and measure quanti- ties of interest in order to elucidate the inner workings of the method. Sample images from Shapes3I) are shown in Figure 3 and consist of a geometric primitive with a floor and background wall. There are six factors of variation in the dataset: three color factors (wall huc, object hue and floor hue) and three geometric primitive factors (scale, shape and orientation). The full dataset consists of every possible combination of these six discrete generative factors.
In order to train ACC with a particular generative factor inactive, for each training step we randomly sample from among its possible values and hold it fixed across set of inputs, while sampling uni- formly across the remaining factors to generate set of size 32. For example, Figure 3a shows two example sets with wall hue as an inactive factor. We train a network to embed the images into two dimensions and visualize one set of learned representations in Figure 3b; the mutual information between object and floor hues and the learned embedding is qualitatively evident. We measure the mutual information [UU G) (see Appendix B) between each of the generative factors and the em- beddings, and repeat each experiment for 50 random seeds. In each subplot of Figure 3c, different factors are made inactive during training. We show the (nonzero) mutual information present even in the output from untrained network, as well the result of training without any set supervision. In this baseline, all generative factors are active during training and nothing is suppressed in the learned embeddings. Interestingly, without supervision, there arises a significant distinction between the hue factors and the others, presumably out of salience with respect to the network's 's capabilities.
The subsequent subplots in Figure 3c show the result of training with various generative factor(s) rendered inactive, indicated by a shaded box. Clear trends arise which allow a more precise definition of active factor isolation: information with respect the inactive factors is noticeably suppressed while information about the remaining factors is enhanced in the learned representation. In other words, given weak set supervision over one factor of variation, ACC leads to learned representations known beforehand. When all three hue
which isolate factors of variation about which nothing factors are inactive we see scale, shape, and orientation feature most prominently in the learned representations, seemingly because the "easy" hue factors have all been suppressed.
Additionally, we compare to settings where only set A has supervision (inactive factors) and the other set B consists of random samples over the entire dataset. Strikingly, there is no significant change in the information content of the generative factors in the learned representations, meaning ACC performs just well in the much looser setting. Finally, in the bottom subplot of Figure 3c,
5

---

Under review as a conference paper at ICLR 2021
a
Example imago ants, wall hoto inactive
C
Uatralted
iructivve
Leatned. Reprosentations
Mutual Information
RivE vojiss lox
1(07: (0):G) (unta)
Oalend soal
Cakeal by skaper
Culond by reasiatioe
((F;C)) (nacs)
Well, abject. de Busr lax Besign ber segruneation
1xx
Slage
Was tae
Obber
Stotp
Geuerative Eactor G
Genmation
Figure 3: Analysis on Shapes3D. (a) An example pair of training sets where wall hue is the only inactive factor in each set. (b) train ACC on pairs of image sets as (a) and embed 2D. Each of the six plots displays the same 256 embeddings colored by their value for different generative factor. The bar chart shows the mutual information between each of the factors and the learned representation. (c) We repeat the experiments with 50 random seeds and display the spread in mutual information values as a violin plot. Each subplot shows a different split of active/inactive factors during training. The outpul from an untrained network is shown in black. In the middle two subplots, we compare to the setting with one unconstrained inpul set, iee with no inactive factors of variation. The bottom subplot compares to a contrastive method which augments the overall hue of each image twice and uses the two versions as a positive pair. In each setting, ACC successfully suppresses information about inactive factor(s) of variation and enhances information about active factors.
compare ACC results where all hue factors were inactive, to the case of hue double augmentation which suppresses all hue information from the learned embeddings. While the factor isolation effects are more pronounced in the latter, we emphasize that augmenting away nuisance factors is often not possible. This is the case in the middle two subplots, where augmenting hue would suppress all three hue factors indiscriminately and inhibit information with respect to the active variables. Thus ACC and double augmentation are complementary tools to operate on factors of variation in dataset.
4.2 DIGIT STYLE ISOLATION
Handwritten digits, such those from MNIST (LeCun & Cortes, 1998), have two main factors of variation: content and style. Here, content refers to the class of the digit (c.g., or 8) and the rest of the factors of variation this experiment, our aim is to learn embeddings that isolate digit style while being invariant to the digit class, with only supervision and embed to 8D using a convolutional network (See Appendix A for specifics). Images of the digit 9 are held out from training to probe the invariance of the learned embeddings to digit class.
be referred to style (stroke width, orientation, writing style etc.). In
the digit class. We group images by class into sets of size 64
Figure 4 (left) shows two-dimensional PCA plots of the learned embeddings next to those of un- trained embeddings. The PCA plots clearly indicate the stronger correlation between digit style (width, orientation etc.) and the learned embeddings in comparison to those of untrained embed- dings. As further analysis, we use digits from each of the 10 classes retrieve the nearest neighbor digits in other classes in Figure 4 (right). We compare to the representations yielded by two VAE-based approaches which utilize grouped data to separate factors of variation: CC- VAE (Jha et al., 2018) 8) in Figure 4 and ML-VAE (Bouchacourt et al., 2018) in Appendix F. Without having to learn a full description of the data, ACC yields style-correlated embeddings 100 times faster than the related generative approaches. This demonstrates the superior potential of ACC to isolate digit style without using any explicit supervision on styles.
4.3 OBJECT POSE TRANSFER FROM SYNTHETIC TO REAL IMAGES
We showcase the unique capabilities of ACC on the challenging task of 3D pose estimation of an object in image. A common data setting, in which there is an abundance of synthetic data

---

Under review as a conference paper at ICLR 2021
Untrained
Trained
Retrieval
8
0
7
r
7
P
0
/
12
3
5
/ P
2
0
1
2
3
6
5
? 8
1
2
a
B
7
dight
Test digit
1
1
5
4
6
6
1
f
2 3
+ 4
7 E 2
Ours, 1 epoch
0
2
4
S 6 8
2
PCA
0
2
3
5
6
P
> 8
PCA
>
4
5
f
3
4
R
8
1
2
3
5
7
8
2
8
6
9 8 P
9
/ /
3
9
7
4
5
6
7
PCA
PCA D
Jha 05 al., 100 spochs
Figure 4: Style isolation in MNIST. We define digit style as a combination of all factors of variation independent of the class. After training ACC with images grouped by digit, we evaluate the isolation of the style factors. We visualize embeddings from the test set using the top two PCA dimensions (accounting for more than 0.99 of the total variance in the trained embeddings). Before the network is trained, all the embeddinggs cluster logether (top left), while after training (top middle) the embed- dings fan out by style factors, primarily thickness and slant (this can be observed both within and across digits). The digit 9 is held out at training, yet embeddings of 9s (bottom middle) are similarly arranged by thickness and slant, showing ACC generalizes to unseen inactive factor values. On the right we show retrieval results, where the boxed images along the diagonal are queries and the other images in each row are the nearest embeddings for each digit class (all from the test set). ACC retrieves images closer in style than CC-VAE (Jha et al., 2018), generative approach which also utilizes set supervision. Notably, ACC required 100x fewer training steps, highlighting a benefü of learning partial descriptions through discriminative approaches as compared to generative models.
combined with unannotated real data, plays to the strengths of ACC. The method allows us to isolate pose information in learned representations by leveraging natural groupings of synthetic images where pose is the only active variable, even without any pose annotations at training. Additionally, an unconstrained second set provides a means to gradually incorporate unannotated real images which helps generalize object pose from the synthetic domain to the real.
We use the dataset included in KeypointNet (Suwajanakorn et al., 2018), which consists of render- ings of ShapeNet (Chang al., 2015) 3D models from viewpoints which randomly distributed over the upper hemisphere. Set supervision is provided by grouping images according to their source 3D model (as in the upper image of Figure 1). Other factors of variations such as object texture, lighting are also fixed, making viewpoint the only active factor within each set. We pair the synthetic images with real images from the CompCars (Yang al., 2015b) and Cars (Krause et al., 2013) datasets for the car category, and 1000 images from the Pascal3d+ (Xiang et al., 2014) training split for chairs. All images are tight cropped and resized 128x128.
In the first experiment, we discard pose annotations entirely and show that ACC yields represen- tations which are pose informative with respect to real images, solely using groupings of synthetic images by particular model. We train with set A purely synthetic and grouped by ShapeNet model, and set B unconstrained. For the first 10k iterations set B is synthetic images randomly sampled across all models and viewpoints, and for the following 10k iterations real images are factored in as 5.5% of each set B. We found it beneficial to suppress nuisance active factors of variation in the images, such the precise position of the object in the frame, by optimizing the double augmen- tation loss explained in Section 3. Each image is randomly augmented twice with a combination of cropping, recoloring, and edge enhancement via Sobel filter. The network adds few layers to ResNet (He et al., 2015) before embedding to 64 dimensions
the base of an Sregeeee-pre-trrainne (specifics in Appendix A). We find cosine similarity with 0.1 outperforms 1.2 distance.

---

Under review as a conference paper at ICLR 2021
Query
ACC, PascalBD+ retzieval
ShapeNet retrieval
Queey ResSes, PascadB1++ retcioned
ResNet, ShapeNes retrieval
Figure 5: Retrieval results from ACC and ResNet embeddings. For each query image from the Puscal3D+ test split, we display the four nearest neighbors embedding space, out of 3200, from the Pascal3D+ train split and the ShapeNet images. Note how ACC yields similar representations for images which are often visually quite different, in contrast to the ResNet output. This serves as qualitative evidence that pose is being effectively isolated in the ACC-trained embeddings.
We evaluate the learned representations the images in Pascal3D+ by using nearest neighbor lookup between embedded test images and a dictionary of 1800 synthetic images with ground-truth pose. We compare the 16.384-dimensional output from the ResNet base network and embed- dings learned with the VAE-based approaches of Jha et al. (2018) and Bouchacourt et al. (2018). Quantitative results are shown in Table and retrieval examples in Figure 5. We additionally report a flip invariant metric to highlight how well ACC performs modulo flip symmetry, a particular dif- ficulty for cars. We consistently outperform the baselines with the gap significant in many cases, especially for chairs, showing the efficacy of ACC. The significant difference between ACC and the generative approaches underscores the importance of meaningfully incorporating unannotated real images during training; there is no simple means to do so with either VAE-based method.
In the second experiment (Table 2), we make use of the pose annotations for the synthetic images by incorporating ACC into the spherical regression framework of (Liao et al., 2019). Specifically, we add small spherical regression head after the ACC-conditioned representations (Figure 6) and train on weighted sum of the two losses. Even without any real images during training. ACC improves performance, presumably by better conditioning the intermediate latent space. A significant boost to performance results when small amount of real images (2%) are titrated in gradually over training, for both object categories. See Appendix E for ablative studies.
Cars
Chairs
Flip inverient: min(8, 189" 6)
Plip invorient min(8, 187 Mail (")
Mod En (*)
Accdl 30
Med Far L
Acc0 30"
Med En (*)
Accde 30
Acc
ResNet (pee wained) CCVAE Ula 2018)
16,0
0.65
10.1
0.93
71.0
0,30
32.5
0,66
$4.8
0.26
30.4
0.48
79.5
0.19
41.3
0.35
ML-VAE (Bouchasount 2018)
75.6
0127
29.5
0.50
87.2
11.66
44.5
0,33
ACC (Oum)
14.1
0.65
8.4
0.95
35.1
0.47
25.7
0.55
Table 1: Pose estimation without any pose annotations during training. Median error and accuracy metrics on Pascal3D+ car and chair test datasets. We obtain the pose with nearest neighbor lookup into 1800 synthetic images with GT pose, using different embeddings shown in different rows. ACC outperforms the VAE-based approaches and the high dimensional ResNet embeddings.
Set A: Synth, constrained
Embeddings t...... desR {0;, 0z)s
Emmbeddnnggs my
Set B: Synth+real, unconstrained
Gice
LSR ALACC
Figure 6: Strengthening spherical regression with ACC. We append a spherical regression head (Liao al., 2019) to the network and find that an ACC loss on the intermediate embeddings signifi- cantly improves performance.
8

---

Under review as a conference paper at ICLR 2021
Cars
Chairs
Med Err (*)
Acc@ 300%
Med Err (*)
Acc@30 1
Liao et al. (2019)
12.3
0.85
30.8
0.49
ACC
11.0
0.79
28.1
0,52
ACC (2% unannotated real)
9.3
0.87
26.0
0.55
Table 2: Performance boost to spherical regression by incorporating ACC. We show the effective- ness of incorporating ACC as an additional loss term when the data consists of annotated synthetic images and unannolated real images. ACC provides means to incorporate the latter which signifi- cantly helps bridge the domain gap from synthetic to real pose estimation.
DISCUSSION
Leveraging only data groupings, ACC produces informative representations with respect factors of variation disjoint from the subset of factors for which there is weak supervision. The loss is optimized when approximate correspondence can consistently be found between all inputs set A and all inputs of set B, for all pairings of A and B. The inactive factor(s) are common to all elements of a set and therefore offer no distinguishing information to help with correspondence; thus they are left out of the representation. One set be unconstrained because only active factors present in both sets can be used to find correspondence, meaning the more constrained the two sets dictates which factors are extracted, For the example sets of Figure the color of the car cannot be used to find correspondence because it does not distinguish between the elements of set A.
It is evident in the mutual information measurements of Figure 3 that only subset of the active factors of variation are present in the learned representations. This can be partly attributed to the low design choice facilitate the measurement of mutual infor-
dimensionality of the embeddings- mation, which is notoriously problematic in higher dimensions though we show in Appendix B that the effect is also present for 64-dimensional embeddings. correspondence can be made when each element is embedded according to only a single active factor of variation common to both sets. This was the case for Dwibedi et al. (2019), where the progression of an action (e.g., bowling) was the only active factor of variation (with scene specifics being inactive, fixed per video). We show that ACC extends naturally to data with multiple active factors of variation. This seems to arise from randomness in the input sets: average the loss is decreased by embedding random points to more independent dimensions (Appendix D). In practice, factors differ in salience. The hue-related generative factors of Shapes3D are easier for the specific network extract, and training effectively ceases once a correspondence utilizing these factors is found. Similarly, nuisance factors of variation in the images of cars and chairs easier to extract than pose, which is why double augmentation helped to encourage the network to isolate pose.
The task of finding correspondence is naturally suited to domain transfer, showcased in the pose estimation experiments of Section 4.3. ACC provides a means to incorporate unannotated images from similar domain, as the loss incentivizes overlooking factors of variation which do not aid in the correspondence task. We found the fraction of images incorporated should remain small, presumably because is possible to use one factor to embed images from one domain and another for second domain, long the representations are co-located in embedding space.
The breadth of the three scenarios explored in this paper showcase the generality ACC, and there is no reason the method should be restricted to image data. The cycle consistency loss operates on embeddings and thus insensitive to the modality of the input data. Additionally, while we experimented with two forms of the distance metric in embedding space L2 and cosine similarity nothing about the loss necessitates these, and presumably in some scenarios employing other distance metrics structures of embedding space would be advantageous.
CONCLUSION
In this work, we show how affinity cycle consistency can be utilized to isolate factors of variation provided only weak set supervision. Through extensive experiments on synthetic and real data, we show the technique can be applied in a number of different set supervision scenarios, including one where only subset of the data requires any set supervision all. This case is particularly impor- tant as it allows training with unsupervised real image data, and we validate this with promising experiments the challenging problem of isolating 3DD object pose in real images.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learningg A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798-1828, August 2013. ISSN 0162-8828.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In AAAI, February 2018.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. https//giihlbbcoom//ecpmi//l/llp 2018.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Ree Rich 3D Model Repository. Technical Report arXiv:1512.0300 [cs.GR], Stanford University - Princeton University - Toyota Technological Institute at Chicago, 2015.
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of dis- entanglement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2610-2620, 2018.
N.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2172-2180, 2016.
Taco S. Cohen and Max Welling. Transformation properties of learned visual representations. In International Conference on Learning Representations (ICLR), 2015.
Emily Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In Advances in Neural Information Processing Systems, pp. 4414 4423, 2017.
Emilien Dupont. Learning disentangled joint continuous and discrete representations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- and Garnett (eds.), Advances in Neural Information Processing Systems, pp. 710-720. Curran Associates, Inc., 2018.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Tem-
poral sycle-consistency learning. CoRR, 2019.
Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S. Cohen. Explorations in homeomorphic variational auto-encoding. CoRR, abs/1801 0407.04689, 2018
Jacob Goldberger, Sam Roweis, Geoff Hinton, and Ruslan Salakhutdinov. Neighbourhood com- ponents analysis. In Proceedings of the 17th International Conference on Neural Information Processing Systems, NIPS'04, Cambridge, MA, USA, 2004. MIT Press.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems 27, pp. 2672-2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint arXiv: 15 /2.05/2222.... 12.0335 2015.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a strained variational framework. In International Conference on Learning Representations (ICLR), 2017.
10

---

Under review as a conference paper at ICLR 2021
Geoffrey E. Hinton, Alex Krizhevsky, and Sida D. Wang. Transforming auto-encoders. In Pro- ceedings of the 21st International Conference on Artificial Neural Networks (ICANN), pp. 44-51, 2011.
Qiyang Hu, Attila Szabó, Tiziano Portenier, Paolo Favaro, and Matthias Zwicker. Disentangling factors of variation by mixing them. In JEEE Conference on Computer Vision and Pattern Recog- nition, pp. 3399-3407, 2018. 2018.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp- 2017-2025, 2015.
Ananya Harsh Jha, Saket Anand, Maneesh Singh, and S. R. Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proceedings of the 35th Interna- fional Conference on Machine Learning (ICMI.), 2018.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
Alexander Kraskov, Harald Stöghauer, and Peter Grassherger. Estimating mutual information. Phys. Rev. E, 69:066138,, Jun 2004. doi: 10.1 www.10//hyysRee.6...6666. org/doi/ 10 1tt.:1...11111pheeeeeee
URL https: hatttssss/iiii... aps.
69. 066138.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydncy, Australia, 2013.
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances Neural Information Processing Systems, pp. 2539-2547, 2015.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018.
Yann LeCun and Corinna Cortes. 1998.
The
mnist database of handwritten digits.
http://yann.eee http://yww..ccc..oo/////////////
Shuai Liao, Efstratios Gavves, and Cees G. M. Snoek. Spherical regression: Learning viewpoints,
surface normals and 3d rotations on n-spheres. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Francesco Locatello, Stefan Bauer, Mario Lučić, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Frederic Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. International Conference on Machine Learning, 2019.
Michael Mathieu, Junbo Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann LeCun. Disentangling factors variation in deep representations using adversarial training. Proceedings of the 30th International Conference Neural Information Processing Systems, pp. 5047-5055. 2016.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions, 2019.
Yair Movshovitz-Atian Alexander Toshev, Thomas K. Leung, Sergey loffe, and Saurabh Singh. No fuss distance metrie learning using proxies. CoRR, abs/1703.07464, 2017. URL http: //arxiv.. or// hwtp///bb//1.....///// 2703.07464.
K L Navaneet, Ansu Mathew, Shashank Kashyap, Wei-Chih Hung, Varun Jampani, and R. Venkatesh Babu. From image collections to point clouds with self-supervised shape and pose networks, 2020.
11

---

Under review as a conference paper at ICLR 2021
Shaul Oron, Tali Dekel, Tianfan Xue, William T. Freeman, and Shai Avidan. Best-buddies similarity robust template matching using mutual nearest neighbors, 2016.
Helge Rhodin, Mathieu Salzmann, and Pascal Fua. Unsupervised geometry-aware representation for 3d human pose estimation. In The European Conference on Computer Vision (ECCV), September 2018.
Ignacio Rocco, Mircea Cimpoi, Relja Arandjelovic. Akihiko Torii, Tomás Pajdla, and Josef Sivic. Neighbourhood consensus networks. CoRR, 1101110011010, 2018. URL http: //axiv.. org/abs//i /1100.110... 10510.
Eduardo Hugo Sanchez, Mathicu Serrurier, and Mathias Ortner. Learning disentangled represen- tations via mutual information estimation. In The European Conference Computer Vision (ECCV), 2020.
2020.
N. Siddharth, Paige Brooks, lan-Willem de Meent, Alban Desmaison, Noah Goodman, Push- meet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi- supervised deep generative models. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5925-5935, 2017.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. CoRR, abs/1 2017. URL hwtp:ttt///aae htatt:://arxiv.oo/ org/ wwwpg////////////////////// abs/1703.0517
Supasorn Suwajanakor,, Noah Snavely, Jonathan Tompson, and Mohammad Norouzi. Discovery of latent 3d keypoints via end-to-end geometric reasoning. Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018
Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Inter- pretable transformations with encoder-decoder networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond PASCAL: benchmark for 3d object detection in the wild. In 2014 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 75 82, March 2014.
Jimei Yang, Scott E Reed, Ming-Hsuann Yang, and Honglak Lee. Weakly-superrvise disentangling with recurrent transformations for 3d view synthesis. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NIPS), pp. 1099-1107. Curran Associates, Inc., 2015a.
Lei Yang, Wenxi Liu, Zhiming Cui, Nenglun Chen, and Wenping Wang. Mapping in cycle: Sinkhorn regularized unsupervised learning for point cloud shapes, 2020.
Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine- grained categorization and verification. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), June 2015b.
Tinghui Zhou, Philipp Krähenbühl, Mathieu Aubry, Qixing Huang, and Alexei A. Efros. Learning dense correspondence via 3d-guided cycle consistency, 2016.
A IMPLEMENTATION DETAILS For all experiments we use the ADAM optimizer (8, 0.9, 8y=0.999). Padding for convolutional layers is always 'valid.'
A.1
SHAPES3D
For the experiments of Figure 3 we trained for 2000 steps with a learning rate of 3 10 10-5, We used stack size of 32 and squared 1.2 distance the embedding space metric, with temperature of 1.
12

---

Under review as a conference paper at ICLR 2021
Layer
Units
Kernel size
Activation
Stride
Conv2l)
32
3x3
ReLU
Conv2D
32
3x3
ReLU
Conv2D
64
3x3
ReLU
2
Conv2D
64
ReLU
Conv2D
128
3x3
ReLU
Conv2D
128
3x3
ReLU
2
Flatten
Dense
128
ReLU
Dense
Embedding dimension (2)
Linear
Table 3: Architecture used for Shapes3D experiments (Section 4.1). Input shape is [64, 64, 3].
Layer
Units
Kernel size
Activation
Stride
Conv2D
32
3x3
ReLU
Conv2D
32
3x3
ReLU
Conv2D
32
3x3
ReLU
2
Conv2D
32
3x3
ReLU
Conv2D
32
3x3
ReLU
Flatten
Dense
128
ReLU
Dense
Emhedding dimension (8)
Linear
Table 4: Architecture used for MNIST experiments (Section 4.2). Input shape is [28, 28, 1].
A.2 MNIST
For the MNIST experiments, the stack size is 64. We use learning rate of 10 and train for 1000 steps. We used squared L,2 distance as the embedding space metric and temperature of 1, though long the length scale set by the temperature larger than the initial point spread from the randomly initialized network, does not seem to matter.
A.3 POSE ESTIMATION
Layer
Units
Kernel size
Activation
Stride
ResNet50, up to conv4_block6
Conv2D
256
3x3
ReLU
Global Average Pooling
Flatten
Dense
128
tanh
Dense
Embedding dimension (64)
Lincar
Table 5: Architecture used for pose estimation experiments (Section 4.3). Input shape [128, 128, 3].
For both the pose estimation lookup and regression tasks, we use the same base network embed the images. For regression, the embeddings are then fed, separately for each Euler angle, as input to 128 unit dense layer with tanh activation, which is then split off into dense layers with 2 and 4 units and linear activation for the angle magnitude and quadrant, respectively, as in (Liao et al., 2019). The angle magnitudes are passed through spherical exponential activation function Liao et al. (2019), which is the square root of softmax. The magnitudes then compared with ground truth (|sincy]. (coséx)], ),. with spanning the three Euler angles, through a cosine similarity loss. The quadrant outputs trained as classification task with categorical cross entropy against the ground truth angle quadrants, defined as (sign(sino;). sign(cosp:)).
For the lookup task, the network trained for 20k steps, with the first half of training purely synthetic images, and then the second half with 5.5% real images folded into the unconstrained stack. For spherical regression, training proceeds for 60k steps with learning rate that starts 10-1 and
13

---

Under review as a conference paper at ICLR 2021
decays by factor of 2 every 20k steps. Each minibatch consists of pairs of image sets, each of size 32. We use cosine similarity and a temperature of 0.1 for lookup and 0.05 for regression. To maintain consistency between how the embeddings processed for the ACC loss and how they fed into the regression sub-network, the embeddings are 1.2-normalized to on the 64-dimensional unit sphere before the regression.
To more closely match the distribution of camera pose in real images, we filter the ShapeNet render- ings by elevation: 0.5 radians and 1.3 radians for the max elevation for cars and chairs, respectively.
B MUTUAL INFORMATION CALCULATION AND SHAPES3D IN HIGHER DIMENSIONS
Embedding dimensions: 641 Additive noise ~ (0, r)
Embedding dimensions: 4
Embedding dimensions:
Embedding dimensions: 64
0.5
0.5-
0.5
0.5+
Classiber accuracy
0.5
0.5+
0.5-
0.5
0.5
0.5
0.5
Wall lase
Object but
Floor bac
Seale
Shapo Oriematiin
Wall lane
Object late Floer lar
Soale Shapp Oriesdation
Wall lase
Objest Ins
Floor lom
Stale
Slape Orimstation
Wall lime
Object lise
Floor lnne
Seale dele
Stapo
Orieatation
Generative factor
G objject
Untrained
Inactive
Will, objort,
Figure 7: Probing information content in higher dimensions via classification. We repeat the ex- periments of Section 4.1 in 4, 16, and 64 dimensional embedding space, though with 10 replicas each instead of 50. As a proxy for the mutual information, we use the test set classification accuracy of simple fully connected networks trained to classify each of the six generative factors. As before, each subplot we display gray the baseline results from embedding with an untrained, randomly initialized network. Also before, the colors of each subplot indicate the same information as the shaded columns: which of the generative factors were inactive while training ACC. In the rightmost subplots, daausiaa-distributeee random noise added to the embeddings to effectively remove in- formation on length scales less than the characteristic length scale of the ACC loss, the square root of the temperature.
For the experiments Section 4.1, we used an embedding dimension in order to facilitate measuring the mutual information. We approximated the 2D distribution by embedding all N 480, 000 images of the Shapes3I) dataset and using histogram with V bins. The same bins were then used for all the various conditional distributions, where the embedding distribution conditioned each possible value of each of the six generative factors was evaluated. The values of entropy calculated from this simple method were found to insensitive to the number of bins, within range, and more reliable than the popular non panameerrttic approach of (Kraskov et al., 2004).
is fair to question whether the low dimensionality the embeddings affects the behavior of ACC. We repeat the experiments of Section 4.1 with higher dimensional latent spaces and probe the information content by training a simple classifier (3 layers with 32 units each, ReLU activation), using the learned representations input, for each of the generative factors.
14

---

Under review as a conference paper at ICLR 2021
The test set classification accuracies, shown in Figure 7, have many noteworthy quirks. As baseline, the embeddings output by a randomly initialized (untrained) network are, sensibly, more success- fully classified the number of dimensions increases. The isolation of active factors that occurs for two dimensional embeddings (Figure 3c) is more subtle in higher dimensions. As in 2D, a subset of the active factors feature more prominently in the learned representations, and that subset is more or less the same: unless all three hue factors are inactive, the three geometric factors (scale, shape and orientation) are hardly affected. When all three hue factors are inactive, information with respect to all three geometric factors is clearly enhanced, regardless of the dimension.
Because even random embeddings are easily parsed by a classifier in higher dimensions. we do not see the obvious suppression of inactive factors as in two dimensions. This is reasonable, however, given that the ACC loss operates over a characteristic length scale, set by the temperature parameter in both the soft nearest neighbor calculation and the classification loss. In other words, two embed- dings separated by much less than this length scale effectively have a separation of zero in the eyes of the loss, and there is no incentive to further collapse them. As information is about the capacity to infer the value of one random variable given another, it is only destroyed in the case where multiple inputs map to the same output. This depends on the granularity of observation, the same is different for float precision versus hyperopic ACC loss. Thus when the ACC training leads embedding separations with respect particular generative factor which much less than the characteristic length scale, the information content has been removed from the perspective of the loss.
what qualifies as
To be specific, when using L2 (Euclidean) distance as the similarity metric, the temperature T is the characteristic length scale. When using L2 squared distance, as in the MNIST and Shapes3D experiments, the square root of the temperature is the characteristic length scale.
We expect, then, that the learned embeddings in 64 dimensions contain information about the active factors on length scales greater than vF and about the inactive factors on length scales less than this. A simple test is to introduce random noise into embedding space, removing information on length scales less than that of the noise. We add embeddings during the training of the classifier, and show the resulting test set classification accu- racies (without noise) in the rightmost plot of Figure 7. The untrained network results are not very informative because the length scale of the embeddings whatever resulted from the randomly ini- tialized weights of the embedding network, which happened to be less than VF. The accuracies all hover around 1/n, with n the number of possible values for generative factor: random guessing, in other words. The classification accuracies for the learned representations, however, now display the same behavior as did the mutual information in two dimensions. The inactive factors are suppressed, and the same active factors are enhanced in each active-inactive factor scenario.
noise with variance 7 to the
While not so clean as the mutual information measurements, the classification accuracies help gen- eralize the notion of active factor isolation: the ACC loss organizes embedding space around a subset of active factors that permit a correspondence between input sets, and information with respect to inactive factors is destroyed to the degree that the ACC loss can do so.
C FOUR INACTIVE FACTORS FOR SHAPES3D: ALL HUE PLUS A GEOMETRIC FACTOR
We expand upon the results of Figure 3 by training with more constrained input sets. We show in Figure 8 results from experiments with four inactive factors: the three hue-related factors and then one of the geometric factors. The aggregate behavior essentially mirrors that of the bottom subplot of Figure 3c where the three hue factors are inactive, but where the additional inactive factor is suppressed.
noteworthy behavior is apparent when visualizing the embeddings directly for individual runs, as in Figure 8a, The shape generative factor, when active, is consistently partitioned in embedding space into only two groups. The four shapes are cube, cylinder. sphere, and pill; embedding space were partitioned perfectly by shape the mutual information would be the natural log of 4, to which it never gets particularly close. The colocated embeddings are cubes with cylinders and spheres with pills. Evidently whether the top of the shape is round flat is more salient than the other details, providing another example where the salience of factors affects their isolation in the learned embeddings.
15

---

Under review as a conference paper at ICLR 2021
b
Unirained
Autive: Shape Orimmtation
a
Learned Representations
Coloted wall
Coloeed by abjoct
Colored
Mutual Information
Astiives Stada, Orientation
re: (1 (nate)
Colored scale
Colored by
Colored orientatios
I(0; G) (mats)
Scale, Shage
02
Olppett Inn
Orimmtaion atore
Wall Ios
-10
Shage
Generative factor
Joe
Soals
Slope
Ohimtaaion
Wall
Obyeet
Fbot
Generative factor
Figure 8: Highly constrained inputs for Shapes3D experiments. We extend the experiments of Sec- tion 4.1 by constraining the inpul sets with one additional generative factor, to better probe the difference between the three hue and the three geometric factors. In (a) we show one example of learned representations where the three hue factors and scale are inactive factors in each training set. Interestingly, the shape factor - of which there are four possible values seems to be split into two groups, one with cylinders and cubes and the other with pills and spheres. We observed this particular grouping to happen in the majority of the cases for this active-inactive split, indicating another level of salience difference with respect to the embedding network. In (b) we measure the mutual information (U:G) with respect to all six generative factors, as in Figure 3c, where the fourth (geometric) inactive factor is the scale, shape, and orientation, respectively.
D MULTIPLE FACTORS VARIATION AND THE EFFECT OF SET SIZE
correspondence between two sets can be found with only a single factor of variation, why do the experiments of this paper suggest ACC isolates multiple factors of variation? To be specific, in almost all of the Shapes3ID experiments, multiple generative factors were present in the learned representations. Presumably correspondence between MNIST digits could be found using stroke thickness, yet the embeddings always contain slant information as well. the pose experiments, only embedding azimuth would suffice to allow correspondence between images, yet elevation information was also clearly present.
In Figure 9 we simple Monte Carlo experiment where two sets of points randomly sampled from a uniform distribution, representing ideal embeddings whose factors of variation are randomly distributed from the same distribution, and the value for the ACC loss is evaluated. The loss is averaged over 10,000 random draws. In the normal setting where the distribution space learned via training, temperature has little effect because the distribution can be expanded or contracted to best fit the length scale set by temperature. In this simulation, the distribution is fixed the temperature which optimizes the loss needs to be found. We observed that the length scale set by the temperature roughly scales with the average inter- inter-point distance, but not exactly, so the value was optimized numerically.
embedding
Interestingly, when the number of points in each set is larger than O(10), the ACC loss can be lowered by increasing the dimension of the distribution. In other words, there an incentive in the loss to find multiple independent factors of variation. Additionally, the effect grows as the set size grows.
16

---

Under review as a conference paper at ICLR 2021
b
1.2
Optimal average loss
300 (K..ooX--B....
Loss
0.66
Loss 1.14
1.0-
Dimension
Loss 1.39
Loss 1.53
0.8
100% Br Loss 2.65
Loss 0.84
10
100
ACC set size
Figure 9: The case for finding more than one factor of variation, through a simple example. We model the embeddings that would be learned from randomly distributed factors of variation as points sampled uniformly over the unil interval in one to five dimensions. (a) Displayed are three random draws, with set size equal to 4 and dimension and with corresponding loss values for the numerically optimized temperature T=0.05. See Appendix text for why temperature needs to be optimized in this simulation but not in general. The X and circle markers designate Set A and B. (b) Same, but for dimension 2 and numerically optimized temperature T=0.11. (c) The ACC loss averaged over 10,000 random draws, for varying set size and dimension of the uniform distribution. While mapping points from two sets along dimension allows correspondence we see that in the presence of stochasticity, multiple independent dimensions lead to lower average values for the loss when the set size is larger than O(10). We take this suggestive for why ACC pulls out more than just single factor of variation, when the factors are of similar salience.
be found,
E POSE ESTIMATION ABLATIVE STUDIES
17

---

Under review as a conference paper at ICLR 2021
Cars
Chairs
Cars
Chairs
Median error (*)
24
Accuracy 30
0.80
22
38
20
36
0.75
10
16
32
0.70
14
30
0.65
12
26
10% 0.05
0.0 0.01
0.05
0.01 0.02 0.05
Fraction of real images in set B
Fraction of real images in set B
Titration
No titration
Titration
No titration
11.8
30
0.83
11.6
Modian error
11.4
0.82
0.54
11.2
0.81
28
0.52
10.8
27
10.6
26
0.48
25
0.78
Cos
12
12 Sq
Cos
12
1.2 Sq
Cos
1.2
1,2 Sq
Cos
1.2
L. Sq
Similarity measure
Similarity measure
0.85
Median error (
30
0.56
12.0
30
0.83
29
20
0.81
0.52
11.0
27
10.5-
26
0.79
0.48
16
256
16
256
Latent space dimensionality
Latent space dimensionality
Median error (*)
18
30
0.84
0.57
17
30
0.80
0.92
0.56
15
29
0.78
0.55
28
27
12
32
48
64
16
32
40
48
Set size
Set size
Figure 10: Ablative studies with spherical regression ACC network. Error bars are the standard error of the mean over 10 random seeds for each configuration, with less than 1% of the runs discarded for lack of convergence. We show results on the Puscal3D test splüt for the car and chair categories. For each row, the training configuration is the same as described in Appendix A with only the listed aspect of training being changed. In the first row, no titration means the fraction of real images in set B are present from the beginning of training. The three similarity measures in the second row are cosine similarity, L2 (Euclidean) distance, and squared L2 distance.
18

---

Under review as a conference paper at ICLR 2021
F
EXTENDED DIGIT STYLE ISOLATION RESULTS
epoch
epochs
epochs
1,000 epochs
0
P
0
2
b
ACC (Ours)
0 01:30
7
6
7
7 2 2
&
Z
7
0 2
OZ
6
0
2
4
2 Q
2
2
3
7
V S
8
2
CC-VAE
(Jha 2018)
2
0 7
$
0 0128476949
0
8
0 8/2254 //28456799
/
0/2795676
0
0
O
3
6 F
0 2 3
4 8
0193 G
Q
5
7
2
2
A
R
A
2
ML-VAE
1
1 P
1 8
1 R
(Bouchacourt 2018)
A.A.0 4
0 2
0
9
0 2
6 9
D 2.33
0
7
2
2 9
2
Figure 11: Retrieval results over the course of training, comparison. We compare retrieval on the test set of MNIST at various stages of training ACC and the two VAE-based approackes mentioned in the main text. As in Figure 4, the query images are the boxed images along the diagonal, and each row is the nearest representative for each class in embedding space. Also as before, all cases digit 9 was withheld during training.
We compare digit style isolation MNIST using the output of ACC and the style part of the latent representations yielded by the VAE-based approaches Jha et al. (2018) and Bouchacourt et al. (2018). Interestingly, MI-VAE appears embed the digits with respect to stroke thickness and slant very similarly to ACC at the beginning training, long before any realistic images are able to be generated, but this clear interpretability of the embeddings fades as training progresses.
19


Reviews:
[{"review_id": "lLAI4msmLGO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes a method for representation learning in a setting where data is annotated by set membership (i.e. grouped data). More specifically, the authors aim to extract representations of the factors of variations that are shared across groups.\nTo do so, the proposed method embeds different sets into a shared latent space using a learning objective called Affinity Cycle Consistency (ACC). ACC imposes a soft version of the cycle consistency on the nearest neighbourg relationship between the two learned sets of embeddings.\nUsing this method, the authors show in experiments on 3DShapes and MNIST that the inactive features (features that fixed in each set) are removed from the embeddings that, in turn, better encodes for the active features (features that vary inside each set).\nThey also show how their methods can be used to accurately estimate the pose of objects (cars and chairs) in unlabeled real pictures by aligning them with sets of synthetic images of cars grouped by pose.\n\n\n################################################\n\nStrong points:\n\n-Representation learning for grouped data is a relevant topic, for which the submission proposes a simple and effective method. \n\n-The authors present a practical use case in section 4.3, showing how the method can help tackle the domain gap problem. Also, experiments presented in Figure 3 uncovers some very intriguing properties that might be worth exploring further.\n\n-The paper is well organized, formalism is clear and related work is informative.\n\n\nWeaknesses:\n\n-ACC, as noted by the authors, is very similar to the previously introduced Temporal Cycle Consistency. The main contribution of this work is to provide empirical results when applied to a more general context.\n\n-The paper lacks crucial discussions about why ACC allows learning a good alignment between sets in the general context. The choice of ACC seems arbitrary, as I find unclear how ACC relates to latent spaces alignment.\n\n-While the experimental results are intriguing, they aren't very convincing in terms of the usefulness of the proposed method. Results in section 4.3 are obtained using a 2-dimensional latent space, which is useful for visualization but is a very odd choice for practical uses. Section 4.5 presents an interesting application but the fact that the authors had to use a complex pipeline for these relatively simple images raises concerns about how well the method can generalize to other problems or scale to more difficult images.\n\n\n################################################\n\nRating motivation:\n\nThe paper advocates for the use of ACC for representation learning but doesn't provide an explanation about why the method should work. This makes it difficult to assess how much of the results can be attributed to the use of ACC and how much is due to other inductive biases. Also, while the car experiments are interesting, the results by themselves aren't convincing enough.\n\nI might change my evaluation if this point is clarified (either via discussion or by additional control experiments).\n\n\n################################################\n\nQuestions and minor remarks:\n\n-Can the authors comment on the importance of the size of the embeddings? Does the capacity bottleneck explain that geometric features are less prominent in figure 3c? What happens when using larger embedding?\n\n-Can the author confirm that phi:A->L and phi:B->M are implemented as two different convolutional networks? The paper is a little ambiguous.\n\n-The method is simple but reproducing would probably need some additional implementation details (architectures, optimization method, hyper-parameters...)\n\n-While the active/inactive features formalism is clear, it still might help to illustrate with examples how some classical tasks (for instance in domain transfer) fit in.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "gtxoHhtx0a-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applies a weakly-supervised learning approach to identify factors of object postures in an image dataset. The core idea is to introduce two sets of images. The first set is the reference data set with grouped objects of different active/inactive posture constraints. This set is used to provide weak supervision information in posture identification. The second set is the probe set. It does not necessarily require posture grouping of objects. Affinity Cycle Consistency loss is set up to automatically map objects of similar active postures between the two image sets (objects of similar postures are supposed to be the nearest neighbors in the learned embedding space). The experimental study verifies the validity of the proposed factor isolation algorithm. \n\nGenerally this paper is well written and clearly explains the motivation/problem definition. However, we have the following concerns on the innovative contribution of this work.\n\nThe innovation of this paper is very limited. The core technology applied in this work was originally proposed by Dwibedi et al in the paper \"Temporal Cycle-Consistency Learning\".  As cited in Section.1 of this paper, this work employs directly the Cycle-Consistency learning mechanism (while in a different application scenario). The only difference is: this work is built based on affinity relation between pairs of images with similar postures, while the original idea was applied for temporal sequence alignment in video processing. Not significant algorithmic innovation is introduced, compared to the previous work.  It is clearly below the threshold for a high-quality venue like ICLR. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "THL-OaEF_kZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses affinity cycle consistency to isolate factors of variation with only weak supervision on set membership. It has extensive experiments with both synthetic and real data.\n \nThe strength is that the problem setting is reasonable and important. The algorithm is sounding, and the evaluation is valid. The weakness is that the experiments and discussions do not cover enough cases for extracting and suppressing factors.\n \nI recommend that this paper is marginally above the border.\n \nThe positive reasons are the following. Isolating factors is a difficult and important problem, so the progress is good. Using weak supervision is practically convenient. Also, the paper shows that it helps a synthetic-to-real transfer problem. The following are some concerns.\n1. Extracting all active factors may need more samples in set A.\nIn the experiment of Figure 3c, second from the top, the learned representation does not isolate active geometric factors, indicating it does not include the information. This means set A does not contain enough variety of elements so that the object and floor hues are enough to distinguish the elements. To isolate all the active factors, set A should contain more elements than used in the experiment. It would be more helpful to find how many elements are necessary from theoretical and / or empirical perspectives.\n2. Suppression of complicated factors.\nThe paper does not tell why the inactive factors can be suppressed for unseen values, especially when it is inactive in both set A and B. When inactive, a factor takes a fixed value. So when it is inactive both in A and B, the factor has at most two values in training. However, in the test, it is suppressed for all (or most) possible values. The generalization capability from two values to many values requires more explanations, especially when the factor is complicated (e.g. non-linear) to extract. From experiment perspective, Figure 3c covers the cases of inactive ‘easy’ ’hue factors. It would be helpful to see what happens when the three hue factors and a geometric factor (e.g. orientation) are all inactive.\n \nAdditional feedback:\nThe last section is named “Discussion”, but it looks like “Conclusion” from its contents and position.\nThe paper does not provide code to reproduce the results.\nSome figures and letters on them are very small.\nGrammars:\n“a affinity cycle consistency loss”\n“for a each generative factor”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NmLlHwc88Hh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach to isolate factors of variation using weak supervision in the form of group labels. The proposed method Affinity Cycle Consistency (ACC)  claims to work with these group labels, which are weaker than the more common, one factor per group type labeling.  An important aspect of this approach is that it does not attempt to disentangle the factors of variation, but only capture (or isolate) them in the latent space. \n\nFollowing are the strengths of the paper: \n+ It uses a very simple strategy of combining soft nearest neighbors with cycle consistency in the latent space to achieve the ability of isolating factors of variation. The training of the network while imposing cycle consistency is done by simply minimizing the cross-entropy to predict the nearest neigbhor of an input point in the embedding space.  \n+ Some of the empirical results are interesting, as the paper reports increased mutual information (MI) between the embeddings and the factors of variation of interest. \n\nWhile I appreciate the simplicity of the approach, there are some important concerns which the paper fails to address adequately. \n- The analysis of empirical results is missing, which raises many questions. \n - Experiments of Fig. 3 indicate that with one inactive factor, two of the remining five active factors are isolated, and with two inactive factors, one of the remaining four active ones is isolated. This behavior is not clear as to why it happens? Why are the other factors not isolated? What if we have other factors (the not so easy ones like scale, shape or pose) inactive, will still the background objects' factors will be isolated? This behavior should ideally be explained through further analysis experiments. Similarly for other datasets. \n - It is not clear how the temperature value (as used in Definition 2) is set. How sensitive is the performance to this parameter. \n - How is the dimensionality of the embedding space determined? How does it effect the isolation / disentangling performance? (Jha et al. 2018) seem to show that the latent space dimensionality impact disentangling performance. Could this be a reason for the other factors not being isolated in Fig. 3? The embedding dimensionality is only 2. \n\n- Comparisons with group level supervision work \n - (Bouchacourt et al., 2018) used group level supervision for disentanglement, and would have been more appropriate for comparisons than e.g., (Jha et al. 2018). It is not clear why no comparisons were made with this work. \n\n- There are other mistakes, possibly typographical in nature. \n - From Definition 2, it appears that the soft nearest neigbhor of l_i has larger \\alpha_j for points m_j that are farther away from l_i. \n - Para before Definition 1, defines B, such that |B|=m, but it seems that it is meant to be n. \n\nMy recommendation is to reject this paper, primarily because of the lack of analysis and ablative experiments, as well as comparisons to the related approach of (Bouchacourt et al. 2018). It is unclear why certain factors get isolated while others do not, based on the active and inactive sets of factors. Well-designed analysis experiments will perhaps bring about some additional insights. \n\nI believe the paper will make a far more compelling case if there are analysis experiments presenting the strengths of the approach that provides insights into why certain factors are easily isolated and others are not. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to employ affinity cycle consistency(ACC) for extracting active (or shared) factors of variation across groups. Experiments shows how ACC works in various scenarios.

Pros:
- The problem is important and relevant.
- The paper is well written.
- The proposed method is simple and effective.

Cons:
- The experimental section is weak:
 It lacks an ablation to validate the contribution of ACC and discussion on
  why the method works and the scalability of the proposed method to more complex cases.
- The novelty is limited because the proposed ACC is similar to previous work temporal cycle consistency(TCC).
- The paper missed some implementation details and could be difficult to reproduce without code
 provided.

Reviewers raised the concerns listed in Cons. The authors conducted additional experiments and added more discussions on the experimental results in the revised paper. The authors also explained that ACC is more general than TCC. However, the reviewers were not convinced by the rebuttal and kept their original ratings.

Due to the two main weaknesses -- limited novelty and weak experimental analysis, I recommend reject.
Decision: Reject