Representation Learning Through Latent Canonicalizations
Authors: Or Litany
Ari Morcos
Srinath Sridhar
Leonidas Guibas
Judy Hoffman
Venue:  (2020)
Keywords: representation learning
latent canonicalization
sim2real
few shot
disentanglement

Abstract: We seek to learn a representation on a large annotated data source that generalizes to a target domain using limited new supervision. Many prior approaches to this problem have focused on learning disentangled representations so that as individual factors vary in a new domain, only a portion of the representation need be updated. In this work, we seek the generalization power of disentangled representations, but relax the requirement of explicit latent disentanglement and instead encourage linearity of individual factors of variation by requiring them to be manipulable by learned linear transformations. We dub these transformations latent canonicalizers, as they aim to modify the value of a factor to a pre-determined (but arbitrary) canonical value (e.g., recoloring the image foreground to black). Assuming a source domain with access to meta-labels specifying the factors of variation within an image, we demonstrate experimentally that our method helps reduce the number of observations needed to generalize to a similar target domain when compared to a number of supervised baselines. 

Under review as a conference paper at ICLR 2020
REPRESENTATION LEARNING THROUGH LATENT CANONICALIZATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
We seek to learn representation large annotated data source that generalizes to a target domain using limited new supervision. Many prior approaches to this problem have focused on learning "disentangled" representations so that indi- vidual factors vary in a new domain, only a portion of the representation need be updated, In this work, we seek the generalization power of disentangled represen- tations, but relax the requirement of explicit latent disentanglement and instead encourage linearity of individual factors of variation by requiring them to be ma- nipulable by learned linear transformations. We dub these transformations latent canonicalizers, as they aim to modify the value of a factor to pre-determined (but arbitrary) canonical value (e.g., recoloring the image foreground to black). Assuming a source domain with access to meta-labels specifying the factors variation within an image, we demonstrate experimentally that method helps reduce the number of observations needed to generalize to a similar target domain when compared number of supervised baselines.
INTRODUCTION
Most state-of-the-art visual recognition models rely on supervised learning using large set of man- ually annotated data (Krizhevsky al., 2012, He al., 2015 He al., 2017, Redmon Farhadi 2018). As recognition task complexity increases, so does the number of potential real world vari- ations visual appearance and hence the size of the example set needed for sufficient test time generalization. Unfortunately. large labeled data sets are expensive to acquire (Deng et al. 2009 Zhou et al. 2017), and may even be infeasible for applications with evolving data distributions.
Often large portion of the variance within collection of data is due to task-agnostic factors of variation. For example, the appearance of a street scene will change substantially based on the time of day, weather pattern, and number of traffic lanes, regardless of whether present. Ideally, the ability to recognize cars and pedestrians would not require labeled examples of street scenes for all combinations of times of day. weather conditions, and geographic locations. Rather it should be sufficient to observe examples from each factor independently and generalize to unseen combinations. However, often the in-domain labeled data available may even linearly cover all factors of variation. This calls for methods that encourage such sample efficiency by focusing on the individual complexities of the factors of variation, opposed their product.
pedestrians are
Prior approaches to learning representations which isolate factors of variation in the data have typically regularized the representation itself, with the aim of learning "disentangled" representa- tions (Kulkarni et al., 20155 Chen et al. 2016 Higgins et al., 2017a Burgess et al. 2018 Kim & Mmnh]/2018 Alternatively, one might regularize the way the representation can be manipulated rather than the representational structure itself. Here, we take such an approach by introducing la- tent canonicalization, in which we constrain the representation such that individual factors can be clamped arbitrary, but fixed value ("canonicalized") by simple linear transformation of the representation. These canonicalizers can be applied independently or composed together to canoni- calize multiple factors.
We assume access to a large collection of source domain data with meta-labels specifying the factors of variation within image. This may be available from meta-data, attribute labels, or from hyper- parameters used for generation of simulated imagery. Latent canonicalizers are optimized by pixel loss over pairs of ground-truth canonicalized examples and reconstructions of images with various

---

Under review as a conference paper at ICLR 2020
factors of variation whose representation has been passed through the relevant latent canonicalizers. By requiring the ability for manipulation of the latent space according to factors of variation, latent canonicalization encourages our representation to linearize such factors.
We evaluate our approach on its ability to learn general representations after observing only a subset of all potential combinations of factors of variation. We first consider the simple dSprites dataset, introduced to study disentangled representations (Matthey et al.) 2017) and show qualitatively that we can effectively canonicalize individual factors of variation. We next consider the more realistic, though still tractable, task of digit recognition on street view house numbers (SVHN) (Netzer et al., 2011] with few labeled examples. Using a digit simulator, we train our representation with latent canonicalization along multiple axes of variation such as font type, background color, etc. and then use the factored representation to enable more efficient few-shot training of digit recognition in the real data. Our method substantially increased performance over standard supervised learning and fine-tuning for equivalent amounts of data, achieving digit recognition performance that was only at- tainable with 5x as much SVHN labeled training data for the best-performing baseline method. Our experiments offer promising evidence that encoding structure into the latent representation guided by known factors of variation within data enable more data efficient learning solutions.
2 RELATED WORK
2.1 DISENTANGLING
number of studies have sought to learn low-dimensional representations of the world, with many aiming to learn "disentangled" representations. In disentangled representations, single latent units independently represent semantically-meaningee
factors of variation the world and can lead to better performance some tasks (van Steenkiste et al. 2019). This problem has been most com- monly studied in an unsupervised setting. often by regularizing latent representations to stay close to isotropic Gaussian (Higgins et al.) 2017a|b] [Burgess al. [0018] Kim & Mnih] (0118). Other popular unsupervised approaches include maximizing the mutual information between the latents and the observations (Chen al. 2016) and adversarial approaches (Donahue et al., 2016 Radford jet al. 2015). When supervision on the sources variation is available, it is possible to use this in a weak way (Kulkarni al. 2015).
Many of these works have explicitly endeavored to learn semantically meaningful representations which both linearly independent and axis-aligned, such that individual latents correspond to individual factors of variation in the world. However, recent work has questioned the importance of axis- aliined representations, showing that many of these methods rely on implicit supervision and finding little correspondence between this strict definition of disentanglement and learning of downstream tasks (Rolinek et al., 2018 Locatello et al., 2019). Further, while axis-alignment is useful for human interpretability. for the purposes of decodability, any arbitrary rotation of these latents would be equally decodable so long as factors are linearly independent (Locatello et al.) 2019). In this work, we use explicit supervision in a simulated setting to encourage linear, but not necessarily axis-alls aligned representations.
2.2 SIM2REAL
The setting of near-unlimited simulated data with ground truth labels and scarce real data occurs often in computer vision and robotics. However, the domain gap between simulated and real data reduces generalization capacity. Many approaches have been proposed to overcome this difficulty which are broadly referred to as sim2real approaches. A simple approach to closing the sim2real gap is to train networks with combinations of real and synthetic data (Varol et al. 2017).
Transfer Learning and Few-shot Learning: Alternatively, synthetic data can be used for pre- training followed by fine-tuning on real data (Richter et af.) (0066)a a form of transfer learning- Often, the reason one uses simulated data is that there is a shortage of labeled real data. In this situation, one may make use of few-shot learning techniques which seek to prevent over-fitting to the few examples by using a metric loss between data tripets (Koch et al. 2015), comparing similarity between individual examples (Vinyals et al. 2016) or between a prototypical example per category and each instance (Snell et al. (017). A separate class of techniques use meta-learning to design a learning solution which is most amenable to learning from few examples (Finn et al. 2017). However, these approaches may be sensitive to the ratio of synthetic and real data.
2

---

Under review as a conference paper at ICLR 2020
Domain adaptation: With access large set of unlabeled real examples, domain adaptation techniques can be used to close the sim2real gap. One class of techniques focuses on matching
domains at the feature level (Ganin & Lempitsky 2013: Long et al. 2015 Tzeng al., 2017).
features aiming to learn domain-invariant features than can make models more transferrable (Zou et al. 2018
Li al. 2018a). In fact, image-to-image translation focuses the appearance gap by bridging the appearance gap in the image domain instead of feature space (Shrivastava et al.) 2017 Liu et al., 2017). Domain adaptation can also be used learn content gap in addition appearance gap (Kar et al., 2019). Additional structural constraints, such as cycle consistency, can further refine this image domain translation (Zhu al., 2017 Hoffman et al.] [0188] Mueller et al.) 20188) Finally,
image stylization methods can also be adapted for style transfer and sim2real adaptation (Li et al.
2018b).
Domain randomization: (DR) offers surprising alternative (Tobin et al. 2017 Prakash et al., 2018| by exploiting control of the synthetic data generation pipeline randomize sources of vari- ation. Random variations will likely be ignored by networks and thus result in invariance to those variations. A particularly interesting instantiation of DR was suggested in [Snddrmmyyer et al. (2018) for pose estimation. Pose is example of a factor of variation which could be ambiguous due to occlusions and symmetries. Instead of explicitly regressing for the pose angle, the authors pro- pose an implicit latent representation. This is achieved by augmennted-autoennoderr wwwoising-autoenncn
form of that addresses all nuisance factors of variation as noise. This idea can be seen a particular case of our method where all factors are being canonicalized at once instead of learning to canonicalize each individually. Another interesting example is the quotient space approach of (Mehr et al. 2018) removes pose information for 3D shape representation by max- pooling encoder features over sampling of object rotations. It, however, does not consider how to perform canonicalization as a linear transformation in latent space, nor how to compose different canonicalizers.
APPROACH
Our goal is to learn representations which capture the generative structure of a dataset by indepen- dently representing the underlying factors of variation present in the data. While many previous works have approached this problem by regularizing the representation itself (Kulkarni et al. 20155] Chen et al. 2016 Higgins al. 2017a b Burgess al. 2018: Kim & Mnih 2018), here we take different approach: rather than directly encourage the representation to be disentangled, we instead encourage the representation to be structured such that individual factors of variation can be manipu- lated by a simple linear transformation. In other words, we constrain the way that the representation can be manipulated rather than the structure of the representation isseff.
3.1 LATENT CANONICALIZATION
In our approach, we augment a standard convolutional denoising autoencoder (AE) with latent canonicalizers. A standard AE learns an encoder, Enc, which takes as input a given image, æ. and produces a corresponding latent vector, z. At the same time, the latent vector is used as input to a decoder, Dec, which produces an output image, å. Both the encoder and decoder are learned according to the following objective, Lae. which minimizes the difference between the original input image and the reconstructed output image:
= Enc(x; 0c)
f = Dec ( 0a);
(00 02; = min | =112.
0..0A
To encourage noise-rohustness, we augment the potential input images following previous work on denoising autoencoders (Vincent al. 2008 2010), noising each raw input image, x, by adding Gaussian noise, blur, and random crops and rotations, leading to our noised input image, ž.
In this work, we additionally constrain the structure of the learned latent space using set of latent canonicalization losses. We define a latent canonicalizer as a learned linear transformation, C, which operates the latent representation, z, in order to transform a given factor of variation (e.g.. color or scale) from its original value to an arbitrary, but fixed, canonical value. So that individual factors can be manipulated separately, we learn unique canonicalization matrices, C,. for each factor of
3

---

Under review as a conference paper at ICLR 2020
C y + a 11 x11
aate) camon
Digit prediction y
canon
Classiiicaioon loss
Bypass
Canonicalization loss
classifier
Bypass
I
FC
encoder
decoder
7
73
Raw input a
Noised input
FS
FC BBC
7
7
canonicalization matrices
Z
candnicaliad
Lo)
FS
FC mmx
Rot mam
BgC m
Sh mxm
FT mxm
Image prediction
Target
Font
Font color
Rotation
Background color
Shear
Font
Figure 1: Schematic representation of latent canonicalization: Colored paths correspond to different com- ponents of the loss (cyan: classification, green: bypass, red: canonicalization) Four possible canonicalizations (two individual and two pairs) are shown along with example simulated SVHN images and reconstructions.
variation, j E [1, K], present in the dataset. In order constrain the latent representation according to canonicalization along one factor, j. our method yields the following basic form:
(i) canon
Enc (z;02) -C,
A (i) canen
Dec( (eemonn 0s)
compare the images generated by canoni- calized latents, Zeanon. to ground truth images with the appropriate factors of variation set to their canonical values, Teanon Canonicalizers can also be composed together to canonicalize multiple
To supervise the learning of latent canonicalizers,
factors (e.g., Zeanon z C₁ Ck). During training, each image is passed through a random subset of both individual and pairs of canonicalizers. Given Q canonicalization paths for given image, I, the corresponding canonicalization loss for that single image (red in Figure] T is written as:
Leaaso
Σ |a
|
canon
J.
(q) canon
11
44)
Since many outputs are canonicalized, it is possible that the decoder will simply learn to only gen- erate the canonical value of a given factor of variation. To prevent this form of input- input-independent memorization, we also include a "bypass" loss which is equivalent the standard denoising auto- encoder formulation (green in Figure defined in Equation ( thus forcing information about each factor to captured in the latent vector, z.
Finally, we ensure that our representation not only allows for linear manipulation along factors of variation, but does so while capturing information necessary train a classifier to solve our end task. To this end, we add supervised cross-entropy loss, LCF, which optimizes our end task using available labeled data (cyan in Figure] T):
Ler = y log
(5)
Combining equations 2 and 5 with loss-scaling factors (a and (9) gives our final per-example loss formulation:
C = Ū + all 211 +
(a()) canon
x (9) cunon
(6)
In practice, two canonicalizers are chosen at random for each input batch, Cn and Cj. and the cor- responding latent representation is passed through {Ch, Cy, CnC,, C,C.) generating four unique canonicalized latents: { Zeanon 2canon
(h.) 2 A diagram of the method shown in Figure illustrating each canonicalization path, the bypass path, and the classification model. All descríp- tion thus far has focused on the single image loss for simplicity of presentation. The full model averages the per image loss over mini-batch before making gradient updates.
4

---

Under review as a conference paper at ICLR 2020
Latent canonicalizer constraints: Our approach relies upon constraining the way representa- tion can be manipulated. As a result, the specific choice of constraints should have a significant impact on the representations which are ultimately learned. Here, we limit ourselves only two constraints: the transformations must be linear and canonicalizers must be composable, at least in pairs. If we were to allow non-linear canonicalizers, there would be little incentive for the encoder to learn an easily manipulable embedding. This would only be exacerbated as the non-linear canonical- izer is made more powerful by e.g., additional depth. By requiring canonicalizers be composable, we encourage independence as each canonicalizer must be able to be applied without damaging any other. We explore some other potential constraints in Section 4.2.4
3.2 SIM2REAL EVALUATION
A main motivation of latent canonicalization is to leverage structure gleaned from large source of data with rich annotations to better adapt to downstream tasks. A natural such setting for per- formance evaluation is sim-tt-real. Specifically, (SVHN) dataset as our real domain. To simulate SVHN, we built a SVHN simulator in which we have full control over many factors of variation such as font, colors and geometric transformations (a detailed description of the simulator is given in Section 4.2.1)
make use of the Street View House Numbers
We first pre-train on the synthetic data with latent canonicalization. Following this step, we freeze the canonicalizers and investigate whether the learned representations can be leveraged as the input to linear classifier for few-shot learning on real examples, labeled only with the class of interest (e.g., no meta-labels for additional factors of variation). During this stage, the encoder is also refined.
Majority vote: Because latent canonicalization manipulates the latent representation, we can use canonicalization form of "latent augmentation." " In this setting, we can aggregate the predictions of the digit classifier across many canonicalization paths, each of which confers a single "vote." Critically, such an approach requires the ability cleanly manipulate the learned representation, and is therefore only possible for our proposed method. For more detailed exploration of the impact of majority vote, see Section 4.2.4
3.3 BASELINES
We compare our proposed latent canonicalization with several baselines. To ensure the most fair comparison possible, we aimed to fix as many hyperparameters as possible: we use the same back- bone architecture in all our network modules detailed description of our network is given in Section 3.1;; the same number of epochs at the pre-training stage (learning from synthetic data); and a carefully chosen number of epochs the refinement stage to fit well the method overfitting rate. For all latent canonicalization experiments we trained three models on the simulated data and then performed five refinements of each pre-trained model, for total of fifteen replicates. Results are reported as mean + standard deviation. For all baseline models, 15 independent replicates were trained.
Our simplest baseline, which is meant as lower bound, is simply a classifier trained only on the low- shot real data. Next is classifier trained on synthetic data and then refined on low-shot real data. The vanilla-AF is an autoencoder pre-trained on synthetic data, after which a linear classifier is trained on low-shot real data, allowing the encoder to be refined. The strongest baseline is an autocncoder augmented with digit classifier at both pre- pre-training (on simulated data) and at refinement. The loss weighting was chosen via a parameee search individually for each model.
4 EXPERIMENTS
4.1 LATENT CANONICALIZATION DSPRITES
Key to our method is the use of latent canonicalizers, which are learned linear transformations optimized to eliminate individual factors of variation. As a first test of the effectiveness of latent canonicalization, we evaluated our framework using the toy data set, dSprites (Matthey al. 2017), which was designed for the exploration and evaluation of disentanglement methods. Specifically, dSprites is dataset of images of white shapes a black background generated by five independent factors of variation (shape, scale, rotation, and x- and y-positions). Training our model (Figure I on dSprites, demonstrate the effect of applying different individual canonicalizers to various values
5

---

Under review as a conference paper at ICLR 2020
Individually canonicalized
Scale + rotation x position canonicalized
Input
Scale
Rotation
position
Y position
Figure 2: Canonicalization of dsprites images: Input dsprites images (left), reconstructions of inputs with factor canonicalized (middle), and rotation, scale, and x- s-position canonicalized (right). Each row demon- strates how images change single factor of variation altered.
of the input factors (Figure Zeeft and middle). We therefore also applied a set of three canonicalizers (scale, rotation, and x-position) sequentially as shown in Figure right. Encouragingly, we found that not only did individual canonicalizers effectively canonicalize their factor of interest, multiple canonicalizers can be applied in sequence. Furthermore. although models were trained with only pairs of canonicalizers, triplets of canonicalizers also performed well (Figure right).
4.2 LATENT CANONICALIZATION SVHN
To evaluate the impact of latent canonicalization in more realistic, though still controllable setting, we turned to the well-known Street View House Numbers (SVHN) dataset [Netzer et al. 2011). In Section 1.2.1 we discuss approach to designing simulator for SVHN, which we in Section 4.2.2 to pre-train our models and measure the impact of latent canonicalization on sim2real trans- fer, finding that latent canonicalization substantially improved performance relative our strongest baseline. In Section 4.2.3] we investigate the structure of the learned representations to measure the representational impact of latent canonicalization. Finally, in Section[4.2.4] discuss potential directions for improvement which, unfortunately, either harmed or left unchanged sim2real perfor- mance relative to our best- rest-performing models.
4.2.1 SIMULATING SVHN
To support our proposed training procedure, we require a comprehensive dataset with detailed meta- data regarding ground-truth factors of variation. While this possible for a natural dataset, such data can also be generated for visually realistic, but fairly simplistic datasets such as SVHN. To this end, we built procedural image generator that simulates the SVHN dataset by rendering images with digits on constant-colored background (see examples in Figure [l]. Apart from the digit class variation, we also simulate additional factors of variation: font color, background color, font size, font type, rotation, shear, fill color for newly created pixels, scale, number of digit instances, translation, Gaussian noise, and blur. detailed description of the simulator is provided Section supervised factors of variation, noise and blur
C Among these factors chose the first six as a joint noise model, and the rest as additional factors which enrich the variety the data without supervised canonicalization. Some of the resulting images can be seen in Figure ELL To enable reproducability across comparisons, and to minimize unaccounted for variability in the data, we generated fixed training set with 75,000 images along with targets for all possible canonicalization paths. We emphasize that this training set represents a small fraction (~ 0.2%) of the total number of possible combinations of factors. We used such small fraction of the total space demonstrate that latent canonicalization is feasible even if the factor space is only sparsely sampled. The simulator along with the generated train set will be made publicly available.
4.2.2 SIM2REAL SVHN TRANSFER USING LATENT CANONICALIZATION
We want to learn representations which enable models to generalize to novel data with consistent underlying structure. Moreover, the effectiveness of disentangling for acquisition of downstream tasks has recently been called into question (Locatello et al. 2019). We therefore evaluate the quality of our learned representations by measuring their ability to adapt to real examples. Specifically, we
Appendix for further simulator details

---

Under review as a conference paper at ICLR 2020
Model
shot
20 shot
50 shot
100 shot
1000 shot
Classifier trained on real only
20.86 + 2.03
36.08 + 2.49
72.75 + 1.99
82.24 + 0.78
92.84 + 0.39
Classifier trained on synth
76.50 +: 2.06
80.07 = 1.09
83,95 + 0.79
85.65 + 1.17
89.82 + 0.60
Vanilla AE
15.39 + 6.12
21.55 # 9.16
41.98 16.62
51.18 19.80
58.51 + 19.68
AE + Classifier
79.61
81.63 + 1.00
84.66 +: 0.76
85.86 +: 0.82
88.80 0.62
Ours
82.55 + 0.86
84.83 + 0.76
87.82 + 0.57
89.40 + 0.48
91:21 + 0:24
Ours + majority vote
83.41 + 1.23
85.41 + 0.88
88.17 + 0.53
89.58 0.57
91.34 0.34
sim2real transfer results Model performance the SVHN test set using low-shot labeled real examples for method and baselines. Table entries represent mean = std.
Table 1:
consider a few-shot setting in which we allow models pre-trained on simulated data access to a small refinement set of few annotated examples per class. We ran this experiment with per-class set sizes of 10, 20, 50, 100 and 1000. To measure sim2real transfer, we train a fresh linear classifier on the representation learned by the encoder mre-cannicaalization, refined as well. We report accuracy on the unseen SVHN test set. As measure of the pre-trained model, we include examples of reconstructions generated by canonicalized latents in Figure
while allowing the encoder to be
When small train-sets are used, re- sults may vary substantially depend- ing on the selected set. To account for this, we (a) use the same train set for all methods, and (b) ran each exper- iment 15 times: 3 different networks were trained on simulated data with different random seeds and 5 repli- cate refinements were performed per pre-trained network.
[Font font type)
4
4
4
4
4
4
4
4
4
|Rotation, Font type)
5
5
5
5
5
5
5
Font type, Background Color,
6
6
6
67
6
6
6
6
(Font color,
3
3
3
3
siee)
[Font color, Background Coler)
1
1
1
Table the sim2real SVHN re- sults for our method along with four baselines: classifier trained solely on low-shot real data, a classifier trained on a large dataset of synthetic data and refined on low-shot real data, linear classifier trained on the bottleneck of a vanilla de- noising autoencoder trained on synthetic data, and a linear classifier trained on an autoencoder pre- trained with digit classifier on synthetic data. For all settings with fewer than one thousand exam- ples per class, we found that our model outperformed the best competing baseline by ~ 3 4%. We further improved our model's 's performance by taking a "majority vote" approach to inference type classification, in which we pass the representation through multiple latent canonicalizers in parallel to generate multiple votes as discussed in Section 3.2 Consistently, we found that majority vote boosted performance, by up to ~ 0.9%. with the largest gains coming for the lowest-shot settings.
50
Ul
e
Figure Example targets and reconstructions of canonicalized simulated SVHN images.
To contextualize the importance of this improvement, one see that to match reported per- formance on 20 shot with the best baseline of an AE + Classifier, a 5 times larger train set of 100 is required. This demonstrates the potential of proposed method in better utilizing access to meta-labels for better adaptation to real data.
4.2.3 ANALYSIS OF REPRESENTATIONS
Linear decodability of factors of variation from representations: While latent canonicaliza- tion encourages representations to be linearly manipulable, it does not explicitly encourage linear decodability. However, since our canonicalizers are constrained to be linear, we conjecture that la- tent canonicalization may also encourage linear decodability. To test this quantitatively, we trained linear classifiers on the pre-trained, frozen encoder for each factor of variation. We ran this experi- ment separately for each factor and compared linear decodability to our best baseline, AE+CIs. For continuous factors of variation, we binned target values, converting the problem into a multi-class classification task. Importantly, the number of bins differs across factors, so chance performance is different per factor. For background color, font color, and rotation angle, the canonicalized repre- sentation noticeably more linear than the baseline (shown here by higher accuracy on held-out
7

---

Under review as a conference paper at ICLR 2020
Rotation
5
5
5.5
75
35
35
2
5
535
15
5
150 5
5 5
5
Font size
5
5
5
5
5
8595
5
5
5
5 5
5
5
Figure 5: Visualization of the linear properties of the representation learned by the canonicalizers, Each row shows the first principal component of =(3] = for a source of variation. A clear pattern is visible for rotation (left to right tilted), and font size (small to big). 20 normally distributed samples from a batch of 1000
shown above. See appendix for visualization of the principal components of the bypassed latent.
test set), whereas font type showed a smaller improvement and font size and shear showed no im- provement in linear decodability (Figure One possible explanation for the discrepancy across factors is that font color, background color and, rotation are the most visually salient factors with the largest range of variability. We leave more thorough analysis of the precise features which make factors linearly decodable to future work.
Visualizing the impact of canonicalization: In the previous experiment, we quantified the linear decoding performance of our represen- tations; here, we attempt to visualize the ex- tent to which they linear. If these repre- sentations were indeed linear, we would expect them be easily decomposable using princi- pal component analysis (PCA), the components of which we can visualize. However, the latent codes from each of the canonicalizers removes the effect of source of variation while keep- ing the others. We therefore compute the prin- cipal components (PCs) of Zcanon z, i.c., the difference between canonicalized latent and the rreccnnnncallze latent, such that PCs now represent the removed factor of variation. In Figure 5 we show sorted images along the first PC, showing a clear linear sorting of rotation, and font size. This visually demonstrates that approach is able to extract latents that have strong linearity.
90
Ours AE + Classifier
Linear decoder
accuracy
60
Background Font Rotation Shear color color
Font
Font size
type
Figure 4: Linear decodability factors of varia- tion. Performance of a linear decoder trained on the frozen, pre-canonicalized representation, z for each of the six factors of variation. Each factor was divided into different number of bins for multi-class classification, such that chance is 1.6% for background and font color, 33.3% for rotation, 10% for shear, and 16.7% for font type and size. Error bars represent mean + std across
three pre-trained networks.
4.2.4 DISCUSSION OF ALTERNATIVE DESIGN DECISIONS
Latent canonicalization opens up many additional avenues for modification to potentially produce better representations and, consequently, better sim2real performance. In the previous section, we showed how incorporating majority vote further increased performance. Here, we discuss several other modifications we explored, which resulted in no change or decrease in sim2real performance.
Idempotency reconstruction loss: To encourage composability of canonicalizers, we trained them in identical pairs with alternating orders for consistency (e.g., z C,C, and z C2C,). We also tried encouraging idempotency, such that the same canonicalizer can be repeatedly applied with- out changing the reconstruction (e.g., z C2C2). We found that applying this loss actually harmed performance, reducing pre-majon pre-majority vote classification accuracy by ~ 0.5% (Table A1 second row).
Classifier location during pre-training: In our model, the classifier is placed at the output of the encoder prior to canonicalization, z. However, one might imagine that latent canonicalization could serve form of data augmentation. such that placing the classifier after the canonicalization step, Zeanon Canon, might increase performance. In contrast, we found that placing the classifier after the canonicalization step harmed performance, reducing maj-matoyyy vote classification accuracy by ~ 3% (Table Al third row). Interestingly, however, we found that the majority vote method, which can also be viewed as form of data augmentation, did in fact increase performance.
8

---

Under review as a conference paper at ICLR 2020
Latent consistency and idempotency loss: The impact of latent canonicalization supervised at the image level, by comparing the reconstruction to a target image. However, we could also use self-supervised loss at the latent level, by enforcing consistency (i.e., minl| C,C₂ and idempotency (i.e., min]| CiC1 2 Ci l2). To account for the scale of this latent loss, which was much larger than the other loss components, to maximize performance (Ic-7). Even with an appropriately scaled loss, however, the latent loss either had little impact harmed sim2real performance (Table Al fourth row).
C2C, l|2)
used very small scale factor for the latent loss
Alternative majority votes: We found that a simple majority vote containing the pre- canonicalizzd and individually canonicalized representations and 6 votes, respectively) increased performance by ~ 0.5%. To further augment the vote set, we also tried adding votes via idempo- tency (6 additional votes) and pairs (30 additional votes). We found that neither of these additions further improved performance over the simplest majority vote approach (Table A2).
5 CONCLUSION
We have introduced the notion of latent canonicalization, in which we train models to manipulate latent representations through constrained transformations which set individual factors of variation to fixed values ("canonicalizers". We demonstrate that latent canonicalization encourages repre- sentations which result in markedly better sim2real transfer on the SVHN dataset than comparable models, even when only a small sample of the possible combination space was used for training. Notably, latent canonicalized pre-trained models reached few-shot performance which required 5x as much data for comparable baselines. Analyzing the representations themselves, we found that the representation of factors of variation was linearized, measured by decodability and linear dimen- sionality reduction (PCA). Finally, we discuss number of alternative constraints which did not help performance. Here, we primarily analyzed SVHN, realistic, but relatively simple dataset. How- ever, the strong performance of latent canonicalization on SVHN is encouraging for larger-scale data with more complex factors of variation. Our results suggest the promise not only of latent canoni- calization, but, more broadly, methods which encourage representational structure by constraining representational transformations rather than particular structure itself.
REFERENCES
Christopher P Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint Stt1:18044035999 2018
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint arXiv:1605.09782 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference in Machine Learning (ICML), 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference in Machine Learning (ICML), 2015.
K. He, G. Gkioxari, P. Dollr, and R. Girshick. Mask r-cnn. 2017 IEEE International Conference Computer Vision (ICCV), pp. 2980-2988, Oct 2017. doi: 10. 10.11000 11.11////CV.221..32222
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770- 778, 2015.

---

Under review as a conference paper at ICLR 2020
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. vet:-vae; Learning basic visual concepts with a con- strained variational framework. In International Conference in Learning Representations (ICLR), 2017a.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1480-1490. JMLR. org, 2017b.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference Machine Learning (ICML), 2018.
Alexander B. Jung, Kentaro Wada, Jon Crall, Satoshi Tanaka, Jake Graving, Sarthak Yadav, Joy Banerjee, Gbor Vecsei, Adam Kraft, Jirka Borovec, Christian Vallentin, Semen Zhydenko, Kilian Pfeiffer, Ben Cook, Ismael Fernndez, Weng Chi- Chi-Hung, Abner Ayala-Acevedo, Raphael Meudec, Matias Laporte, et al. imgaug. https: http//gtttbcco. ale ju julejuuimmmne /mmguug 2019. [Online; accessed Sepppt1-
Amlan Kar, Aayush Prakash, Ming- Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-sim: Learning to generate synthetic datasets. In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proceedings of the 35th Interna- tional Conference on Machine Learning (ICML), 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In International Conference in Machine Learning (ICML), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097- 1105. Curran Associates, Inc., 2012. URL http:: nips cc/paper/
4824
with convoluinnllli -eetiiiiionnnnuuallll neural -eterrr.l-etwwssss.....
pdf
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in neural information processing systems, pp. 2539-2547, 2015.
Peilun Li, Xiaodan Liang, Daoyuan Jia, and Eric P Xing. Semantic-aware grad-gan for virtual-to- real urban scene adaption. arXiv preprint 111.0:180..01726, 2018a.
Yijun Li, Ming- Yu Liu, Xueting Li, Ming Haug- Yang, and Jan Kautz. A closed-fone solution to photorealistic image stylization. In Proceedings of the European Conference on Computer Vision (ECCV). pp.
2018b.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image- to-image translation networks. In Neural Information Processing Symposium (NeurIPS), pp. 700-708, 2017.
Francesco Locatello, Stefan Bauer, Mario Lui, Gunnar Rtsch, Sylvain Gelly, Bernhard Schlkopf, and Olivier Frederic Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, 2019. URL http//////1/ http///oaaaaellleee. 9a. html Best Paper Award.
httpp://proceedings www.::al...rr.r... vv97
Mingshen Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In International Conference in Machine Learning (ICML), 2015.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http:///hw...mmmmmmmmmmiiiiiaaaa 2017.
10

---

Under review as a conference paper at ICLR 2020
Eloi Mehr, Andre Lieutier, Fernando Sanchez Bermudez, Vincent Guitteny, Nicolas Thome, and andMatthicu Cord. Manifold learning in quotient spaces. In Computer Vision and Pattern Recogn- tion (CVPR), 2018.
Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt. Ganerated hands for real-time 3d hand tracking from monocular rgb. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 49-59, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http:://tfidii stanford. hoosemuuberssmmps2200 hww.oooooooosseeeeebs..... pdf
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary VeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Aayush Prakash, Shaad Boochoon, Mark Brophy, David Acuna, Eric Cameracci, Gavriel State, Omer Shapira, and Stan Birchfield. Structured domain randomization: Bridging the reality gap by context-aware synthetic data, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.064333 2015.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv, 2018.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision, pp. 102-118. Springer, 2016.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea direc- tions (by accident). arXiv preprint arXiv: (tf.::881.0667. 2018.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. Learning from simulated and unsupervised images through adversarial training. In Computer Vision and Puttern Recogntion (CVPR), 2017.
Jake Snell, Kevin Swerky, and Richard Zemel. Prototypical networks for few-shot learning, In International Conference in Learning Representations (ICLR), 2017.
Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, Manuel Brucker, and Rudolph Triehel. Implicit 3d orientation learning for 6d object detection from rgb images. In Proceed- ings of the European Conference on Computer Vision (ECCV), pp. 699-715, 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23-30. IEEE, 2017.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Adversarial discriminative domain adaptation. In Computer Vision and Puttern Recogntion (CVPR), 2017.
Sjoerd van Steenkiste. Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? arXiv preprint 1it.::9995.2000 2019.
Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 109-117,7 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference Machine Learning, ICML '08, pp. 1096-1103, New York, NY, USA, 2008. ACM. ISBN 978-1-68000 -0558 -05-4. doi: 10. 10.1145/1390156. 1390294. URL http: doi acm. org/ 10. Ne..114/13990 TTT.33900111111119053 390294

---

Under review as a conference paper at ICLR 2020
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of machine learning research, L1(Dec):3371-3400 2010.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Neural Information Processing Symposium (NeurIPS), 2016.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Domain adaptation for semantic segmen- tation via class-balanced self train-riiing.. arXiv preprint arXiv:/810.079/ 2018.
12

---

Under review as a conference paper at ICLR 2020
A
ALTERNATIVE DESIGN DECISIONS ADDITIONAL MATERIAL
Model
10 shot
20 shot
50 shot
100 shot
IK shot
(no maj vote)
82.55 0.86
84.83 0.76
87.82 + 0.57
89.40 0.48
91.21 + 0.24
Ours idem
81.77 1.23
84.08 0.98
87.08 = 0.64
88.96 0.51
90.74 0.35
Ours classifier after
79.69 1.22
81.14 1.00
84.21 = 0.74
86.42 0.54
88.62 0.24
Ours latent loss 1c-7
82.74 + 0.58
84.74 + 0.84
87.47 = 0.50
88.88 + 0.33
90.58 = 0.18
Table A1: Summary of model additions which not change or harmed performance. Table entrics represent mean + std.
Model
10 shot
20 shot
50 shot
100 shot
1K shot
Ours with maj majority vote idem
83.41 1.23
85.41 + 0.88
88. 0.53
89.58 + 0.57
91.34 0.34
83.44 1.21
85.44
88.20 0.55
0.58
91.35 0.36
Ours + majority vote all-pairs
83.26 1.12
85.26 + 0.80
88. 0.53
89.59 + 0.58
91.33 + 0.33
Table A2: Summary of alternative majority vote methods. Table entries represent mean + std.
B NETWORK IMPLEMENTATION DETAILS
All the models in this paper based on the modules: Encoder (Enc), Decoder (Dec), latent canon- icalizers (Can) and a linear classifier (Cls). Specifically the classifier baseline is a composition of Enc+Cls; the AE is a composition Enc+Dec. For AE+CIs we compose a linear classifier and su- pervised for both reconstruction and classification loss. For our proposed network, we also have 6 latent canonicalizers, used both individually and in pairs.
Loss weights: We performed a hyper-parameter sweep over the classification loss weight (E 10, 20, 50 and found 50 to be the strongest baseline. We also used 50.0 as the classification loss for our network.
Network architecture: The Encoder comprised of modules, each includes layers of 3x3 2D CNN with 64 latent dimensions followed by a batch norm and teaky-ReLU (parameter=0.1), A 2x2 max-pool and 50% dropout is done after each module and finally max-pool to get dimensional latent vector, z.
The Decoder is comprised of layers of 3x3 transposed convolutions with dimensionalities: (64, 64, 32) each followed by batch normalization and ReLU, Finally, last transposed convolution from 64 to dimensions and a ReLU.
Training hy hyperparameters: The networks were implemented using the PyTorch framework Paszke et al. (2017). Pre-training on simulated data and refinement on real data were done using the Adam optimizer Kingma & Ba (2114) with default parameters, with learning rates of le 3 and 1e - respectiverly. Pretraining was done for 400 epochs and refinement was done for roughly 50 epochs. The implementation
done
The classifier and the latent canonicalizers are simply single linear layers with 64 dimensions (cor- responding the dimensionality of the latent, 2).
13

---

Under review as a conference paper at ICLR 2020
C SIMULATOR DETAILS
To support our training requirements, we built a generator for SVHN-like images. Importantl, we do not aim to mimic the true SVHN dataset statistics. We make only very general simplistic assumptions such as 32x32 image size, that images contain centralized digit, etc. We build the simulator based on the imgaug library (Jung et al. 2019) which allows to easy control of the different factors of variation. The specific factors used to generate the images are given in the table together with their set of values. The first row of Table A3 shows the factors used for canonicalization. For each of these (except for noise, blur and crop which are canonicalized together to serve as the bypass), we generate a copy of the image with that factor set to its canonical value. The canonical values reported in row 3 of the table. They chosen arbitrarily but the same for the entire train set. Following the 73257 real SVHN train set samples (not used in this work) we generated a synthetic set of size 75000 total images. Some examples of generated images can be seen in Figure The set together with the simulator code will be made publicly available.
Al
2
5
9
7
2
6
6
35
2
9
1
6
8
1
9
7
6
5
3
24
3 4
55
23
7
1
9
3
53
8
2
2
3
8
65
Figure A1: Examples of simulated SVHN images.
Supervised factors
Rotation
Font color
Background color
Font scule
font type
shear
Noise Blur Crup
Range
[-15,16] 16]
(06).
{0...., 2553*
[1.0.166]
G
[-5,0] A/(0, 0.04), [0, 1.5), [0,0.]]
Canonical value
(200, 200, 200)
(100, 100, 100)
1.0
type 1
(0,0, 0)
Unupervised factors
Cval
Scale
Number of digits
Translation #Poise(1)
Range
0, .55
[0.95, 1.2]
I,...
Table A3: Simulator factors and their range values.
Domain gap between synthetic and real images: Having created quite generic set of digit images with naive assumptions, one may ask whether it is at all useful for the target domain. By taking intermediate checkpoints during training and measuring the classification error without fine- tuning on the real test set, we see in Figure A2 that indeed there is good correlation the train set error and the test set accuracy.
Sampling the space of compositions: Scene complexity grows exponentially with the number of factors. One goal this work is to show we can get good performance even when sampling a fraction of this space. To get rough estimate of that percentage we can bin evenjust just the supervised factors to [30, 64, 64, 6, 6. 10] (following the order of table A3) discrete values. This gives a space of xs 44M possibilities. Noticing the coarse binning of color-space sploc- and the fact haven't 't included the unsupervised factors, we conclude that our 75000 samples set size is at least order of magnitude smaller than the number of possible factor combinations.
14

---

Under review as a conference paper at ICLR 2020
77.5
75.0
Test accuracy
72.5
70.0
67.5
65.0
62.5
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Train loss
Figure A2: Train error synthetic images vs. Test accuracy on real SVHN images.
D
ANALYSIS OF REPRESENTATIONS ADDITIONAL MATERIAL
Bypass
65
5
5
5
5
5
5
53
5
5
35
5
450
5
55
5
B5
9525
5
Rotation
5
5
555
5
75
35
35
25
55
535 15
5
1505
(
5
5
Font size
5
5
5
5
85
552
5
5
5 5
5
5
5
Figure A3: Comparison between the representation learned by the canonicalizers and the bypassed representation. Here we show an extended version of Figure S also visualizing the bypassed latent code (top row). The first principal component of z does not show any obvious grouping or pattern.
15


Reviews:
[{"review_id": "Skg1tSgV5S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to relax the assumption of disentangled representation and encourage the model to learn linearly manipulable representations. \nThe paper assumes that the latent canonicalizers are predefined for each task and that it is possible to obtain the ground-truth image of different latent canonicalizations. I find these assumptions too strong for the task of learning disentangled representation. \nFirstly, most prior works such as beta-vae, info-gan do not assume that the factors / canonicalizers are known beforehand. In fact, this is a very difficult part of learning disentangled representation. Secondly, if it is possible to obtain the ground-truth image of different latent canonicalizations, you can simply train a network to predict the canonicalizations by simple supervised learning. Hence, these overly simplified and unrealistic assumptions make the task too trivial. \nThe proposed method is very simple and frames the problem basically as a supervised learning problem. Although experiments show that learning such representations are beneficial for low-shot setting of SVHN, it is not clear whether such improvement generalizes to more realistic datasets such as ImageNet. If the goal is to learn representation for low-shot setting, the method needs to be compared with other representation learning methods such as jigsaw[1], colorization[2] and rotation[3].\n\n[1] Noroozi, Mehdi, and Paolo Favaro. \"Unsupervised learning of visual representations by solving jigsaw puzzles.\" European Conference on Computer Vision. Springer, Cham, 2016.\n[2] Zhang, Richard, Phillip Isola, and Alexei A. Efros. \"Colorful image colorization.\" European conference on computer vision. Springer, Cham, 2016.\n[3] Gidaris, Spyros, Praveer Singh, and Nikos Komodakis. \"Unsupervised representation learning by predicting image rotations.\" arXiv preprint arXiv:1803.07728 (2018).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HylTsRJxqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for unsupervised learning  of data representations that can be manipulated to remove factors of variation via linear transformations. These transformations are called canonicalizations in the paper. The canonicalizations are trained such that images for arbitrary values of the corresponding factor of variation are transformed into images with a fixed, canonical, value for that factor. The paper proposes a model architecture based on a denoising autoencoder, where the canonicalizations are applied to the encoded representation. It also proposes a loss function and sampling scheme for training the model. The paper demonstrates the method on the dSprites dataset, showing that it can effectively learn linear canonicalizations, and that multiple of these canonicalizations can be applied to the same image representation. The paper goes on to test the method on a digit classification task, where the model is trained to learn a representation in a simulator for SVHN data where the transformations to be canonicalized can be controlled, and  used to train a classifier on unseen real data from the SVHN test set.\n\nI think this paper should be accepted as it proposes a novel idea, which does not seem too difficult to reproduce, describes a simulator for synthetic data for digit recognition, and proposes it as a benchmark for learning representations, and provides experimental results that help in better understanding the representation learned by the model.\n\n\nA couple things I thought were missing in the paper:\n\nDid you try applying the classification loss to both the encoded representation and the canonicalized representation at the same time?\n\nFor Figure 2, Did you try applying canonicalizations in different orders? Do they give the same results?\n\nInstead of trying to learn idempotency by gradient descent, you could try to parametrize the canonicalizations with a matrix X, such that C =  X (X^T X)^{-1} X^T. C will be idempotent (although restricted to be symmetric). There might be other constructions that are more efficient and less restrictive.\n\nI'm not sure I understand the PCA figures. Can you please explain how the first principal component was used to generate them?\n\n\nMinor comments:\n\n* \"data  tripets\" on page 2\n* Figure 5 should appear after Figure 4.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hyguk1hhKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n1. I had hard time to understand latent canonicalization. Do you mean that each latent variable is fixed to a value, such that this factor of variation is disabled? Are the canonicalizers pre-specified using meta-labels? Are they updated/learned during model training?   More explanation of canonicalization is needed. Perhaps an example in linear algebra is needed.\n\n2. The learning of the proposed model relies on meta-data description such that the learning is supervised. Can the method be applicable to situations where no meta-data and no class labels are available? \n  \n3. How can the proposed method be generalized to non-image data? The experiments were only done on simple image datasets. I am wondering this method can be applied to other complex datasets whose latent factors are unknown. \n\n4. I do not understand this: \"to fit well the method overfitting rate\" in Section 3.3.\n\nMinors:\n(1) than -> that \n(2) Eq. (3): is there a superscription \"(j)\" on z_canon in decoder?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to allow models to generalize more effectively through the use of latent linear transforms.

Overall, I think this method is interesting, but both R2 and R4 were concerned with the experimental evaluation being too simplistic, and the method not being applicable to areas where a good simulator is not available. This seems like a very valid concern to me, and given the high bar for acceptance to ICLR, I would suggest that the paper is not accepted at this time. I would encourage the authors to continue with follow-up experiments that better showcase the generality of the method, and re-submit a more polished draft to a conference in the near future.
Decision: Reject