Title: INVARIANT-EQUIVARIANT REPRESENTATION LEARN-ING FOR MULTI-CLASS DATA

Abstract: Representations learnt through deep neural networks tend to be highly informative, but opaque in terms of what information they learn to encode. We introduce an approach to probabilistic modelling that learns to represent data with two separate deep representations: an invariant representation that encodes the information of the class from which the data belongs, and an equivariant representation that encodes the symmetry transformation defining the particular data point within the class manifold (equivariant in the sense that the representation varies naturally with symmetry transformations). This approach to representation learning is conceptually transparent, easy to implement, and in-principle generally applicable to any data comprised of discrete classes of continuous distributions (e.g. objects in images, topics in language, individuals in behavioural data). We demonstrate qualitatively compelling representation learning and competitive quantitative performance, in both supervised and semi-supervised settings, versus comparable modelling approaches in the literature with little fine tuning.

1. INTRODUCTION:

Representation learning (Bengio et al., 2013) is part of the foundation of deep learning; powerful deep neural network models appear to derive their performance from sequentially representing data in more-and-more refined structures, tailored to the training task.

However, representation learning has a broader impact than just model performance. Transferable representations are leveraged efficiently for new tasks (Mikolov et al., 2013) , representations are used for human interpretation of machine learning models (Mahendran & Vedaldi, 2015) , and meaningfully structured (disentangled) representations can be used for model control (e.g. semisupervised learning as in Kingma et al. (2014) , topic modelling as in Blei et al. (2003) ).

Consequently, it is often preferable to have interpretable data representations within a model, in the sense that the information contained in the representation is easily understood and the representation can be used to control the output of the model (e.g. to generate data of a given class or with a particular characteristic). Unfortunately, there is often a tension between optimal model performance and cleanly disentangled or controllable representations.

To overcome this, some practitioners have proposed modifying their model's objective functions by inserting parameters in front of particular terms (Bowman et al., 2016; Higgins et al., 2017) , while others have sought to modify the associated generative models (Mansbridge et al., 2018) . Further still, attempts have been made to build the symmetries of the data directly into the neural network architecture in order to force the learning of latent variables that transform meaningfully under those symmetries (Sabour et al., 2017) . The diversity and marginal success of these approaches point to the importance and difficulty of learning meaningful representations in deep generative modelling.

In this work we present an approach to probabilistic modelling of data comprised of a finite number of distinct classes, each described by a smooth manifold of instantiations of that class. For convenience, we call our approach EQUIVAE for Equivariant Variational Autoencoder. EQUIVAE is a probabilistic model with 2 latent variables: an invariant latent that represents the global class information, and an equivariant latent that smoothly interpolates between all of the members of that class. The EQUIVAE approach is general in that the symmetry group of the manifold must not be specified (as in for example Cohen & Welling (2014) ; Falorsi et al. (2018) ), and it can be used for any number of classes and any dimensionality of both underlying representations. The price that must be paid for this level of model control and flexibility is that some labelled data is needed in order to provide the concept of class invariance versus equivariance to the model.

The endeavor to model the content and the style of data separately is certainly not new to this work (Tenenbaum & Freeman, 2000) . Reed et al. (2014) and Radford et al. (2016) go further, disentangling the continuous sources of variation in their representations using a clamping technique that exposes specific latent components to a single source of variation in the data during training. In the same vein, other approaches have used penalty terms in the objective function that encourage the learning of disentangled representations (Cheung et al., 2014; Chen et al., 2016) . EQUIVAE does not require any modification to the training algorithm, nor additional penalty terms in the objective function in order to bifurcate the information stored in the two latent variables. This is due to the way in which multiple data points are used to reconstruct a single data point from the same-class manifold, which we consider the primary novel aspect of our approach. In particular, our invariant representation takes as input multiple data points that come from the same class, but are different from the data point to be reconstructed. This invariant representation thus directly learns to encode the information common to the overall class, but not the individual data point, simply due to the information flowing through it.

Of further note, we deliberately use a deterministic latent for the invariant representation, and a stochastic latent for the smooth equivariant representation (an idea also employed by Zhu et al. (2014) ). This choice is why we do not need to explicitly force the equivariant latent to not contain any class-level information: it is available and easier to access from the deterministic latent.

EQUIVAE is also comparable to Siddharth et al. (2017) , where the authors leverage labelled data explicitly in their generative model in order to force the VAE latent to learn the non-class information (Makhzani et al. (2016) do similarly using adversarial training). The primary difference between those works and ours is that EQUIVAE provides a non-trivial representation of the global information instead of simply using the integer-valued label. Furthermore, this invariant representation can be deterministically evaluated directly on unlabelled data. Practitioners can reuse this embedding on unlabelled data in downstream tasks, along with the equivariant encoder if needed. The invariant representation provides more information than a simple prediction of the class-label distribution.

The encoding procedure for the invariant representation in EQUIVAE is partially inspired by Eslami et al. (2018) , who use images from various, known coordinates in a scene in order to reconstruct a new image of that scene at new, known coordinates. In contrast, we do not have access to the exact coordinates of the class instance, which in our case corresponds to the unknown, non-trivial manifold structure of the class; we must infer these manifold coordinates in an unsupervised way. Garnelo et al. (2018a; b) similarly explore the simultaneous usage of multiple data points in generative modelling in order to better capture modelling uncertainty.

2. EQUIVARIANT VARIATIONAL AUTOENCODERS:

We consider a generative model for data comprised of a finite set of distinct classes, each of which occupies a smooth manifold of instantiations. For example, images of distinct objects where each object might be in any pose, or sentences describing distinct sets of topics. Such data should be described by a generative model with two latent variables, the first describing which of the objects the data belongs to, and the second describing the particular instantiation of the object (e.g. its pose).

In this way the object-identity latent variable r would be invariant under the transformations that cover the set of possible instantiations of the object, and the instantiation-specifc latent variable v should be equivariant under such transformations. Note that the class label y is itself an invariant representation of the class, however, we seek a higher-dimensional latent vector r that has the capacity to represent rich information relevant to the class of the data point, rather than just its label.

Denoting an individual data point as x n with associated class label y n , and the full set of class-y labeled data {x n |label(x n ) = y} as D y lab , we write such a generative model as:

p {x n , y n } N n=1 = N n=1 dv n dr n p θ (x n |r n , v n ) δ r n -r(D yn lab \ {x n }) p(v n ) p(y n ) (1)

where θ are the parameters of the generative model. We make explicit with a δ function the conditional dependency of p θ on the deterministically calculable representation r n of the global properties of class y n . The distribution p(y n ) is a categorical distribution with weights given by the relative frequency of each class and the prior distribution p(v n ) is taken to be a unit normal describing the set of smooth transformations that cover the class-y n manifold.

To guarantee that r n will learn an invariant representation of information common to the class-y n data, we use a technique inspired by Generative Query Networks (Eslami et al., 2018) . Instead of encoding the information of a single data point (x n , y n ) into r n , we provide samples from the whole class-y n manifold. That is, we compute the invariant latent as:

r D yn lab \ {x n } concise notation = r yn = E x∼D yn lab \{xn} f θinv (x) ≈ 1 m m i=1 f θinv (x i ) (2)

where θ inv are the parameters of this embedding. We explicitly exclude the data point at hand x n from this expectation value; in the infinite labelled data limit, the probability of sampling x n from D yn lab would vanish. We include the simplified notation r yn for subsequent mathematical clarity. This procedure invalidates the assumption that the data is generated i.i.d. conditioned on a set of model parameters, since r y is computed using a number of other data points x i with label y. For notational simplicity, we will suppress this fact, and consider the likelihood p(x, y) as if it were i.i.d. per data point. It is not difficult to augment the equations that follow to incorporate the full dependencies, but we find this to obfuscate the discussion (see Appendix B for full derivations). We ignore the bias introduced from the non-i.i.d. generation process; this could be avoided by holding out a dedicated labelled data set (Garnelo et al., 2018a; b) , but we find it empirically insignificant.

The primary purpose of our approach to the invariant representation used in Equation 2 is to provide exactly the information needed to learn a global-class embedding: namely, to learn what the elements of the class manifold have in common. However, our approach provides a secondary advantage. During training we will use values of m (see Equation 2) sampled uniformly between 1 and some small maximal value m max . The r y embedding will thus learn to work well for various values of m, including m = 1. Consequently, at inference time, any unlabelled data point x can be immediately embedded via f θinv (x). This is ideal for downstream usage; we will use this technique in Section 3.1 to competitively classify unlabelled test-set data using only f θinv (x).

In order to approximate the integral in Equation 1 over the equivariant latent, we use variational inference following the standard VAE approach (Kingma & Welling, 2014; Rezende et al., 2014) :

EQUATION

where φ cov are the parameters of the variational distribution over the equivariant latent. Note that v is inferred from r y (and x), not y, since r y is posited to be a multi-dimensional latent vector that represents the rich set of global properties of the class y, rather than just its label. As is shown empirically in Section 3, v only learns to store the intra-class variations, rather than the class-label information. This is because the class-label information is easier to access directly from the deterministic representation r y , rather than indirectly through the stochastic v. We choose to provide both r y and x to q φcov (v|r y , x) in order to provide the variational distribution with more flexibility.

We thus arrive at a lower bound on log p(x, y) given in Equation 1 following the standard arguments:

EQUATION

where the various model parameters are suppressed for clarity.

The intuition associated with the inference of r y from multiple same-class, but complementary data points, as well as the inference of v from r y and x is depicted heuristically in Figure 1 .

EQUIVAE is designed to learn a representation that stores global-class information, and as such, it can be used for semi-supervised learning. Thus, an objective function for unlabelled data must be specified to accompany the labelled-data objective function given in Equation 4.

We marginalise over the label in p(x, y) (Equation 1) when a data point is unlabelled. In order to perform variational inference in this case, we use a variational distribution of the form:

q(v, y|x) = q φcov (v|r y , x) q φy-post (y|x) (5)

Figure 1 : Heuristic depiction of class-y manifold. The invariant latent r y will encode globalmanifold information, whereas equivariant latent v will encode coordinates of x on the manifold.

where q φcov (v|r y , x) is the same distribution as is used in the labelled case, given in Equation 3.

The unlabelled setting requires an additional inference distribution to infer the label y, which is achieved with q φy-post (y|x), parametrised by φ y-post . Once y is inferred from q φy-post (y|x), r y can be deterministically calculated using Equation 2 from the labelled data set D y lab for class y, of which x is no longer a part. With r y and x, the equivariant latent v is inferred via q φcov (v|r y , x).

Using this variational inference procedure, we arrive at a lower bound for log p(x):

L unlab = E q(y|x) E q(v|ry,x) log p(x|r y , v) -D KL q(v|r y , x) p(v) -D KL q(y|x) p(y) (6)

where the model parameters are again suppressed for clarity.

We will compute the expectation over the discrete distribution q(y|x) in Equation 6exactly in order to avoid the problem of back propagating through discrete variables. However, this expectation could be calculated by sampling using standard techniques (Brooks et al., 2011; Jang et al., 2017; Maddison et al., 2017) .

Therefore, the evidence lower bound objective for semi-supervised learning becomes:

x, unlab. L lab (7) In order to ensure that q φy-post (y|x) does not collapse into the local minimum of predicting a single label for every x value, we add log q φy-post (y|x) to L lab . This is done in Kingma et al. (2014) and Siddharth et al. (2017) , however, we do not add any hyperparameter in front of this term unlike those works. We also do not add a hyperparameter up-weighting L lab overall, as is done in Siddharth et al. (2017) . The only hyperparameter tuning we perform is to choose latent dimensionality (either 8 or 16) and to choose m max , between 1 and which m (see Equation 2) varies uniformly during training.

3. RESULTS:

We carry out experiments on both the MNIST data set (LeCun et al., 1998) and the Street View House Numbers (SVHN) data set (Netzer et al., 2011) . These data sets are appropriately modelled with EQUIVAE, since digits from a particular class live on a smooth, a-priori-unknown manifold.

EQUIVAE requires some labelled data. Forcing the model to reconstruct x through a representation r y that only has access to other members of the y class is what forces r y to represent the common information of that class, rather than a representation of the particular instantiation x. Thus, the requirement of some labelled data is at the heart of EQUIVAE. Indeed, we trained several versions of the EQUIVAE generative model in the unsupervised setting, allowing r to receive x directly.

The results were as expected: the equivariant latent is completely unused, with the model unable to reconstruct the structure in each class, as it essentially becomes a deterministic autoencoder.

In Section 3.1, we study the invariant-equivariant properties of the representations learnt with EQUIVAE in the supervised setting, where we have access to the full set of labelled training data. The semi-supervised learning setting is discussed in Section 3.2. The details of the experimental setup used in this section are provided in Appendix A. The training curves on MNIST are shown on the right in Figure 2 . The equivariant latent is learning to represent non-trivial information from the data as evidenced by the KL between the equivariant variational distribution and its prior not vanishing at convergence.

However, when visualised in 2 dimensions using UMAP for dimensional reduction (McInnes & Healy, 2018) , the equivariant latent v appears not to distinguish between digit classes, as is seen in the uniformity of the class-coloured labels in the middle plot of Figure 2 . The apparent uniformity of v reflects two facts: the first is that, given that the generative model gets access to another latent containing the global class information, the equivariant latent does not need to distinguish between classes. The second is that the equivariant manifolds should be similar across all MNIST digits. Indeed, they all include rotations, stretches, stroke thickness, among smooth transformations.

Finally, on the left in Figure 2 , the invariant representation vectors r y are shown, dimensionally reduced to 2 dimensions using UMAP, and coloured according to the label of the class. These representations are well separated for each class. Each class has some spread in its invariant representations due to the fact that we choose relatively small numbers of complementary samples, m (see Equation 2). The model shown in Figure 2 had m randomly selected between 1 and 7 during training, with m = 5 used for visualisation. The outlier points in this plot are exaggerated by the dimensional reduction; we show this below in Figure 4 by considering a EQUIVAE with 2D latent.

The SVHN results are similar to Figure 2 , with slightly less uniformity in the equivariant latent.

All visualisations in this work, including those in Figure 2 , use data from the validation set (5,000 images for MNIST; 3,257 for SVHN). We reserve the test set (10,000 images for MNIST; 26,032 for SVHN) for computing the accuracy values provided. We did not look at this test set during training and hyperparameter tuning. In terms of hyperparameter tuning, we only tuned the number of epochs for training, the range of m values (i.e. m max = 7 for MNIST; m max = 10 for SVHN), and chose between 8 and 16 for the dimensionality of both latents (16 chosen for both data sets). We fixed the architecture at the outset to have ample capacity for this task, but did not vary it in our experiments.

In order to really see what information is stored in the equivariant and invariant latents, we consider them in the context of the generative model. We show reconstructed images in various latent-variable configurations in Figure 3 . To show samples from the equivariant prior p(v) we fix a single invariant representation r y for each class y by taking the mean r y over each class in the validation set. On the left in Figure 3 we show random samples from p(v) reconstructed along with r y for each class y ascending from 0 to 9 in each column. Two properties stand out from these samples: firstly, the samples are (almost) all from the correct class for MNIST (SVHN), showing that the invariant latent is representing the class information well. Secondly, there is appreciable variance within each class, showing that samples from the prior p(v) are able to represent the intra-class variations.

The middle plot in Figure 3 shows interpolations between actual digits of the same class (the top and bottom rows of each subfigure), with the invariant representation fixed throughout. These interpolations are smooth, as is expected from interpolations of a VAE latent, and cover the trajectory between the two images well. This again supports the argument that the equivariant representation v, as a stochastic latent variable, is appropriate for representing the smooth intra-class transformations.

To create the right-hand side of Figure 3 , a validation-set image for each digit i is encoded to create the set of latents {r i y=i , v i } 9 i=0 , from which we reconstruct images using the latent pairs (r i y=i , v j ) for i, j = 0, . . . , 9. Thus we see in each row a single digit and in each column a single style. It is most apparent in the SVHN results that the equivariant latent controls all stylistic aspects of the image, including the non-central digits, whereas the invariant latent controls only the central digit.

In Figure 4 we show an EQUIVAE trained with 2-dimensional invariant and equivariant latents on MNIST for clearer visualisation of the latent space. On the right, reconstructions are shown with fixed r y for each y = 3, 4, 5, 6 and with the identical set of evenly spaced v over a grid spanning from -2 to +2 in each coordinate (2 prior standard deviations). The stylistic variations appear to be similar for the same values of v across different digits, as was partially evidence on the right of Figure 3 . On the left of Figure 4 we see where images in the validation set are encoded, showing significant distance between clusters when dimensional reduction is not used. Finally, in the middle of Figure 4 we see the evenly-spaced reconstruction of the full invariant latent space. This shows that, though the invariant latent is not storing stylistic information, it does contain the relative similarity of the base version of each digit. This lends justification to our assertion that the invariant latent r y represents more information that just the label y. 12.30 ± 0.28

We have thus seen that the invariant representation r y in EQUIVAE learns to represent globalclass information and the equivariant representation v learns to represent local, smooth, intra-class information. This is what we expected from the theoretical considerations given in Section 2.

We now show quantitatively that the invariant representation r y learns the class information by showing that it alone can predict the class y as well as a dedicated classifier. In order to predict an unknown label, we employ a direct technique to compute the invariant representation r y from x.

We simply use f θinv (x) from Equation 2. We can then pass f θinv (x) into a neural classifier, or we can find the nearest cluster mean r y from the training set and assign class probabilities according to p(label(x) = y) ∝ exp(-||f θinv (x) -r y || 2 ). We find that classifying test-set images using this 0-parameter distance metric performs as well as using a neural classifier (2-layer dense dropout network with 128, 64 neurons per layer) with f θinv (x) as input. Note that using p(y|x) ∝ p(x, y) is roughly equivalent to our distance-based classifier, as p(y|x) will be maximal when r y (computed from the training data with label y) is most similar to f θinv (x). Thus, our distance-based technique is a more-direct approach to classification than using p(y|x).

Our results are shown in Table 1 , along with the error rate of a dedicated, end-to-end neural network classifier, identical in architecture to that of f θinv (x), with 2 dropout layers added. This benchmark classifier performs similarly on MNIST (slightly better on SVHN) to the simple classifier based on finding the nearest training-set cluster to f θinv (x). This is a strong result, as our classification algorithm based on f θinv (x) has no direct classification objective in its training, see Equation 4. Our uncertainty bands quoted in Table 1 use the standard error on the mean (standard deviation divided by √ N -1), with N = 5 trials.

Results from selected, relevant works from the literature are also shown in Table 1 . Kingma et al. (2014) do not provide error bars, and they only provide a fully supervised result for their most- 

3.2. SEMI-SUPERVISED LEARNING:

For semi-supervised learning, we maximise L semi given in Equation 7. Test-set classification error rates are presented in Table 2 for varying numbers of labelled data. We compare to a benchmark classifier with similar architecture to q φy-post (y|x) (see Equation 5) trained only on the labelled data, as well as to similar VAE-based semi-supervised work. The number of training epochs are chosen to be 20, 25, 30, and 35, for data set sizes 100, 600, 1000, and 3000, respectively, on MNIST, and 20 and 30 epochs for data set sizes 1000, and 3000, respectively, on SVHN. We use an 8D latent space and m max = 4 for MNIST, and an 16D latent space and m max = 10 for SVHN. Otherwise, no hyperparameter tuning is performed. Each of our experiments is run 5 times to get the mean and (standard) error on the estimate of the mean in Table 2 .

We find that EQUIVAE performs better than the benchmark classifier with the same architecture (plus two dropout layers appended) trained only on the labelled data, especially for small labelled data sets. Furthermore, EQUIVAE performs competitively (within error bars or better) relative to its most similar comparison, Siddharth et al. (2017) , which is a VAE-based probabilistic model that treats the labels and the style of the data separately. Given its relative simplicity, rapid convergence (20-35 epochs), and lack of hyperparameter tuning performed, we consider this to be an indication that EQUIVAE is an effective approach to jointly learning invariant and equivariant representations, including in the regime of limited labelled data.

4. CONCLUSIONS:

We have introduced a technique for jointly learning invariant and equivariant representations of data comprised of discrete classes of continuous values. The invariant representation encodes global information about the given class manifold which is ensured by the procedure of reconstructing a data point through complementary samples from the same class. The equivariant representation is a stochastic VAE latent that learns the smooth set of transformations that cover the instances of data on that class manifold. We showed that the invariant latents are so widely separated that a 99.18% accuracy can be achieved on MNIST (87.70% on SVHN) with a simple 0-parameter distance metric based on the invariant embedding. The equivariant latent learns to cover the manifold for each class of data with qualitatively excellent samples and interpolations for each class. Finally, we showed that semi-supervised learning based on such latent variable models is competitive with similar approaches in the literature with essentially no hyperparameter tuning. that transforms r y , the output of which is then passed to one more dense hidden layer for each µ and σ 2 separately. This is shown in the bottom shaded box of Figure 5 .

The generative model p θ (x|r y , v) is based on the DCGAN-style transposed convolutions (Radford et al., 2016) , and is assumed to be a Bernoulli distribution for MNIST (Gaussian distribution for SVHN) over the conditionally independent image pixels. Both the invariant representation r y and the equivariant representation v, are separately passed through a single-layer dense network before being concatenated and passed through another dense layer. This flat embedding that combines both representations is then transpose convolved to get the output image in a way the mirrors the 5-layer convolution network used to embed the representations in the first place. That is, we use (64, 128) hidden units in the first two dense layers, and then (64, 32, 16, 8, n colours ) filters in each transpose convolution layer, all with 5x5 kernals and stride 2, except the last layer, which is a stride-1 convolution layer (with padding to accommodate different image sizes).

In our semi-supervised experiments, we implement q φy-post (y|x) using the same (5-CNN, 1dense) encoding block to provide an initial embedding for x. This is then concatenated with stop grad(f θinv (x)) and passed to a 2-layer dense dropout network with (128, 64) units. The use of stop grad(f θinv (x)) is simply that f θinv (x) is learning a highly relevant, invariant representation of x that q φy-post (y|x) might as well get access to. However, we do not allow gradients to pass through this operation since f θinv (x) is meant to learn from the complementary data of known same-class members only.

As discussed in Section 2, the number of complementary samples m used to reconstruct r y (see Equation 2) is chosen randomly at each training step in order to ensure that r y is insensitive to m.

For our supervised experiments where labelled data are plentiful, m is randomly select between 1 and m max with m max = 7 for MNIST (m max = 10 for SVHN), whereas in the semi-supervised case m max = 4 for MNIST (m max = 10 for SVHN).

We perform standard, mild preprocessing on our data sets. MNIST is normalised so that each pixel value lies between 0 and 1. SVHN is normalised so that each pixel has zero mean and unit standard deviation over the entire dataset.

Finally, all activation functions that are not fixed by model outputs are taken to be rectified linear units. We use Adam (Kingma & Ba, 2015) for training with default settings, and choose a batch size of 32 at the beginning of training, which we double successively throughout training. unlab given in Equation 6, is a lower bound on the data log likelihood.

:



Figures and Tables:

log p(x) + (x,y), lab. log p(x, y) ≥ L semi = x, unlab.L unlab + (x,y), lab.

Figure 2: Validation-set results for EQUIVAE on MNIST. Invariant (left) and equivariant (middle) latent representations are shown reduced to 2D using UMAP. Learning curves are shown (right) with the ELBO broken down into the sum of the reconstruction and (negative) KL terms, as in Equation 4.

Figure 3: Generated images (left) sampled from the prior p(v) for each r y , (middle) reconstructed from equivariant interpolations between the embeddings of same-class digits with fixed r y , and (right) reconstructed from latent pairs (r i y , v j ) where (r i y , v i ) is an encoded image with y = i.

Figure 4: Latent variables for EQUIVAE with 2 dimensional latents. The invariant latent space (left), reconstructions from evenly-spaced variations of r y covering the full space at fixed v = 0 (middle), and reconstructions from 2-prior-standard-deviation variations of v at fixed r y (right) are shown.

Figure 5: Example of variational encoding of a particular MNIST digit x (from the 6 class) both in terms of its invariant representation (top) and its equivariant representation (bottom).

which gives log p {x n } Nunlab n=1 D lab = log Nunlab n=1E q φ y-post (yn|xn) E q φcov (vn|r(D yn lab ),xn)p θ x n |r(D yn lab ), v n p(v n ) p(y n ) q φcov v n r(D yn lab ), x n q φy-post (y n |x n ) q φ y-post (yn|xn) E q φcov (vn|r(D yn lab ),xn) log p θ x n |r(D yn lab ), v n p(v n ) p(y n ) q φcov v n r(D yn lab ), x n q φy-post (y n |x n ) = Nunlab n=1 E q φ y-post (yn|xn) E q φcov (vn|r(D yn lab ),xn) log p θ x n |r(D yn lab ), v n(15)-D KL q φcov (v n |r(D yn lab ), x n ) p(v n ) -D KL q φy-post (y n |x n ) p(y n )Thus, we have Equation6augmented with the notational decorations that were omitted in Section 2

Supervised error rates on MNIST (10,000 images) and SVHN (26,032 images) test sets.

Table content: <table><tr><td/><td>Technique</td><td>Error rate</td></tr><tr><td/><td>EQUIVAE Benchmark neural classifier</td><td>0.84 ± 0.03</td></tr><tr><td>MNIST</td><td>EQUIVAE (neural classifier using f θinv (x)) EQUIVAE (distance based on f θinv (x)) Stacked VAE (M1+M2) (Kingma et al., 2014)</td><td>0.82 ± 0.03 0.82 ± 0.05 0.96</td></tr><tr><td/><td>Adversarial Autoencoders (Makhzani et al., 2016)</td><td>0.85 ± 0.02</td></tr><tr><td>SVHN</td><td>EQUIVAE Benchmark neural classifier EQUIVAE (neural classifier using f θinv (x)) EQUIVAE (distance based on f θinv (x))</td><td>10.04 ± 0.14 11.97 ± 0.34</td></tr></table>

Semi-supervised test-set error rates for various labelled-data-set sizes. / pre-trained VAE M1+M2, but it appears as if their learnt representation is less accurate in its classification of unlabelled data.Makhzani et al. (2016) perform slightly worse than EQUIVAE, but within error bars.Makhzani et al. (2016) train for 100 times more epochs than we do, and use over 10 times more parameters in their model, although they use shallower dense networks. Note also thatKingma et al. (2014)  andMakhzani et al. (2016) train on a training set of 50,000 MNIST images, whereas we use 55,000. We are unable to compare toSiddharth et al. (2017) as they do not provide fully supervised results on MNIST, and none of these comparable approaches provide fully supervised results on SVHN.

Table content: <table><tr><td/><td colspan="2">Labels EQUIVAE</td><td colspan="3">Benchmark Siddharth et al. (2017) Kingma et al. (2014)</td></tr><tr><td>MNIST</td><td>100 600 1000</td><td colspan="2">8.90 ± 0.70 21.91 ± 0.66 3.99 ± 0.17 6.64 ± 0.35 3.34 ± 0.17 5.43 ± 0.31</td><td>9.71 ± 0.91 3.84 ± 0.86 2.88 ± 0.79</td><td>(M2) 11.97 ± 1.71 4.94 ± 0.13 3.60 ± 0.56</td></tr><tr><td/><td>3000</td><td>2.23 ± 0.14</td><td>2.96 ± 0.11</td><td>1.57 ± 0.93</td><td>3.92 ± 0.63</td></tr><tr><td>SVHN</td><td colspan="3">1000 37.95 ± 0.66 39.64 ± 1.47 3000 24.95 ± 0.57 25.50 ± 0.91</td><td>38.91 ± 1.06 29.07 ± 0.83</td><td>(M1+M2) 36.02 ± 0.10 -</td></tr><tr><td colspan="3">powerful, stacked</td><td/><td/></tr></table>

Peer review (iclr2019):
 This paper is well written, and the quality of the figures is good. In this paper, the authors propose an invariant-covariant idea, which should be dated back at least to the bilinear models. The general direction is important and should be pursued further. However, the literature is not well addressed. Eslami et al. 2018 have been cited, but some very important and related earlier works like: [1] Kulkarni et al. 2015, Deep Convolutional Inverse Graphics Network [2] Cheung et al. 2015, Discovering Hidden Factors of Variation in Deep Networks were not discussed at all. The authors should certainly make an effort to discuss the connections and new developments beyond these works. At the end of section 1, the authors argue that the covariant vector could be more general, but in fact, these earlier works can achieve further equivalence, which is much stronger than the proposed covariance. There is also an effort to compare this work to Sabour et al. 2017 and the general capsule idea. I would like to point out, the capsule concept is a much more fine-grained what & where separation rather than a coarse-grained class & pose separation in one shot. In a hierarchical representation, what & where can appear at any level as one class can consist of several parts each with a geometrical configuration space. So the comparison of this work to the generic capsule network is only superficial if the authors can not make the proposed architecture into a hierarchical separation. Besides different capsule network papers, I found another potentially useful reference on a fine-grained separation: [3]Goroshin et al., Learning to Linearize Under Uncertainty In the paper, it is argued several times that the latent vector r_y contains a rich set of global properties of class y, rather than just its label and the aim is that it can learn what the elements of the class manifold have in common. But this point is not supported well since we can always make a label and this latent vector r_y equivalent by a template. I think this point could be meaningful if we look at r_y*s for different y, where each of the dimension may have some semantic meaning. Additional interpretation is certainly needed. Under equation (3), "Note that v is inferred from r_y" should be "inferred from both r_y and x", which is pretty clear from the fig 5. Related to this, I could imagine some encoder can extract the *style* directly from x, but here both r_y and x are used. I couldn*t find any guarantee that v only contains the *style* information based on the architecture with even this additional complication, could the authors comment on this? Equation (5) is not really a marginalization and further equation (6) may not be a lower bound anymore. This is probably a relatively minor thing and a little extra care is probably enough. The numbers in table 2 seems a little outdated. To conclude, I like the general direction of separating the identity and configurations. The natural signals have hierarchical structures and the class manifold concept is not general enough to describe the regularities and provide a transparent representation. Rather, it*s a good starting point. If the authors could carefully address the related prior works and help us understand the unique and original contributions of this work, this paper could be considered for publication.