CIGMO: Learning categorical invariant deep generative models from grouped data
Authors: Haruo Hosoya
Venue:  (2021)
Keywords: variational autoencoders
disentangling
mixture
clustering

Abstract: Images of general objects are often composed of three hidden factors: category (e.g., car or chair), shape (e.g., particular car form), and view (e.g., 3D orientation).   While there have been many disentangling models that can discover either a category or shape factor separately from a view factor, such models typically cannot capture the structure of general objects that the diversity of shapes is much larger across categories than within a category.   Here, we propose a novel generative model called CIGMO, which can learn to represent the category, shape, and view factors at once only with weak supervision.  Concretely, we develop mixture of disentangling deep generative models, where the mixture components correspond to object categories and each component model represents shape and view in a category-specific and mutually invariant manner.  We devise a learning method based on variational autoencoders that does not explicitly use label information but uses only grouping information that links together different views of the same object.  Using several datasets of 3D objects including ShapeNet, we demonstrate that our model often outperforms previous relevant models including state-of-the-art methods in invariant clustering and one-shot classification tasks, in a manner exposing the importance of categorical invariant representation.


Under review as a conference paper at ICLR 2021
CIGMO: LEARNING CATEGORICAL INVARIANT DEEP GENERATIVE MODELS FROM GROUPED DATA
Anonymous authors
Paper under double-blind review
ABSTRACT
Images of general objects are often composed of three hidden factors: category (e.g., car or chair), shape (e.g., particular car form), and view (e.g.. 3D orienta- tion). While there have been many disentangling models that can discover either a category or shape factor separately from a view factor, such models typically cannot capture the structure of general objects that the diversity of shapes is much larger across categories than within a category. Here, we propose a novel gen- erative model called CIGMO, which can learn to represent the category, shape, and view factors at once only with weak supervision. Concretely, we develop mixture of disentangling deep generative models, where the mixture components correspond to object categories and each component model represents shape and view in category-speciff and mutually invariant manner. We devise learning method based on variational autoencoders that does not explicitly use label infor- mation but uses only grouping information that links together different views of the same object. Using several datasets of 3D objects including ShapeNet, we demonstrate that our model often outperforms previous relevant models including state-of-the of-thofft methods in invariant clustering and one-shot classification tasks, in a manner exposing the importance of categorical invariant representation.
INTRODUCTION
In everyday life, we see objects in a great variety. Categories objects are numerous and their shape variations are tremendously rich; different views make an object look totally different (Figure A Recent neuroscientific studies have revealed how the primate brain organizes representation of com- plex objects in the higher visual cortex (Freiwald & Taa 2010 Srihasam et al. 2014 Bao et al.) 2020). According to these, it comprises multi-stream networks, each which is specialized to particular object category, encodes category-specife visual features, and undergoes multiple stages with increasing view invariance. These biological findings inspire us a new form of learning model that has multiple modules of category-specific invariant feature representations.
More specifically, our goal is, given an image dataset of general objects, to learn a generative model representing three latent factors: (1) category (e.g., cars, chairs), (2) shape (e.g., particular or chair types), and (3) view (e.g.. 3D orientation). A similar problem has been addressed by recent disentangling models that discover complex factors of input variations in way invariant to each other (Tenenbaum & Freemann 2000; Kingma et al.) [014]] Chen et al. 2016 Higgins et al., 2016 Bouchacourt et al.] [2018] [osoyy| (011)). However, although these models can effectively inter category or shape factor separately from view factor, these typically cannot capture the structure in general object images that the diversity of shapes is much larger across categories than within category.
In this study, we propose novel model called CIGMO (Categorical Invariant Generative MOdel), which can learn to represent all the three factors (category, shape, and view) at once only with weak supervision. Our model has the form of mixture of deep generative models, where the mixture components correspond to categories and each component model gives disentangled representation of shape and view for a particular category. We develop a learning algorithm based on variational autoencoders (VAE) method (Kingma & Welling 2014) that does not explicit labels, but uses only grouping information that links together different views of the same object (Bouchacourt et al. 2018 Chen et al. 2018 Hosoya 2019)

---

Under review as a conference paper at ICLR 2021
(A)
category
chair category
(B) CIGMO
(C)
category shape
views
view variation
data group
shape variation
Figure I: (A) Examples of general object images. These include two categories (car and chair) each with shape variations. addition, the object of each shape is shown in three different views. (B) The graphical model. Each instance Fą in data group is generated from a category shape 2, and a view yk- Round boxes are discrete variables and circles are continuous variables. (C) The inference flow. Each hidden variable is inferred from the set of incoming variables.
Using two image datasets of 3D objects (one derived from ShapeNet (Chang al., 2015)), demonstrate that CIGMO can solve multiple unconventional visual tasks on objects that are unseen during training. These include invariant clustering, i.e., clustering of objects regardless of the view, one-shot classification, i.e., object recognition given one example per class, and other various feature manipulations using the disentangled representation. Quantitative comparison indicates that our model often outperforms many existing approaches including state-of-ther of-ttr-rtt methods.
Our key contributions are (1) development of the new deep generative model CIGMO, together with the VAE-based weakly supervised learning algorithm, (2) experiments of the model on two datasets with quantitative comparisons that show performance advantages over several existing models, and (3) empirical exposition of the importance of separate representation of category, shape, and view factors in modeling general object images.
2 RELATED WORK
The present work is closely related to recently proposed disentangling models for discovering mutu- ally invariant factors of variation in the input. In one direction, some models have used unsupervised learning with certain constraints on the latent variable, though these seem to be effective only in lim- ited cases (Higgins et al., 2016 Chen et al. 20166.. Thus, more practical approaches have made use of explicit labels, such as semi-supervised learning for part of dataset (Kingma et al.) 22014 Sid- dharth al. 2017) or adversarial learning to promote disentanglement (Lample et al. 2017 Mathieu al. 2016 However, labels are often expensive.
To find out a good compromise, weaker forms of supervision have been investigated. One such direction is group-based learning, which assumes inputs with the same shape to be grouped together [Bouchacourt et al., 20188] Chen et al. 2018, Hosoya 2019). In particular, our study here techni- cally much influenced by Group-based VAE (GVAE) (Hosoyal 2019) in the algorithm construction (Section 3). However, these existing group-based methods are fundamentally limited in that the fac- tors that can be separated are two-a group-common factor (shape) and an instance-specific factor (view)-arn and there is no obvious way to extend it to more than two. Thus, our novelty here is to in- troduce mixture model comprising multiple GVAE models (each with shape and view variables) so that fitting the mixture model to grouped dataset can give rise to the third factor, categories, as mix- ture components.
Section 4 we examine the empirical merits this technique in several tasks. Note that grouping information object identity is often stable over time, cf. classical temporal coherence principle (Földiák) 1991). Indeed, some weakly supervised disentangling approaches have explicitly used such temporal struc- ture Yang et al. 2015).
most naturally be found in temporal data (like videos) since the
Some recent work has used deep nets for clustering of complex object images. The most typical approach is a simple combination deep generative model (e.g.. VAE) and a conventional clus- tering method (e.g., Gaussian mixture), although such approach seems to be limited in capturing
2

---

Under review as a conference paper at ICLR 2021
large object view variation (Jiang al. 2017p. A latest approach proposes feedforward approach that takes pairs of image data, similarly to ours, and maximizes the mutual information between the categorical posterior probability distributions for such paired image; this has shown remarkable clustering performance on natural images under various view variation (Ji et al. 2019). In Sec- tion 4 experimentally compare their method with ours. Note, however, that these methods specialized to clustering and throw away all information other than the category.
CIGMO: CATEGORICAL INVARIANT GENERATIVE MODEL.
3.1
MODEL
In our framework, we assume grouped dataset
I
{(x}] (m)
x
(m) K
(n) 2 E ER... . 1,
(1)
where each data point is group (tuple) of K data instances (e.g., images); we assume independence between groups but not instances within group. For data group (#I,...., xx). we consider three types of hidden variables: category E {l..... C). shape (eliding the superscript (n) for brevity), where the category and shape are common for the group while the views specific to each instance. We consider the following generative model (Figure[(B))
RM and views yı..... YK
p(c) = Te
(2)
p(z) I))(O,I)
(3)
p(yx) = =NZ(0, I)
(4)
p k yk. 2, c) =ND((c(y),)):)
(5)
for L,..., Candd 1, ..... K. Here, fe a decoder deep net defined for each category e and Te a category prior with Σe= Re In the generative process, first, the category e drawn from the categorical distribution (T....... ,c). while the shape and views yı: drawn from standard Gaussian priors. Then, each data instance F& is generated by the decoder deep net fe for category c applied to the group-common shape z and the instance-specific view yı: (added with Gaussian noise of unit variance). In other words, the decoder fa generates different data instances in a group from the same shape and different views. Having defined a mixture of deep generative models as above, we expect that, after fitting it to view-grouped object image dataset, object categories will arise as mixture components and category shapes and views will be represented in each component model.
3.2 LEARNING
We construct learning algorithm following the VAE approach (Kingma & Welling, 2014). As the most important step, we specify inference models to encode approximate posterior distributions (Figure [C)). First, estimate the posterior probability for category as follows:
(clar......
(0)) (xk)
K
Here, u is a classifier deep net that takes an individual instance x and outputs a probability distribu-
tion over the categories (EE_ (x))(x) 1), similarly to (Kingma et al. 2014). We then take the average over the instance-specific probability distributions and use it as the group-common distribu- tion. This is simple approach to make the instance-specife distributions converge to equal values, ie., u(x1) se u(x2). This is an adaptation of key technique of GVAE used for computing the group-common shape representation (Hosoyal 2019); see below.
Then, given the estimated category e, we infer each instance-specific view ya from the input æg as follows:
q 24, c) = Nz (gngea))
3

---

Under review as a conference paper at ICLR 2021
where Ie and re encoder deep nets that defined for each category to specify the mean and variance, respectively. To estimate shape 2, we compute the following:
k
K
g( 11, IK c) = NMs
Σ diag(se(<x))
(8)
K
K
Here, again, encoder deep nets he and 8. are defined for each category c. These compute the mean and variance, respectively, of the posterior distribution for the individual shape for each instance Za- Then, the group-common shape is obtained as the average over all the individual shapes. In this expected to converge
equal value in the course of training, i.c., he(21) z h.(22) (Hosoya 2019). Note that the way the view and shape inferred here mostly borrowed from Hosoya (2019).
way, again, the instance-specific shape representations
For training, we define the following variational lower bound of the marginal log likelihood for data point:
L(o; (10:,,....,.80 xK) Crecon LKL
(9)
where
Lrecon Eg(y 2. www. _x xx)
log p yy
(20. 2.
(10)
LKL - -Dx VK. 2, c/ 11, Øx llp(),) UK, c))
(11)
with the set Φ of all weight parameters in the classifier, encoder, and decoder deep nets. We compute reconstruction term Lreenn as follows:
Crecon g(c.1........................... KK Eggn UK
pog
(12)
K
q(c/ar,...... Øx log p( yv: c)
(13)
where we approximate the expectation using one sample z ((=/21.... æk. e) and yk ((yss c) for each k; but directly use the probability value g(c]æ11..... ....,,...,.... xx) for The KL term LKL is computed as follows:
CKL = -DLL((( xK (x)))p()))
q(clær..... σx)
ux Σ "x, DKI. .x. cl(()))
(14)
Finally, our training procedure is to maximize the lower bound for the entire dataset with respect to the weight parameters: arg max, N Σ 2(): ,...,,K (.)))
(m)). A diagrammatic outline of
the algorithm is given in Figure
4 EXPERIMENTS
We have applied the model described in Section to two image datasets: ShapeNet (general objects) and MultiPie (natural faces). Below, we outline the experimental set-up and show the results.
4. SHAPENET
For the first set of experiment, created dataset of multi-viewed object images derived from 3D models in ShapeNet database (Chang et al., 2013). We selected 10, out of 55, pre-defined object classes that each have relatively large number of object identities (car, chair, table, airplane, lamp,
4

---

Under review as a conference paper at ICLR 2021
category (probabilities)
shape
shape
views
views
average
X
average
X
average
h/s
9
9.6
O
(P'
loss
data group
Figure 2: A diagrammatic outline of CIGMO learning algorithm. The structure of the entire work- flow consists C modules (light-gray boxes) of group-based VAE corresponding to C categories and classifier network (right-most). An input is a group of data instances Ta (bottom). Given such input, each module с estimates an instance-specific view y/: from each data instance Ih using the encoders ge/Te. The module also estimates a group- shape by first estimating individual shapes for instances F& using the encoders he/s. and then by taking their average. Then, new data instances are generated by the decoder fa from the views and shape, and compared with the original input data for obtaining the reconstruction error (loss). This process is repeated for all modules. In parallel, the posterior probability for category is computed by the classifier u on each data instance Ia and then by averaging over the instances. Each computed probability is multiplied with the construction error for the corresponding module. Other probabilistic mechanisms (e.g., priors) are omitted here for brevity.
boat, box, display, truck, and vase); avoided heavily overlapping classes with many visually similar objects, e.g., chair, sofa, and bench. We then rendered each object in 30 views in a single lighting condition. We split the training and test sets, which consisted of 21888 and 6210 object identities, respectively. We also created subset versions with 2, 3, or 5 object classes. For training data, we formed groups of images of the same object random 3 views (K = 3). We used object identity labels (not class labels) for forming grouped data, but, after this step, we never used any label during the training. We converted all images to gray-scale to make the model concentrate on shape information rather than color information, as the latter easily becomes obvious clues for clustering (Ji et al. 2019). See Appendix A for more detail on the dataset.
To train CIGMO model, we used the following architecture. First, we set the number of categories in the model to the number of classes in the data (C 2, 5, or 10). We set the shape dimension M 100 and the view dimension T. = 3. Here, using very low view dimension was crucial since otherwise the view variable yv would take over all the information in the input and the shape variable would become degenerate (Hosoyal 2019). The classifier deep net consisted of three convolutional layers and two fully connected layers and ended with a Softmax layer. The shape and view encoder deep nets had a similar architecture, except that the last layer was linear for mean encoding (ge and ha) and ended with Softplus for variance encoding (re and sc). The decoder deep nets fo had an inverse architecture ending with Sigmoid. Since the model had so many deep nets, large part of the networks was shared to save the memory space. See Appendix B for details of the architecture. For simplicity, we fixed the category prior कe = 1/C. For optimization, we used Adam (Kingma & Ba 2015) with mini-batches of size 100.
In addition to CIGMO models, we trained a number of related models for comparison. To in- vestigate the effect of decoupled representation of category and shape, trained GVAE models (Hosoya) 2019), which can be obtained as a special case of CIGMO with single category (C= 1). To examine the effect of disentangling of shape and view, trained mixture of VAEs, again ob- tained as CIGMO with no grouping (K = 1; the shape and view variables are integrated). We also trained plain VAEs for basic comparison (C K 1). For part of evaluation below, since GVAE
5

---

Under review as a conference paper at ICLR 2021
(A)
CIGMO
B
Mix. of VAEs
GVAE + k-means C
(D)
IIC
category
category 2
category 3
Figure 3: Examples of invariant clustering from (A) CIGMO, (B) mixture of VAEs, (C) GVAE with k-means, (D) an IIC, in the case of categories. Random 24 images belonging to each estimated category are shown in a box. Note that the categories quite precisely correspond to the car, chair, and table image classes in (A), whereas such correspondence is less clear in the other models (B- (B-D; in particular, cars mixed with many other objects).
and VAE themselves have no clustering capability, we performed k-means on the shape variable of GVAE or the latent variable of VAE. In addition, we incorporated two completely different methods: Multi-Level VAE (MLVAE) (Bouchacourt et al. (2118), another group-based generative model, and Invariant Information Clustering (IIC) (ii et al. 2019), method specialized to invariant clustering. We tested two versions of IIC, with and without regularization using 5-times overclustering (Ji al, 2019). For each method, we trained 10 model instances from the scratch.
We evaluated the trained models using test data, which were ungrouped and contained objects of the same classes training data but of different identities. The evaluation involved three tasks: (1) invariant clustering, (2) one-shot classification, and (3) feature manipulation.
4.1.1 INVARIANT CLUSTERING
In this task, we simply inferred the most probable category from given image, è arg max, g(c/2)). Figure A) illustrates results from CIGMO model with categories, where each box shows random images belonging to each estimated category. This demonstrates a very precise clustering of objects achieved by the model, which is quite remarkable given the large view variation and category label used during training. Figure 3 B-BDD shows analogous examples for alternative models (mixture of VAEs, GVAE with k-means, and IIC). The result shows clear degrade of performance.

---

Under review as a conference paper at ICLR 2021
# of categories
2
10
chance level IIC
50.00
33.33
20.00
10.00
95.12
85.25 + 13.74
7.33
60.84 1.45
IIC (overcluster.)
93.93 2 6.44
$1.8744.5
59.73 + 1.49
VAE + k-means
71.98 + 7.04
66.41 5.69
+3.85
37.07 + 1.00
Mix. of VAEs
81.56 + 10.91
82.35
6.223
40.86 = 3.58
MLVAE k-means
90.56 = 6.86
$2.04
70.68 5.04
54.47 = 1.92
GVAE k-means
90.07 = 4.97
73.20 + 10.93
69.42 #344
52.55 + 2.74
CIGMO
98.41 = 0.50
94.83 + 6.06
89.36 4.53
68.53 + 4.24
Table 1: Invariant clustering accuracy (%). The mean and SD over 10 model instances are shown.
For comparison of the methods, we quantified the performance of invariant clustering in terms of classification accuracy. Here, we used the best category- -ttooass assignment computed by the Hun- garian algorithm (Munkres) 1957). Table [ summarizes the results. Generally, CIGMO outper- formed the other methods in all cases with a large margin. More specifically, first, CIGMO always performed better than mixture of VAEs, showing the importance of shape-view disentangling, Sec- ond, CIGMO was also always better than GVAE (and MLVAE) plus k-means, also showing the importance of category-shapp into a single latent variable, category information could not be clearly represented. These two points, taken together, emphasize the categorical and invariant nature of our model. Third, CIGMO gave performance superior to IIC, surpassing the state-of-the both with and without overclustering; in fact, we could not find consistent improvement by over- clustering in IIC, contrary to the claim by Ji et al. (2019).
decoupling. In other words, if shapes of all categories were packed
method for this task. This was the case
4.1.2 ONE-SHOT CLASSIFICATION
In this task, we split test data into gallery and probe, where gallery holds one image for each object and probe the rest, and then identify the object of each probe image. Note that our purpose here is not to infer the class but the object identity, unlike invariant clustering. Note also that, since the test objects are disjoint from the training objects, both gallery and probe images contain only unseen objects for the model. We used this task here since its performance can serve as a criterion for evaluating disentangled representations (Hosoyal 2019). The rationale is that, if shape code is perfectly invariant in view, then all images of the same object should be mapped to an identical point in the shape space.
Thus, compared overall accuracy of one-shot classification for CIGMO and other models. For this, we performed a nearest-neighbor method according to cosine distance in the shape space. Here, the shape space was defined depending the method. For GVAE (or MLVAE), the shape variable = fx(x:) directly defined the shape space. For CIGMO, since the shape representation depended on the category,
first constructed C M matrix Z such that Ze,. har)): for é and Ze = 0) otherwise, and then flattened the matrix to a vector. This gave
eategory-dependee shape vectors that could be directly compared and, in particular, those of different categories would give cosine distance 1, the maximum value. For VAE or mixture of VAEs, we used similar scheme except that we used the entire latent variable in place of shape variable.
Table 2 Summmaiize the results. Overall, CIGMO performed the best among the compared methods in all cases. particular, it outperformed, by far, mixture of VAEs, showing the success of shape in- formation disentangled from view. Further, CIGMO also performed significantly better than GVAE and MIVAF, which indicates that shapes can be represented more efficiently with category spe- cialization than without. These results, again, reveal the advantage of the categorical and invariant representations in modeling general object images. (Note also that the scores were remarkably high for up to 6210-way classification by shot.)
4.1.3 FEATURE MANIPULATION
CIGMO provides various ways of manipulating latent feature representations and generating new images. These include (1) swapping, to generate an image from the view of one image and the shape of another, (2) interpolation, generate image from the shape and view that linearly

---

Under review as a conference paper at ICLR 2021
of categories
2
chance level
0.05
0.03
0.02
0.02
VAE
1.29 0.03
1.81 0.04
3.09 - 0.05
2.97
Mix. of VAES
1.40
1.91 0.06
3.30 0.07
3.15 0.06
MLVAE
20.00 0.77
20.56 0.50
17.69
+ 0.28
15.68 +: 0.28
GVAE
19.71 + 0.87
21.44
17.85
0.28
15.93 0.18
CIGMO
22.10 + 0.99
23.95 = 0.52
21.66 + 0.79
19.49 + 0.71
Table 2: One-shot classification accuracy (%).
The mean and SD over 10 model instances are shown.
# of categories
2
3
10
MIVAE
54.80 1.37
53.59
0.57
47.37 0.85
42.99 + 0.63
GVAE
55.35 + 1.35
55.17
+ 0.80
48.23 + 0.58
43.85 + 0.29
CIGMO
55,33 + 1.47
57.69 - 0.94
51.00 0.89
46.30 = 0.93
MLVAE
0.21 + 0.02
0.24 + 0.02
0.62 + 0.04
0.53 + 0.09
GVAE
0.21 + 0.02
0.23 + 0.04
0.55 + 0.05
0,48 + 0.05
CIGMO
0.27 + 0.03
0.26 + 0.04
0.65 0.05
0.66 0.08
Table Quality of shape-view disentanglement, measured neural network classification accuracy (%) for object identity from the shape (top rows) or view variable (bottom rows); weighted average over categories. The mean and SD over 10 model instances are shown.
interpolates those of two images, and (3) random generation, to generate an image from shape and view randomly drawn from Gaussian distributions. Analogous manipulations have commonly been used in previous studies (Mathieu et al. 2016 Bouchacourt et al.) 2018 Hosoya 2019), but our cases conditioned category. Appendix gives examples of these feature manipulations, in which we can qualitatively confirm that our model performed these tasks as intended, e.g., reasonably clear aligament of views in and shapes in columns in swapping. Although generating sharp images was not the focus here, improvement in this direction could be done, e.g., by incorporating adversarial learning for regularization (Mathieu al. 2016).
Relevant to these, we quantitatively evaluated the quality of shape-view disentanglement in each category. Specifically, we measured how much information the shape or view variable contained on object identity and, for this purpose, we trained two-layer neural networks on either variable for classification (Mathieu et al., 2016 Bouchacourt al., 2018). A better disentangled representation was expected to give higher accuracy from the shape variable and a lower accuracy from the view variable. We performed this for each category using the belonging test images and took the average accuracy weighted by the number of those images. Table Jubleisummarizes the results. Overall, CIGMO gives fairly comparable quality to the existing disentangling methods.
4.2 MUL.TIPIE
For the second set of experiment, we used a dataset of multi-viewed face images derived from MultiPie dataset (Gross et al. 2010). We followed the same data preparation Hosoya (2019) except that we included the images in all lighting conditions and converted them to gray-scale. We split the training and test sets consisting of disjoint 795 and 126 identities, and grouped together the training images that have the same identity. hair/cloth style, and expression. For this dataset, there was no pre-defined class unlike ShapeNet, so we arbitrarily set the number of categories in CIGMO models. Otherwise, the training condition was the same as before.
Figure Iillustrates the results from CIGMO model on (A) invariant clustering and (B) category- wise swapping, where only 3 categories were shown as the other 2 categories were degenerate to which no input belonged. By inspection, the effective categories seemed to represent faces with long hair, round faces with short hair, and oval faces with short hair, respectively, in view-invariant manner (although this observation was not verified since the dataset lacked relevant labels). Other CIGMO model instances trained in the same way had 1 to 3 effective categories and seemed to have slightly different categorization strategies, but generally based on hair length, hair color, facial aspect ratio, beard, or skin color; could not find categorization by expression. We also measured
8

---

Under review as a conference paper at ICLR 2021
(A) Invariant clustering
(B) Swapping
shape
view
category
category
category
Figure 4: Results from CIGMO model trained for MultiPie dataset. (A) Invariant clustering. Random 25 images belonging to each estimated category are shown in a box. (B) Swapping. For each category, we generate matrix of images from two lists of sample images, where each generated image has the view of an image the left list and the shape of an image in the top list. Both together indicate that the model provides shape-view disentangled representation of specific categories corresponding to faces with long hair, round faces with short hair, and oval faces with short hair.
one-shot classification accuracy and shape-view disentangling quality. However, CIGMO overall gave a similar performance to GVAE or MIVAE, indicating a lesser importance of category-shap decoupling in this task for this dataset. This is understandable since, in a sense, all faces look alike and therefore, in what way faces are categorized, cross-category diversity would not be so large compared to within-category diversity.
CONCLUSION
In this paper, we have proposed CIGMO deep generative model that discover category, shape, and view factors from general object images. The model has the form of mixture of disen- tangling generative models and comes with a VAE-based algorithm that requires no explicit label but only view-grouping information. By application to two image datasets, we showed that our model can learn represent categories the mixture components and category-specific disentan- gled information of shape and view in each component model. We demonstrated that our model can outperform existing methods including state-of-the-art methods invariant clustering and one- shot classification tasks, emphasizing the importance of categorical invariant representation. Future investigation will include improvement on image generation quality, category degeneracy, and scal- ability, and application more realistic datasets. Lastly, CIGMO's biological relationship to the primate inferotemporal cortex would be interesting to pursue, as our present work was originally inspired

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pinglei Bao, Liang She, Mason McGill, and Doris Y. Tsao. A map of object space in primate inferotemporal cortex. Nature, (33(7814):103- jul 2020.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin, Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations. In AAAI Conf. Artif. Intell., jan 2018.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. 2015.
Mickaël Chen, Ludovic Denoyer, and Thierry Artières. Multi-Vier Data Generation Without View Supervision. In Int. Conf. Learn. Represent, nov 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. Adv. Neural Inf. Process. Syst. jan 2016.
P Földiák. Learning invariance from transformation sequences. Neural Comput., 3(2):194-200, jan 1991.
Winrich A Freiwald and Doris Y Tsao. Functional Compartmentalization and Viewpoint General- ization Within the Macaque Face- Pace-Processing System. Science, 330(6005):8455- 851, nov 2010.
Ralph Gross, lain Matthews, Jeff Cohn, Takeo Kanade, and Simon Baker. Multi-PIE. Proc. Int. Conf. Autom. Face Gesture Recognition, IEEE Int. Conf. Autom. Face Gesture Recognit. 28(5): 807-813, may 2010.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In Int. Conf. Learn. Represent. nov 2016.
Haruo Hosoya. Group-based learning of disentangled representations with generalizability for novel contents. Int. Jt. Conf. Artif. Intell., jan 2019.
Xu Ji, Andrea Vedaldi, and Joao Henriques. Invariant information clustering for unsupervised image classification and segmentation. Proc. IEEE Int. Conf. Comput. Vis., pp. 9864 09744,987, 2019.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised generative approach to clustering. Int. Jt. Conf. Artif. Intell., pp- 1965-1972, 2017.
D Kingma and J Ba. Adam: A method for stochastic optimization. In Int. Conf. Learn. Represent., jan 2015.
D P Kingma and M Welling. Auto-encoding variational bayes. In Int. Conf. Learn. Represent. jan 2014.
DP Kingma, S Mohamed, and D J Rezende. Semi-supervised learning with deep generative models. Adv. Neural Inf. Process. Syst., jan 2014.
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Mare' Aurelio Ranzato. Fader Networks:Manipulattn Images by Sliding Attributes. Adv. Neural Inf. Process. Syst., pp. 5967-5976, jan 2017.
Michaël Mathieu, Junbo Jake Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann LeCun. Disen- tangling factors of variation deep representation using adversarial training. Adv. Neural Inf. Process. Syst., jan 2016.
James Munkres. Algorithms for the Assignment and Transportation Problems. Soc. Ind. Appl. Math., 5(1):32-3, mar 1957.
10

---

Under review as a conference paper at ICLR 2021
N Siddharth, Brooks Paige, An--iillem de Meent, Alban Desmaison, Frank Wood, Noah D Goodman, Pushmeet Kohli, and Philip H S Torr. Learning Disentangled Representations with Semi-Supervised Deep Generative Models. Adv. Neural Inf. Process. Syst., jan 2017.
Krishna Srihasam, Justin L. Vincent, and Margaret S. Livingstone. Novel domain formation reveals proto-architecture in inferotemporal cortex. Nat. Neurosci., 17(12)1177-1178,, 2014.
J B Tenenbaum and W Freeman. Separating style and content with bilinear models. Neural
Comput., 12(6):1247-1283, jun 2000.
Jimei Yang, Scott Reed, Ming- Hsuan Yang, and Honglak Lee. Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis. Adv. Neural Inf. Process. Syst., jan 2015.
A DATASET DETAILS
Our object image dataset stemmed from a core subset of ShapeNet database of 3D object models (Chang et al. 2015)) used in SHREC2016 challenge The subset contained 55 object classes and did not include material data. Out of the 55 classes, we selected 10 classes: car, chair, table, airplane, lamp, boat, box, display, truck, and vase. Our criterion here was select classes with a large number of object identities but avoid including visually similar classes, e.g., chair, sofa, and bench. We rendered each object in 30 views consisting 15 azimuths (equally dividing 360%) and 2 elevations (09 and 22.5% downward) in single lighting condition: the images were gray-scale and had size 64 x 64 pixels. All the rendering used Blender software We divided the data into training and test following the split given in the original database. We also created versions of the dataset consisting of the first 2, 3, or object classes listed above.
B ARCHITECTURE DETAILS
In CIGMO model, the classifier deep net consisted of three convolutional layers each with 32, 64, and 128 filters (kernel 5 x 5; stride 2; padding 2). followed by two fully connected layers each with 500 intermediate units and C' output units. These layers were each intervened with Batch Normalization and ReLU nonlinearity, except that the last layer ended with Softmax. The shape and view encoder deep nets had similar architecture, except that the last layer was linear for encoding the mean (ge and he) or ended with Softplus for encoding the variance (r. and sc). The decoder deep nets fa had two fully connected layers (103 input units and 500 intermediate units) followed by three transposed convolutional layers each with 128, 64, and 32 filters (kernel 6 x 6; stride 2; padding 2). These layers were again intervened with Batch Normalization and ReLU nonlinearity. but the last layer was Sigmoid.
To save the memory space, the shape encoders shared the first four layers for all categories and for mean and variance. The view encoders shared the entire architecture for all categories, but with separate last layer for mean or variance specification. The decoders shared all but the first layer for all categories.
In the quantitative comparison, we obtained a mixture of VABs, a GVAE model, and a VAE model special case of CIGMO model. Namely, mixture of VAEs was a CIGMO with grouping (K = 1), GVAE was a CIGMO with a single category (C' = 1). and a VAE was a CIGMO with both constraints (K 1 and C 1). In the of grouping, since structure could differentiate the shape and view dimensions, we combined these into a single latent variable of 103 dimensions.
C ADDITIONAL RESULTS
We performed the following feature manipulation tasks introduced in Section 4.1.3
httpss://shpeenet.cs.ou cs. wt..saan.ts...........a.o wow...ddd/sreee.... https://ww.bler

---

Under review as a conference paper at ICLR 2021
Swapping For category and for images æ1 and æ2 belonging to c, obtain Tawap fc((11. where Y/1 = ge(21) and 22 = hx2)).
Interpolation a category e and images æ and aa belonging to e, obtain Zinterp - -f((ay) +
a yx 3z1 + - (2) where ge(x;) and (xx) with 0 S a, S 1 and
1,2.
Random generation For category c, obtain Trand z) with y (0, and (0, I).
Figure shows examples of these feature manipulations on a 3-category CIGMO model trained on ShapeNet. As we see, swapping gives clear alignment of views in rows and shapes in columns. Interpolation gives smooth changes of images from one image to another in both shape and view dimensions. Random generation gives new images most of which are recognizable as each category.
(A) Swapping
shape
view
(B)
Interpolation
shape
view
(C) Random generation
category
category 2
category 3
Figure 5: Examples of feature manipulation tasks from 3-category CIGMO model for ShapeNet dataset. (A) Swapping. For each category, we generate a matrix of images from two lists of sample images, where each generated image has the view of image in the left list and the shape of an image in the top list. (B) Interpolation. For each category, we generate a matrix images from two sample images (corresponding to the top-left and bottom-right images), where each generated image has the view and shape that linearly interpolates those of the two images. (C) Random gener- ation. For each category, generate images from shapes and views that drawn from Gaussian distributions.
12


Reviews:
[{"review_id": "tmPQJN3Lhbb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a probabilistic model which disentangles view, class and shape attributes explicitly (it does not rely on an emergent phenomena of disentangled factors in the latent space). In comparison to a similar approach, GVAE, CIGMO additionally disentangles the content factor into category and shape factors. Obtained results are reasonable and show advantage of the proposed method over related approaches.\n\nStrengths:\n- I believe the idea, while not being novel compared to GVAE, shows that one can scale VAEs into disentangling more factors. I imagine the model that can additionally incorporate the data where additional factors are present such as explicit camera viewpoint.\n- Theoretical grounding is well formulated.\n- Baselines are well chosen and show superiority over the basic baseline that came to my mind while reading this work - mixture of VAEs.\n\n\nWeaknesses:\n- The authors mentioned that their approach is heavily influenced by GVAE but do not mention how these two differ in any principal way. I perceive CIGMO as an incremental improvement over GVAE. To mitigate this issue, it would be great to see this work as a meta approach over GVAE where the authors include more factors that are easily obtainable (since ShapeNet is rendered either way, hence the authors can extract illumination, camera position and other factors) and how including these factors influence the results. With the presented model formulation, it seems to be possible to include more such factors. Finally, I would argue that such a meta approach would increase the applicability and extensibility of the work significantly.\n- Choosing 10 classes is not well supported. What classes are chosen? Does the final dataset include classes that are rare in the ShapeNet (besides, it would show how the model is able to handle such objects thanks to the new class factor)\n- The main part lacks a good diagram of the method. The plate model is not sufficient since details that make the model converge in the first place cannot be presented in such a diagram. The one presented in the Appendix does more harm than good and I spent more time to understand it than to follow equations directly. \n\nOther comments:\n- Overall, the paper is well written, however the authors should focus on typos present in the paper. Also, fixing or reformulating the sentence “our choice [ … ] is orthogonal to our goal” may reveal the main difference with GVAE. \nThe authors refer to biological motivation which is not mentioned later in the work. In this case, the whole paragraph is highly irrelevant to me. It is also formulated in such a way that one would argue that mammals are limited in terms of how many objects they can recognize by shape.", "extra_scores": {}, "extra_text": {}}, {"review_id": "V_2efhR_ja", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strong Points:\n1) The paper was fairly well written and easy to follow.\n2) The results are quite promising.\n\nQuestions to the author:\n1) It is unclear how the model learns the category without any supervision. It is recommended to provide more explanations as it is the major contribution.\n2) It is unclear how \\pi_c in equation 2 is determined. Does it follow the same distribution of the dataset?\n3) For figure 2, why was the proposed CIGMO not compared to GVAE plus k-means or IIC?\n4) For figure 3A, are the numbers of examples balanced in the 3 categories?\n5) For figure 3B (swapping), based on data preparation, shape should capture identity, but it is not obvious from the figure. Any thoughts?\n6) For the experiment with MultiPIE, grouping by emotion category might reveal more about the model.\n7) The model is C(category) times larger than GVAE. Is it possible that the performance gain comes from using larger model?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KuO96g3IQbe", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "============================\nSummary\n\nThis paper proposes a categorical invariant generative model (CIGMO) from a set of 2D images that tries to disentangle the factors of data category, intra-category geometry, and rendering viewpoint. CIGMO trains a VAE with a hierarchical graphical model that explicitly factors out the three components by design. Experiments on two datasets (ShapeNet rendered images, MultiPie face images) show that the proposed method can discover the concept of data category without using explicit supervision. It also supports feature manipulation over the geometry and viewpoint factors.\n\n\n============================\nPros\n\n1) The proposed task and approach are valid, reasonable and technically sound. Also, the paper is very easy to follow.\n2) This paper brings in a novel approach that explicitly models data category as a crucial factor for learning disentangled representations. \n3) Experiments show that given no supervision, the clustering based on the category latent variables makes sense. And, with 1-shot supervision, the classification performance is better than baseline methods.\n4) The learned geometry and viewpoint factors are well disentangled, as shown in the manipulation tasks.\n\n\n============================\nCons\n\n1) Can you show some quantitative evaluations or quantitative user studies to show the quality of the learned geometry and viewpoint disentanglement? How would you compare to previous works? I understand that your primary goal is to introduce data category factor into the game, but it is helpful for readers to know how much your model perform in terms of disentangling the standard two factors: geometry and viewpoint, and how do you compare to previous works?\n2) In the manipulation experimental figures you show in the paper, the reconstruction quality seems pretty bad. Especially for the experiments on face dataset (Fig 3, B), it's really hard for me to tell the person identity from the reconstructed face images. Can you provide more discussions/explanations on this?\n\n============================\nOverall Rating\n\nI like the novel task/method in this paper that brings in explicit modeling for the object category factor. Experiments comparing to baseline methods on unsupervised object category clustering and 1-shot object classification is quite convincing to me.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8Pp_ob19UEx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n- The proposed model can simultaneously model the category, shape, and view factors of general object images.\n- The supervision is only from the view-grouping information, which can be easily obtained. \n\nWeakness:\n- The proposed model is a mixture of disentangling models, which is not in an end-to-end manner.\n- The authors only compare the proposed model with other methods in two down-stream tasks. The authors should compare the generated results directly in quantitative and qualitative ways.\n- Experimental results are not convincing.  The quantitative comparisons of invariant clustering and one-shot classification are interesting settings and clearly show good performance of the proposed method. However, it does not seem very fair to compare methods with differently structured input. As the GVAE baseline does not model category (C=1) and the mixture of VAEs are trained with ungrouped data (K=1), it is easy to see why the proposed method performs better in such tasks. Adding a stronger baseline with the same level of supervision (grouped input) would make the comparison more convincing. In Figure 3 (B), the qualitative results of different views are not clearly shape-invariant. It would be clearer if the authors can show the swapping results on the ShapeNet objects (and possibly of other baseline methods). \n\nWhile the authors have validated the proposed model on the ShapeNet and MutliPIE datasets, the variations in these images are limited with simple backgrounds (in well-controlled lab setups). It is important to demonstrate whether the proposed method performs well on real-world datasets (e.g., LFW face, Pascal3D+). \n\nNovelty: The novelty of this work is generally good. The proposed model can discover the category, shape, and view factor at once, so as to generate general object images in a much larger space. The model capacity is much stronger than previous generative models which try to model general object images. What’s more, the model is trained in a weakly-supervised manner. The supervision information can be easily obtained. \n\nPresentation: The mathematical representation is easy to understand theoretically. On the other hand, the authors should give more initiative representations when describing the workflow of the proposed model and training procedure, which can make the paper much easier to understand. In addition, there should be more implementation details in the paper.\n\nEvaluation: As discussed above, the authors only compare their method with other methods in two downstream tasks. As such, the experimental results are not convincing. Could the authors directly show the comparison results between the proposed method and other approaches quantitively and qualitatively?  For example, as the authors claimed, the proposed method can model the category, shape, and viewpoint of images at once, but other previous models cannot achieve this goal. Can the authors directly compare their results with the results from other models to support this claim?  The authors should provide such results to support their claim more directly.\n\nContribution: The contributions of this paper are good, which can simultaneously discover those factors of general object images.\nNevertheless, the experimental results are lacking to clearly demonstrate the merits of this work.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Description:
The paper presents a weakly-supervised model CICGMO for disentangling category, shape and view information from images. Label information is not need as  the weak supervision is done by grouping together different views of the same object. They show that this outperforms other techniques on tasks such as invariant clustering and one-shot classification.

Strengths:
- Paper is well written
- Data category is explicitly modeled 
- The weakly supervision approach is appealing,  since the grouped data used as supervision information is easy to obtain
- Invariant clustering and one-shot classification results outperforms other methods significantly, showing CIGMO is doing a decent job at those tasks. This could be explained by CIGMO ability to better  disentangle category-shape-view.

Weaknesses:
- It is unclear how well (quality) the generative model is able to disentangle shape from view
- The reconstruction quality is quite low, such that it is difficult often times, in the MULTIPIE example, to clearly identify a face geometry.
- Generated results are not evaluated directly, but rather evaluation is done through down-stream tasks such as invariant clustering. This makes it difficult to show the quality of shape and view information.
Decision: Reject