Multi-Facet Clustering Variational Autoencoders
Authors: Fabian Falck
Haoting Zhang
Matthew Willetts
George Nicholson
Christopher Yau
Christopher C. Holmes
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-facet clustering
deep clustering
variational autoencoders
deep generative models

Abstract: Work in deep clustering focuses on finding a single partition of data. However, high-dimensional data, such as images, typically feature multiple interesting characteristics one could cluster over. For example, images of objects against a background could be clustered over the shape of the object and separately by the colour of the background. In this paper, we introduce Multi-Facet Clustering Variational Autoencoders (MFCVAE), a novel class of variational autoencoders with a hierarchy of latent variables, each with a Mixture-of-Gaussians prior, that learns multiple clusterings simultaneously, and is trained fully unsupervised and end-to-end. MFCVAE uses a progressively-trained ladder architecture which leads to highly stable performance. We provide novel theoretical results for optimising the ELBO analytically with respect to the categorical variational posterior distribution, correcting earlier influential theoretical work. On image benchmarks, we demonstrate that our approach separates out and clusters over different aspects of the data in a disentangled manner. We also show other advantages of our model: the compositionality of its latent space and that it provides controlled generation of samples.

Multi-Facet Clustering Variational Autoencoders
Fabian Falck ..6 Haoting Zhang *.2,5 Matthew Willetts 3,6
George Nicholson Christopher Yau 4.5.6 Chris Holmes 1,5,6
University Oxford UUniversity of Cambridge University College London University of Manchester HHellh Data Research UK The Alan Turing Institute fabian. falckestats. ox. ac.uk, uk, hz3818c1. cam ac. uk, mwilletts@turing... acc ac. uk,
george wwwhoolooossaaae..... ox ac.uk, cyaušturing. uk, cholmes@stats.co ac
Abstract
Work in deep clustering focuses finding single partition of data. However, high-dimensional data, such as images, typically feature multiple interesting char- acteristics could cluster over. For example, images of objects against back- ground could be clustered over the shape of the object and separately by the colour of the background. In this paper, Autoencoders (MFCVAE), a novel class of variational autoencoders with a hierarchy of latent variables, each with a Mixture-of -faasssiann prior, that learns multiple clusterings simultaneously, and is trained fully unsupervised and end-to-end. MFC- VAE uses a progressively-trainee ladder architecture which leads to highly stable performance. We provide novel theoretical results for optimising the ELBO analyt- ically with respect to the categorical variational posterior distribution, correcting earlier influential theoretical work. On image benchmarks, we demonstrate that our approach separates out and clusters over different aspects of the data in a disentan- gled manner. We also show other advantages of our model: the compositionality of latent space and that it provides controlled generation of samples.
introduce Multi-Facee Clustering Variational
Introduction
Clustering is the task of finding structure by partitioning samples in finite, unlabeled dataset according to statistical or geometric notions of similarity [ 3 For example, we might group items along axes of empirical variation in the data, maximise internal homogeneity and external separation of items within and between clusters with respect to a specified distance metric. The consequently validates clustering quality is fundamentally
choice of similarity measure and how a subjective one: depends on what is useful for a particular task [2] 4. this work, we are interested in uncovering abstract, latent setrrattrrsiee//ccetssseeteeeees and characterise the data-generative process. We further assume a fully exploratory, unsupervised setting without prior knowledge on the data, which could be exploited while fitting the clustering algorithm, and in particular without given ground-truth partitions at training time.
of the data understand
When being faced with nigh-dimensional data such images, speech electronic health records, items typically have more than one abstract characteristic. Consider the example of the MNIST dataset [5): MNIST images possess at least such characteristics: The digit class, which might impose the largest amount of statistical variation, and the style of the digit (e.g. stroke width). This naturally raises a question: By which characteristic is clustering algorithm supposed to partition the data? In MNIST, both digit class and (the sub-categories of) style would be perfectly reasonable candidates answer this question. In our exploratory setting described above, there is one "correct" partition of the data.
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

ooo
ton
ooo
21
21
(a) single-facet model
(b) multi-facet model (J=3)
Figure 1: Latent space of a (a) single-facet model and a (b) multi-facet model (J = 3) with two dimensions (21, 22) per facet. Both models perfectly separate the abstract characteristics of the data. However, the multi-facet model disentangles them into three sensible partitions (one per facet) and its required clusters scale linearly as opposed to exponentially w.r.t. the number of aspects in the data.
Deep learning based clustering algorithms, so-called deep clustering, were particularly successful in recent years in dealing with high-dimensional data by compressing the inputs into lower-dimensional latent space in which clustering is computationally tractable [6] 77. However, almost all of these deep clustering algorithms find only a single partition of the data, typically the one corresponding to the given class label in supervised dataset 88 9 TO 12 [ [5]. When evaluating their model, said approaches validate clustering performance by treating the one supervision label (e.g. digit class in the case of MNIST) as the de-facto "ground truth clustering". We argue that restricting our view to single facet C11 rather than all or at least multiple facets (C,,C...... C,....,,,))) is an arbitrary, incomplete choice of formulating the problem of clustering tigh-dimensional dataset.
To this end, we propose Multi-Facer Clustering Variational Autoencoders (MFCVAE), a principled, probabilistic model which finds multiple characteristics of the data simultaneously through its multiple couuures-o-GGuussaane (MoG) prior structure. Our contributions are as follows: (a) Multi-Facet Clustering Variational Autoencoders (MFCVAE), novel class of probabilistic deep learning models for unsupervised, multi-facet clustering in high-dimensional data that can be optimised end-to-end. (b) Novel theoretical results for the optimisation of the corresponding ELBO, correcting and extending an influential, related paper for the single-facet case. (c) Demonstrating MFCVAE's stable empirical performance in terms of multi-facet clustering of various levels of abstraction, compositionality of facets, generative, unsupervised classification, and diversity of generation.
2 Multi-facet clustering
High-dimensional data are inherently structured according to a number of abstract characteristics, and in exploratory setting, it is clear that arbitrarily clustering by one of them is insufficient. However, the question remains whether these multiple facets should also be explicitly represented by the model. In particular, one might argue that a single partition could be used to represent all cross-combinationn]]) of facets C = C₁ x C2 x ... x CJ where Cj = (1,2,.... Kj}. as in Fig. (a). In this work, we explain that explicitly representing and clustering by multiple facets, as we do in MFCVAE and illustrated in Fig. (b). has the following four properties that are especially desirable in an unsupervised learning setting:
(a) Discovering a multi-facet structure. We adopt a probabilistically principled, unsupervised approach, specifying independent, multiple Mixtures of Gaussians (MoG) prior the latent space. This induces a disentangled representation across facets, meaning that in addition to examples assigned to certain clusters being homogeneous, the facets (and their corresponding clusters) represent different abstract characteristics of the data (such as digit class or digit style). Because of this multi- facet structure, the total number of clusters required to represent given multi-partition structure of the data scales linearly w.r.t. the number of data characteristics. In comparison, the number of clusters required in a single-facet model scales exponentially (see Fig. and Appendix A for details).
Nooe that in practice, not all cross- combinaomsssss of facets might be present. For example, in a dataset like MNIST, one might not observe 'right-tilted zeros', even though observe 'right-tilted" digits and 'zeros'.

---

(b) Compositionality of facets. A multi-facet model has a compositional advantage: different levels of abstraction of the data are represented in separate latent variables. As we will show, this allows qualitatively diverse characteristics to be meaningfully combined.
(c) Generative, unsupervised classification. Our method joins a myriad of single-facet clustering models in being able to accurately identify known class structures given by the label in standard supervised image datasets. However, contrast to previous work, we are also able find interesting characteristics in other facets with homogeneous clusters. We stress that while we compare generative classification performance against other models to demonstrate statistical competitiveness, this task is not the main motivation for our fully unsupervised model.
(d) Diversity of generated samples. In generative sense, the structure of the latent space allows us to compose new, synthetic examples by a set of pairs of (continuous, discrete) latent variables. We can in partieular intervene each facet separately. This yields rich set of options and fine-grained control for interventions and the diversity of generated examples.
We illustrate these four properties in our experiments in Section
3 Multi-Facet Clustering Variational Autoencoders
Our model comprises latent facets, each learning its own unique clustering of samples via Mixture- of-Gaussians (MoG) distribution:
x
go
C ~ Cat(TT)) Zj cj ~ (N Ee,) (I)
where n the jth facet's Ky-dimensional vector of mixing weights, and (He, Ee,) are the mean and covariance of the cyth mixture component in facet j (Ee, can be either diagonal or full).
fo
N
N
Figure 2: Graphical model of MFCVAE. [Left] Variational posterior, qs ( c/x). [Right] Gen- erative model, po po ž, c)
The multi-facet generative model (Fig. 2 [Right]) thus structured as
pe (x (x (CC))po()) (x(x II po po(2)1cg)po()
{z1. where с = {c1, C2. ....... c. = Z2... }J}. and pe (x(z)) is a user-defined likelihood model, for example a product of Bernoulli Gaussian distributions. which is parameterised with a deep neural network fzz 0). Importantly, this structure in Eq. (2) encodes prior independence across facets, i.c. po(Z. c) = II Po (202 C,), thereby encouraging facets to learn clusterings that span distinct subspaces of The overall marginal prior po (z) can be interpreted as a product of independent MoGs.
3.1 VaDE tricks
To train this model, we wish to optimise the evidence lower bound (ELBO) of the data marginal likelihood using an amortised variational posterior gs (z, c/x) (Fig. 2 [Left]), parameterised by neural network (x: Φ), within which we will perform Monte Carlo (MC) estimation where necessary to approximate expectations
log P(D) 2 L(D: ExnD [nstttccooloog
Po Po(x, 8. go (Z. c|x) ]]
What should we choose for 9o c(x)? Training deep generative models with discrete latent variables can be challenging, as reparameterisation tricks so far developed, such as the Gumbel-Softma trick 16 17, necessarily introduce bias into the optimisation, and become unstable when a discrete latent variable has high cardinality. Our setting where we have multiple discrete latent variables is even more challenging. First, the bias from using the Gumbel-Softmax trick compounds when there a hierarchy of dependent latent variables, leading to poor optimisation [18] Second, we cannot necessarily avail ourselves of advances in obtaining good estimators for discrete latent variables as
either they do not carry over to the hierarchical case [19] restricted to binary latent variables
3

---

20 Third, we wish for light-weight optimisation, avoiding the introduction of additional neural networks whenever possible as this simplifies both training and neural specification.
Thus, we sidestep these problems, bias from relaxations of discrete variables and the downsides of additional amortised-postert poatterir neural networks for the discrete latent variables, by developing the hierarchical version of the VaDE trick. This trick was first developed for clustering VAEs with single Gaussian mixture in the generative model [00]. Informally, the idea (for single-facet model) is to define a Bayes-optimal posterior for the discrete latent variable using the responsibilities of the constituent components of the mixture model: these responsibilities are calculated using samples taken from the amortised posterior for the continuous latent variable.
Estimating the ELBO for models of this form does not require to take MC samples from discrete distributions- data likelihood conditioned only on the continuous latent variable Z, which we sample using the reparameterization trick [21]. and the posterior for ž is conditioned only when calculating the ELBO, we can cheaply marginalise out discrete latent variables where needed. In other words, we do not have to perform multiple forward passes through the decoder as neither nor the z samples we feed it depend on
Thus,
As it is fundamental to our method, we now briefly recapitulate the original VaDE trick for VAEs with a single latent mixture (correcting a misapprehension in the original form of this idea) and will then cover our hierarchical extensior
Single-Facet VaDE Trick: Consider a single facet model, so the generative model po (x, 2, c) =
(o(x p) (olc)))))) (2)c)po(() (c). Introduce a posterior gz(2, c|x) 2s (z x) ()g((cl)) where 2s (z x) is multi-
variate Gaussian with diagonal covariance. The ELBO for this model for one datapoint is
[(x: 0, Φ) = (x [log po(x(x)po(()c))p() = E (z.c)) flog Po po (z)po
(4)
9 (z)x))x)((
ls [z
where we have chosen to rewrite the generative model factorisation, pa (z) = Eg Pe (o)) is the marginal mixture of Gaussians, and po (c(z)) (p))))))) is the Bayesian posterior for c.
Expanding out the ELBO, we get
C(x; 0, 6) (x1x)) log (x KL (20 - E (a/x) KL [co(c))))lpp((c) (5)
We can define (c(x) such that (m)x)) KL (cs (lox)))lp((()2)))) [c|z)] is minimal, by construction, which is the case if we choose (c(x)) xx exp (B (z) x) log Po (c|z)) as we will show in Theorem This means that we can simply use samples from the posterior for z to define the posterior for c, using Bayes' rule within the latent mixture model.
Remark:
We note, however, that the original description of this idea in [0] it was claimed that (xlx) KL (pc(c)))lpppc( could, in general, be set to zero, which is not the case. Rather, this KL can be minimised, in general, detail and why the empirical results in [10] are still valid in Appendix
non-zero value. We discuss this misapprehension in more
Theorem 1. (Single-Face Facet VaDE Trick) For any probability distribution (z(z))), the distribution qs (c(c)) that minimises Bxg(s(x)) KL 14 [[((c) in (5)
argmin (x(()) KT. [go(cc))))pp (clgg(2)x))
(6)
go (c/x)
with the minimum value attained being
min E (xie)) KL [go(c(x)]lpo(ccz)] = log (z(x))
(7)
go(c(x)
exp (B (2/x) logp(c/z))
where
T(clg((zi)))
fore= 1,.., K
(8)
7 (2)x2
K
Z Z(qo(zix)) =
Σ exp (Exx( logp(clz))
(9)
Proof:
See Appendix B.I
'e note that the original VaDE paper. besides the misapprehension discussed in Section 3.1 and Ap- pendix B.I.I] proposed a highly complex training algorithm with various pre-training heuristics which we significantly simplified while maintaining or increasing performance (details in Appendix

---

Multi-facet VaDE Trick: In this work, we consider the case of having J facets, each with its own pair of variables Cj), Perhaps surprisingly, we do not have to make a mean-field assumption between the J facets for once we have made one for z. In other words, once we have chosen that (z, c|x) = 4s (c|x) I go(2) (x)), where (z(z)) (x) is defined to be a multivariate Gaussian with diagonal covariance for each j. the optimal qs (c(x)) similarly factorise: We formalise this:
Theorem 2. (Multi-Facei VaDE Trick for factorized (x(2)x), p(z,c) For any factorized probability distribution 9
II, ge the distribution (s(c(x) that minimises (xo(Ex) KL ( x) pe (clz)) under factorized prior p(ž, 11, p(z,,)))) o2) of
argmin (((x)) KL [1o(cc))]l (c(x))
II ny (c)9990(2z)x))
(10)
where the minimum value is attained at min (xx)x KL [plocccc))llpll(]]]
Σ log Z, (g0((())))))
(11)
(e(x))
exp(
(B ye (2, (z, (x) log po (2j) .fore c2 1....,KK (12)
where
T)((/19(2)/12))
Z, (g0(Z)|xx) xx]
K
[z(9o(z/1x))
exp (x) log pc
(13)
Proof: See Appendix B.2
Note that we use Eq.
12 the probability distribution of assigning input to clusters of facet j. Armed with these theoretical results, we can now write the ELBO for our model, with the optimal posterior for c, in a form that trivially admits stochastic estimation and does not necessitate extra recognition networks for c,
CMPCVAE (D;0, = ExnD (xx( po
Σ (eyl)x |
(14)
where the optimal (c(c//x) is given by Eq. for each
To obtain the posterior distributions for c, we take MC samples from 2o (Z(x and use these to construct the posterior as Eq. (12). We found one MC sample (L. 1; for each facet and for each x) to be sufficient. We derive the complete MC estimator which we use as the loss function of our model and ablations on two alternative forms Appendix C
3.2 Neural implementation and training algorithm
is worth pausing here to consider what neural architecture best suits our desire for learning multiple disentangled facets, and then further how we best train model robustly elicit from it well-separated facets. In the introduction, we discussed the different plausible ways to cluster high- dimensional data, such as in MNIST digits by stroke thickness and class identity. These different aspects intuitively correspond to different levels of abstraction about the image. It is thus natural that these levels would be best captured by different depths of the neural networks in each amortised posterior. These ideas have motivated the use of ladder networks in deep generative models that aim to learn different facets of the input data into different layers of latent variables. Here, we take inspiration from Variational Ludder Autoencoders (VLAES) 22 A VLAE architecture has a deterministic "backbone" both the recognition and generative model. The different layers of latent variables branch out from these at different depths along. This inductive bias naturally leads to stratification and does so without having bear the computational cost of training completely separate encoder (say) for each layer. Here, use this ladder architecture for MFCVAE, as illustrated in Fig. and refer to Appendix D.2 for further implementation details.
*e also provide the VaDE trick for the general form of posterior for Z, i.e. without assuming the factorisation (2(x) = II/ go (2, (x), in Appendix B.3
5

---

Further, we found progressive training [23], pre- viously shown to help VLAEs learn layer-by- layer disentangled representations, to be of great use in making each facet consistently represent the same aspects of data. The general idea of progressive training is to start with training a sin- gle facet (typically the one corresponding to the deepest recognition and generative neural net- works) for certain number of epochs, and pro- gressively and smoothly loop in the other facets one after the other. We discuss the details of our progressive training schedule in Appendix [D.3] We find that both the VLAA architecture and pro- gressive training are jointly important to stabilise training and get robust qualitative and quantitative results as we show in Appendix E.
h2
h
N
Figure 3: Ladder-MFCVAS architecture. [Left]
Variational posterior. [Right] Generative model.
4 Experiments
In the following, we demonstrate the usefulness our model and its prior structure in four experimen- tal analyses: (a) discovering multi-facet structure (b) compositionality of latent facets (c) generative, unsupervised classification, and (d) diversity of generated samples from our model. We train our model three image datasets: MNIST (5), 3DShapes (two configurations) [24] and SVHN (25). We refer to Appendices D and E for experimental details and further results. We also provide our code implementing MFCVAE, using PyTorch Distributions (26). and reproducing our results at http://gtthub..oooaaaaaaiaa/a
cvae
4.1 Discovering multi-facet structure
We start by demonstrating that model can discover multi-facet structure in data. Fig. 4 visualises input examples representative of clusters in a two-facet (J = 2) model. For each facet j. input examples æ with latent variable zj are assigned latent cluster сj argmax, イ 40 go (x) according to Eq. 122 Surprisingly, we find that
represent the two most striking data characteristics digit class and style (mostly in the form of stroke width, e.g. 'bold', 'thin') ' in MNIST, object shape and floor colour in 3DShapes (configuration 1), and digit class and background colour in SVHN-in two separate facets of the data. In each facet, clusters are homogeneous w.r.t. a value from the represented characteristic. When comparing our results on MNIST with LTVAE (27). the model closest to ours in its attempt to learn clustered latent space of multiple facets, LTVAE struggles separate data characteristics into separate facets (c.f. (27 Fig. 5; in particular, both facets learn digit class, i.c. this characteristic is not properly disentangled between facets), whereas MFCVAE better isolates the two.
To quantitatively assess the degree of disentanglement in the learned multi-faces structure of our model, we perform set of supervised experiments. For each dataset, formulate three classification tasks, for which we use latent embeddings 2,, 22 and Z, respectively, sampled from their corresponding amortised posterior, as inputs, and the label present in the dataset (e.g. digit class in MNIST) as the target. For each task and dataset, we train (on the training inputs) a multi-layer perceptron of one hidden layer with 100 hidden units and ReLU activation, and an output layer followed by softmax activation, which are the default hyperparameters in the Python package sklearn. Table [ shows accuracy of these experiments. We find that the supervised classifiers predict the supervised label with high accuracy when presented with latent embeddings which we found to cluster the abstract characteristic corresponding to this label, or with the concatenation of both latent embeddings. However, when presented with latent embeddings corresponding to the "non- label" facet, the classifier should-if facets are strongly disentangled- information to learn the supervised mapping, and this is indeed what we find, observing significantly worse performance. This demonstrates the multi-facet structure of the latent space, which learns separate abstract characteristics of the data.
presented with useful

---

148
7
9
00
4:
14
22
1
3
3
3
3
3
3
3
3
3
3 333
3.
3 303
3
digit/shape facel
7
7
7
7
7
7
4
4 4244444
41
0
0
0
O
O
0
0 6 5 4 0 1 6 2 6 7
17 17
10
6
6
6
6
6
2 2
2 212
5 4
717 C7 7 57 7 37875
4 4
4
4
4
4
18 8 8 16 2618 6 8 65
0 0
2 2. 25 28 21 2
(
Lo
L
9 9 9 9 9
9
2
2
2
2
2
2
2
7 7:7
74 777Y 7
8
Q
6
0 3
0
105
2.4
( 29. 681 8
305
3 5
7
5
16 5
1 3
4 9 5
1E 6
style/colour facet
9
2
1
7
2
6 9 3
2
4AE
1
7 7
A
7
3
Q
8
1 7 0 2 37 5 7
61
10
16
37
40
17
18
2
9
7
+
9
3
51
4:
6
11
10
6
1
9
5
2
29
6
6
0
11
90
3
5
2
20
37
30
0
7
4
4
9 q 5
14 16
29
69
15
4
2
3
4
8
6
32
90
10
1:
1
0 7
0
3
3
7
5
1 11:55:
MNIST
3DShapes
SVHN
Figure 4: Input examples for clusters of MFCVAE with two-facets (J= 2) trained on MNIST, 3DShapes and SVHN. Clusters (rows) in each facet j are sorted in decreasing order by the average assignment probability of test inputs over each cluster. Inputs (columns) are sorted in decreasing order by their assignment probability max,, π y 14 (z)(x))). We visualise the first 10 clusters and inputs from the test set (see Appendix E.3 for all clusters).
Table 1: Supervised classification experiment to assess the disentanglement of MFCVAE's multi-facet structure all three datasets. Values report test accuracy in %. Error bars the sample standard deviation across runs.
MNIST
3DShapes config.
3DShapes config.
SVHN
digit class
object shape
floor colour
object shape
wall colour
digit class
17.34 (0.24)
95.00 (0.45)
20.00 (0.68)
98.26 (0.16)
73.40 (1.48)
69.46 (0.36)
Z2
94.95 (0.04)
32.43 (1.38)
100.00 (0.00)
24.41 (1.34)
100.00 (0.00)
22.30 (0. (6.16)
z
95.27 (0.07)
95.18 (0.42)
100.00 (0.00)
98.19 (0.30)
99.97 (0.06)
70.39 (0.29)
4.2 Compositionality of latent facets
unique advantage the prior structure of MECVAE compared to other unsupervised generative models, say a VAE with isotropic Gaussian prior, is that it allows different abstract characteristics to be composed in the separated latent space. Here, we show how this enables interventions on per-facet basis, illustrated with two-facet model where style/colour is learned in one facet and digit/shape is learned in the other facet. Let us have two inputs x and x assigned to two different style clusters according to Eq. [22 (and two different digit clusters). For both inputs, we obtain their latent representation 2, as the modes of qo (2, respectively. Now, we swap the style/colour facet's representation, i.e. ă of both inputs for MNIST, and 22 of both inputs for 3DShapes and SVHN, and pass these together with their unchanged digit/shape representation (ăg for MNIST and ă, for
3DShapes and SVHN) through the decoder f(ž; 6) to get reconstructions x(1) f((z(") 2
(2) })
and (2) f((z(2) (2) ) () which visualise in Fig. (see Appendix
E.4 for more rigorous
explanation this swapping procedure).
Surprisingly, by construction of this intervention in our multi-facet model, we observe reconstruc- tions that "swap" their style/hackground colour, yet in most cases preserve their digit/shape. This intervention is successful across a wide set of clusters MNIST and 3DShapes. It works less so

---

Input
3
4
22
81
8g 1
81
82
22
2 2g
"swap"
"sswap"
"swap"
Recon- struction
3
Figure 5: Reconstructions of two input examples when swapping their latent style/colour.
Table 2: Unsupervised clustering accuracy (%) of single-facet (SF) and multi-facet (MF), generative (G) and non non-generative (NG) models on the test set. Error bars (if available) are the sample standard deviation across multiple runs. Results marked with " do not provide error bars.
Method
MNIST
SVHN
DEC (8); SF; NG)
84.3 V
11.9 (0.4)
VaDE [00: MLP; SF: G)
94.46 "; 89.09 (3.32)
27.03 (1.53)
VaDE (I0 conv.; SF: G)
92.65 (1.14)
30.80 (1.99)
IMSAT (II; SF; NG)
98.4 (0.4)
57.3 (3.9)
ACOL-GAR (15): SE; NG)
98.32 (0.08)
76.80 (1.30)
VLAC ([88;; MF; G)
37.8 (2.2)
LTVAE ((77: MF; G)
86.3
MFCVAE (ours; MF; G)
92.02 (3.18)
56.25 (0.93)
SVHN where we hypothesise that this is due to the much more diverse dataset and (consequently) the model reaching a lower fit (see Section 4.3). We show further examples including failure cases in Appendix E.4 which show that model learns multi-facet structure allowing complex inventions.
4.3 Generative, unsupervised classification
Recall our fully unsupervised, exploratory setting of clustering where the goal is to identify and characterise multiple meaningful latent structures de novo. In practice, we have no ground-truth data partition- if labels were available, the task would be better formulated in the first place. That said, it is often reasonable to assume that the class label in a supervised dataset represents a semantically meaningful latent structure that contributes to observed variation in the data. Indeed, this assumption underlies the common approach for benchmarking clustering models on labelled data: the class label hidden during training: afterwards it is revealed as a pseudo ground-truth partitioning of the data for assessing clustering "accuracy". MFCVAE aims to capture multiple latent structures and can be deployed as a multi-facee generative classifier, distinct from standard single-face discriminative classifiers [ p.30J. But we emphasise that high classification accuracy is attained as by-product, and is not our core goal-an we do not explicitly target label accuracy, nor does high label accuracy necessarily correspond to the "best" multi-facet clustering.
supervised classification
we
Following earlier work, in Table we report classification performance on MNIST and SVHN in terms of unsupervised clustering accuracy on the test set, which intuitively measures homogeneity w.r.t. a set of ground-truth clusters in each facet (see Appendix F..5 for formal definition). We compare our method against commonly used single-facet (SF) and multi-facet (MF), generative (G) and non- -onneeeereeevee (NG) deep clustering approaches (we use results as reported) of both deterministic and probabilistic nature. We report the mean and standard deviation (if available) of accuracy over T runs with different random seeds, where 10 for MFCVAE. For VaDE [1O], we report results from the original paper, and our two implementations, one with multi-layer perceptron encoder and decoder architecture, one using convolutional layers. Models marked with explicitly state that they instead report the best result obtained from R restarts with different random seeds (DEC: R = 20, VaDE: 10). Both of these types of reporting in previous work-not providing error bars over several runs and picking the best run (while not providing error w.r.t. initialisation. We further discuss this issue and the importance stability in deep clustering approaches in Appendix E.I
stability of the model
8

---

3
3
3
3
3
3
3
3
3
6
3
6
3
3
3
digitshape facet
7
7
7
7
7
7
7
7
7
0
0
0
O
O
0
O
O
0
0
7
7
7
6
6
6
6
6
6
2
5 5
5
S
S
5 4 0
5
5 4
4 4 4 4
4
4
4
4
6
6
8
6
6
8
6
8
8
8
0
0
0
0
2
2
2
2
2
6
6
6
6
6
6
6
6
2
2
2
2
2 9
0
8
5
7
I
0
5
7
366
1
18
style/colour facet
1
5 1
+
2
8
3
9 3
6
3
O
0 8
10
7
5
3
5
6
4
8
4
7
4
9
9
2
7
D
9
61
8
0
7
5
5
8
0
4
50 8
15
6
0 8
2
7
5
31
34
5
0
7
9
/
5 5
2
3
0
5
8
I
4
/
6
50
19
2015
80
2 6
4
7
4
2
5
18177
MNIST
3DShapes
SVHN
Figure 6: Synthetic samples generated from MFCVAE with facets (J 2) trained MNIST, 3DShapes, and SVHN. For each cluster Cj in facet j. 2j is sampled from p(z, (cy) and Zy is sampled from p(zy') for the other facet j' f j. Each Clusters (rows) are sorted and selected (and are from the same trained model) as in Fig. (see Appendix F.6 for visualisation of all clusters and comparison with LTVAE).
corresponds to 10 random samples from a cluster.
MFCVAE is able to recover the assumed ground-truth clustering stably. It achieves competitive performance compared to other probabilistic deep clustering models, but clearly outperformed by ACOL-GAR on SVHN, single-facet, non-generative and deterministic model which does not possess three of the four properties demonstrated Sections 4.1 (.4) and 4.2p. Besides the results presented
in the table, we also note that MFCVAE performs strongly on 3DShapes, obtaining 99.46% + 1.10% for floor colour and 88.47% + 1.82% for object shape on configuration 1, and 100.00% + 0.00% for wall colour and 90.05% + 2.65% for object shape configuration 2. Lastly. it is worth noting that we report classification performance for the same hyperparameter configurations and training runs of our model that used in all experimental sections and in particular for Fig, 4 5 and i.c. trained model has a pronounced multi-facet characteristic. contrast, while it is somewhat unclear, LTVAE seems to report its clustering performance when trained with only single facet, not when performing multi-faces clustering 27
4.4 Diversity of generated samples
We lastly show that MFCVAE enables diverse generation synthetic examples for each given cluster in the different facets, as downstream task in addition to clustering. To obtain synthetic examples for cluster Gj in facet j. we sample zj from pcz,le;). and sample zy from p(z,)) for other facets j' # j. We then take the modes of px(x] (2) where z (Zı,.... ..j....," (,,) as the generated images. Fig. shows synthetic examples generated from the models (J = 2) trained on MNIST, 3DShapes and SVHN.
For all three datasets, we observe synthetic samples that are homogeneous w.r.t. the characteristic value (e.g. 'red background") of cluster in the chosen facet (as we sample this continuous latent variable from the conditional distribution). but heterogeneous and diverse w.r.t. all other facets (as we sample all other continuous latent variables from their marginal distribution). For example, on MNIST, when fixing a cluster the digit facet, we observe generated samples that have the same digit class (e.g. all "1'), but are diverse in style (e.g. different 'stroke width'). Conversely, when fixing a cluster the style facet, we get samples homogeneous in style, but heterogeneous in digit class. Likewise, on 3DShapes, fixing cluster in the wall colour facet produces generations diverse in shape.

---

but having the same wall color, and conversely when fixing the shape facet. Besides, in all clusters. generated samples are diverse w.r.t. other factors of variation on 3DShapes, such as orientation and scale. On SVHN, while less strong than in Fig. 4 these patterns extend here to the two facets style (background colour is particularly distinct) and digit class. These results are consistent with and underline the observed disentanglement of facets that we found in our previous experimental analyses. We also compare sample generation performance between MFCVAE and LTVAE and assess the diversity of generations quantitatively in Appendix E.6
5 Related work
Within the deep generative framework, various deep clustering models have been proposed. VaDE [T] is the most important prior work, a probabilistic model which has been highly influential in deep
clustering. Related approaches, GM-VAEs [29] and GM-DGMs 30 31 [2], have similar overall
performance and explicitly represent the discrete clustering latent variable during training. Non-
parametric approaches include DLDPMMs [30,, and HDP-VAEs [33]. Further, many non-generative
methods for clustering have been proposed that use neural components [
15
29
12
All
these approaches, however, propose single-facet models.
Hierarchical VAEs can both be way to learn more powerful models 34
35
36
37
38)
but can also
enable to separate out representations where each layer of latent variables represents different aspect of the data. Variational Ladder Autoencoders (VLAEs) [2] aim to do the latter: to learn independent sets of latent variables, each representing some part of the data; but each group of latent variables within this set has a (0, 1) prior, so it does not perform clustering. Recently, progressive training for VLAEs has been proposed 23 which sharpens the separation between layers. Here, we also mention disentanglement methods 33 40
42 which likewise attempt to find separated latent variables. However, rather than discovering facets through the prior and a hierarchical structure, these techniques attempt to find to deviate from the ELBO. Unfortunately, these methods require lucky selection of hyperparameters to work 43 and do not provide clustered latent space.
41
-nndeeenll representations via regularisation, leading the loss
Learning multiple clusterings simultancously has been studied in the case of very low dimensional under the names alternative clusterings and non-redundant clustering. However, when comes to clustering nigh-dimensional data like images, approaches are rare. The recently proposed LTVAE (27] aims to perform this task, proposing variational autoencoder with a latent tree model prior for set of continuous latent variables ž, of which each Zj has a GMM prior. The neural components are trained via stochastic gradient ascent under the ELBO; this is interleaved with a heuristic (hill-climbing) search algorithm to grow or prune the tree structure and message- passing to learn its nodes' GMM parameters of the current structure of the tree prior in a manner reminiscent of SVAEs [49]. rendering the entire training algorithm not end-to-end differentiable (in contrast to MFCVAE). LTVAE learns multiple clusterings over the data, however, lacks proper disentanglement of facets, as discussed in Section 4.1
datasets 45
46
47
48
Conclusion
We introduced Multi-Facet Clustering Variational Autoencoders (MFCVAE), novel class of proba- bilistic deep learning models for unsupervised, multi multi-partition clustering in high-dimensional data which is end-to-end differentiable. We provided novel theoretical results for optimising its ELBOO, correcting and extending an influential related paper for the single-facet case. We demonstrated MFCVAE's empirical performance in terms of multi-facet clustering of various levels of abstraction, and the usefulness of its prior structure for composing, classifying and generating samples, achieving state-of-the-ar performance among deep probabilistic multi-facet models.
An important limitation of our work shared with many other deep clustering algorithms is the lack of procedure to find good hyperparameters through metric known at training time. Future work should explore: MFCVAE with J > 2: automatic tuning of hyperparameters J and Ký; application to lcage-scale datasets of other modalities; and regularising the model wise to further enforce disentangled representations in the latent space (50). While we successfully stabilised model training, further work will be key to harness the full potential of deep clustering models.
10

---

Acknowledgments and Disclosure of Funding
FF and HZ acknowledge the receipt of studentship awards from the Health Data Research UK-The Alan Turing Institute Wellcome PhD Programme in Health Data Science (Grant Ref: 218529/7/19/Z). HZ acknowledges the receipt of Wellcome Cambridge Trust Scholarship. MW grateful for the support of UCL Computer Science and The Alan Turing Institute. GN acknowledges support from the Medical Research Council Programme Leaders award MC_UP
UP A390 1107. CY is funded by UKRI Turing AI Fellowship (Ref: EP/VO23233/I) CH acknowledges support from the Medical Research Council Programme Leaders award MC UP A390 1107, The Alan Turing Institute, Health Data Research, U.K., and the U.K. Engineering and Physical Sciences Research Council through the Heyes4Health programme grant.
The authors report no competing interests.
We thank Tomas Lazauskas, Jim Madge and Oscar Giles from the Alan Turing Institute's Research Engineering team for their help and support. We thank Adam Huffman, Jonathan Diprose, Geoffrey Ferrari and Colin Freeman from the Biomedical Research Computing team the University of Oxford for their help and support. We thank Angela Wood and Ben Cairns for their support and useful discussions.
References
[1] Kevin P. Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.
[2] Rui Xu and Donald Wunsch. Survey of clustering algorithms. IEEE Transactions neural networks, 16(3):445677, 2005.
[3] Pierre Hansen and Brigitte Jaumard. Cluster analysis and mathematical programming. Mathe- matical programming, 19(1-3):19--215 1997.
[4] Ulrike Von Luxburg, Robert C. Williamson, and Isabelle Guyon. Clustering: Science or art? In Proceedings of ICML workshop unsupervised and transfer learning, pages 65-79. JMLR Workshop and Conference Proceedings, 2012.
[5] Yann LeCun, Corinna Cortes, and C. Burges. MNIST handwritten digit database. http
//yann. lecun. com/ exdb/ weweeeendissseeeeeeeeeeeeeeeedee mwtst/miist//t////////mmm//mmmss 2010.
[6] Erxue Min, Xifeng Guo, Qiang Liu, Gen Zhang, Jianjing Cui, and Jun Long. A survey of clustering with deep learning: From the perspective of network architecture. IEEE Access, 6:39501-39514, 2018.
[7] Elie Aljalbout, Vladimir Golkov, Yawar Siddiqui, Maximilian Strobel, and Daniel Cremers. Clustering with deep learning: Taxonomy and new methods, arXiv preprint 1rXiv:1801.0664 2018.
[8] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pages 478-487. PMLR, 2016.
[9] Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. Towards k-means-frieer friendly spaces: Simultaneous deep learning and clustering. In International Conference on Machine Learning, pages 3861-3870. PMLR, 2017.
[10] Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. In Proceedings of the Twenty-Sixth International Join! Conference on Artificial Intelligence, IJCAI-17, pages 1965-1972, 2017.
[11] Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In International Conference on Machine Learning, pages 1558-1567. PMLR, 2017.
[12] Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri, and Yuval Kluger. SpectralNet: Spectral clustering using deep neural networks. In International Conference on Learning Representations, 2018.
[13] Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, and Wei Liu. Deep spectral clustering using dual autoencoder network. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4066-4075, 2019.

---

[14] Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan. ClusterGAN: Latent space clustering in generative adversarial networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4610-4617, 2019.
[15] Ozsel Kilinc and Ismail Uysal. Learning latent representations in neural networks for clustering through pseudo supervision and graph-based activity regularization. In International Conference on Learning Representations, 2018.
[16] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous re- laxation of discrete random variables. In International Conference on Learning Representations, 2017.
[17] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztion with gumbel-softmax. In International Conference on Learning Representations, 2017.
[18] Valentin Liévin, Andrea Dittadi, Lars Maalóe, and Ole Winther. Towards hierarchical discrete variational autoencoders. In 2nd Symposium on Advances in Approximate Bayesian Inference. 2019.
[19] Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, and David Duvenaud. Backprop- agation through the void: Optimizing control variates for black-box gradient estimation. In International Conference Learning Representations, 2018.
[20] Adeel Pervez, Taco Cohen, and Elstratios Gavves. Low bias low variance gradient estimates for boolean stochastic networks. In International Conference on Machine Learning, pages 7640. PMLR, 2020.
[21] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13X1v:1312.61 14,61111100011 2013.
[22] Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep generative models. In International Conference on Learning Representations, 2017.
[23] Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progressive learning and disentanglement of hierarchical representations. In International Conference on Learning Representations, 2020.
[24] Chris Burgess and Hyunjik Kim. 3D shapes dataset. http::/giitubbcoomddeppnnn///// dataset/, 2018
[25] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[26] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024-8033 2019.
[27] Xiaopeng I.,, Zhourong Chen, Leonard K. M. Poon, and Nevin L. Zhang. Learning latent super- structures in variational autoencoders for deep multidimensional clustering. In International Conference Learning Representations, 2019.
[28] Matthew Willetts, Stephen J. Roberts, and Chris Holmes. Disentangling to cluster: Gaussian mixture variational ladder autoencoders. In 4th Workshop on Bayesian Deep Learning (NeurIPS 2019), 2019.
[29] Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnelo, Matthew C. H. Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture VAE. arXiv preprint arXiv:
2017.
[30] Eric Nalisnick, Lars Hertel, and Padhraic Smyth. Approximate inference for deep latent gaussian mixtures. In Workshop on Bayesian Deep Learning Workshop (NIPS 2016), 2016.
[31] Matthew Willetts, Stephen J. Roberts, and Christopher C Holmes. Semi-unsupervised learning using deep generative models. In 3rd Workshop on Bayesian Deep Learning (NeurIPS 2018), 2018.
12

---

[32] Matthew Willetts, Stephen J. Roberts, and Christopher C Holmes. Semi Semi-unsuperviser leaming: Clustering and classifying using ultra-sparse labels. In JEEE Big Data Workshop, 2020.
[33] Prasoon Goyal, Zhiting Hu, Xiaodan Liang, Chenyu Wang, and Eric P. Xing. Nonparametric variational auto-encoders for hierarchical representation learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 5094-5102 2017.
[34] Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, 2016.
[35] Casper Kaae Sønderby, Tapani Raiko, Lars Maalge, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in Neural Information Processing Systems, 2016.
[36] Lars Maalée, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: A very deep hierarchy of latent variables for generative modeling. In Advances in Neural Information Processing Systems, 2019.
[37] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems, 2020.
[38] Rewon Child. Very deep VAEs generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021.
[39] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, 2017.
[40] Emile Mathieu, Tom Rainforth, N. Siddharth, and Yee Whye Tch. Disentangling disentangle- ment in variational autoencoders. In International Conference on Machine Learning, pages 4412. PMLR, 2019.
[41] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pages 2649-2658. PMLR, 2018.
[42] Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement variational autoencoders. arXiv preprint arXiv:/802.04942, 2018.
[43] Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pca directions (by accident). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12406-12415, 2019.
[44] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14 4124. PMLR, 2019.
[45] Ying Cui, Xiaoli Z. Fern, and Jennifer G. Dy. Non-redundant multi-view clustering via orthogonalization. Proceedings IEEE International Conference on Data Mining, ICDM, 3:133-142, 2007.
[46] Ian Davidson and Zijie Qi. Finding alternative clusterings using constraints. Proceedings IEEE International Conference on Data Mining, ICDM, pages 773-778, 2008.
[47] Zijie Qi and Ian Davidson. A principled and flexible framework for finding alternative cluster- ings. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 717-725, 2009.
[48] Emmanuel Muller, Stephan Gunnemann, Ines Farber, and Thomas Seidl. Discovering multiple clustering solutions: Grouping objects in different views of the data. In 2012 IEEE 28th International Conference on Data Engineering, pages 1207- IEEE, 2012.
[49] Matthew James Johnson, David Duvenaud, Alexander B. Wiltschko, Sandeep R. Datta, and Ryan P. Adams. Composing graphical models with neural networks for structured representa- tions and fast inference. In Advances in Neural Information Processing Systems, 2016.
[50] William Peebles, John Peebles, Jun- Yan Zhu, Alexei A. Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement, In Proceedings of European Conference Computer Vision (ECCV), 2020.
13

---

[51] M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 1004 2018.
[52] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel Smith, et al. Array programming with NumPy. Nature, 585(7825):357-3366 2020
[53] Lukas Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.
[54] D. Hunter. Matplotlib: A 2D graphies environment. Computing in Science & Engineering. ((3):90-95, 2007.
[55] Michael L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6(60))3021, 2021.
[56] Guido Van Rossum. The Python Library Reference, release 3.8.2. Python Software Foundation, 2020.
[57] Andrew Collette. Python and HDF5. O'Reilly, 2013.
[58] G. Bradski. The OpenCV library. Dr. Dobb's 's Journal of Software Tools, 2000.
[59] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python, Journal of Machine Learning Research, 12:2825-2883 2011.
[60] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256, 2010.
[61] Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Data-dependent initializa- tions of convolutional neural networks. In International Conference on Learning Representa- tions, 2016.
[62] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pitrre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
[63] Harold W. Kuhn. The Hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[64] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[65] Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolu- tions. In Advances in Neural Information Processing Systems, 2018
[66] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference Muchine Learning, pages 4055-4064. PMLR, 2018
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We set out our theoretical and methodological contri- butions in the abstract and introduction. We then motivate the problem and state the properties of our model in Section 2 discuss our theoretical contributions in Section 3 and demonstrate the model's properties empirically Section
(b) Did you describe the limitations of your work? [Yes] We describe the limitations of our approach both explicitly, and implicitly as part of the future work in Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We address this question in Appendix D.4
14

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Prior inde- pendence across facets and the mean-field assumption of the posterior required for Theorems I and 3 are stated in Section 3
(b) Did you include complete proof's of all theoretical results? [Yes] The complete proofs of Theorems
2 and
3 in Section 3.1 are shown in Appendix B
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We provide our code, instructions on how to download the data, and on how to reproduce the main experimental results on httts://ggg thub. com htt//Fabiaaaaaaaa ck mf cvae
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix D.4
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] Throughout the paper (Section 4.3) Appendices [E.1 E.3) and across all datasets, we report error bars (standard deviation over 10 runs with different random seeds) for our model (MFCVAE). Specifically, we discuss the importance in deep clustering research of considering error bars to ensure stability of results in Appendix [F. and discuss the shortcomings of not reporting error bars and picking the best of multiple runs with different random seeds among previous work in Section 4.3
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] We address this question in Appendix D.4
4. If you are using existing assets (c.g., code, data, models) or curating/releusing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] We discuss all existing
assets used in Appendix and briefly refer to key ones in Section 4
(b) Did you mention the license of the assets? [Ye]] We mention the license of the assets
(if known) in Appendix D.4]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We provide source code with detailed instructions installation and how to reproduce our results, pretrained models, and logged runs with additional metrics and figures on https::///// gi thub. com hamm/aabianFallloc paragraph New assets for further details.
cvae See Appendix D.4
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We address this question in Appendix D.I
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We address this question in Appendix D.I
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "Xfp1zCqi2L6", "reviewer": "Reviewer_vHkY", "summary": "The paper presents Multi-Facet Clustering Variational Autoencoder which is a probabilistic model base on multiple mixtures of Gaussians that find multiple characteristics of the data simultaneously. Moreover, the paper shows theoretical results for the optimisation of ELBO analytically with respect to the categorical variational posterior distribution.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe authors of this paper deal with a known problem of clustering a multi-facet that is less common than single-facet models. In the proposed method, they use well-known techniques such as Monte Carlo, progressive training, a mixture of Gaussians, or estimating the ELBO but it is a novel combination of them. The novel theoretical results for optimising its ELBO is extending an influential related paper for the single-facet case. \n\nQuality and clarity:\nIn general, the paper is a complete piece of work but section Experiments needs some refinement. The authors show the results of the proposed method MFCVAE and severals samples generated from MFCVAE trained on MNIST, 3DShapes, and SVHN. In my opinion, quantitative results are lacking compared to other methods. \nThe paper was presented classification performance in terms of unsupervised clustering accuracy on the test set, comparing commonly used single-facet and multi-facet deep clustering approaches. These results show the average effectiveness of the proposed method, the authors emphasize that high classification accuracy was not their goal. A table would be a better form of presenting these results. It is not clear why actors do not use the same comparative methods for different datasets.\n\nSignificance:\nThe paper presents a new approach to a less-known problem of clustering a multi-facet. An important result of this work is novel theoretical results for optimising its ELBO analytically concerning the categorical variational posterior distribution. This result may be used by other researchers. Unfortunately, the evaluation of the proposed method requires additional work (I have given some of the comments above). The paper presents only the results for J = 2, while for most of the data sets J is much greater than 2. The positive fact is that the authors know about it, they indicate that this problem will be solved in the future.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "OqFaIRMN0d8", "reviewer": "Reviewer_NacR", "summary": "There are multiple interesting aspects of high-dimensional data.\nThis paper proposes a Multi-Facet Clustering Variational Autoencoders (MFCVAE) to cluster the data in multi-aspects.\nMFCVAE utilizes and extends the trick of VaDE, and provides the methods to learn multi-aspect clustering in an end-to-end fashion.\nThis paper provides various qualitative results to validate the multi-aspect clustering performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "< originality >\n1. MFCVAE utilizes a mixture of Gaussian prior for multiple clustering simultaneously, and the task is important and novel.\n2. For simultaneous clustering, MFCVAE utilizes and extends the VaDE tricks. The extension is somewhat novel.\n3. MFCVAE is related to disentangled representation, but this paper does not discuss the relationship between MFCVAE and various disentanglement models, [1,2,3]. Discussion on the relationship between MFCVAE and various disentanglement methods will be interesting\n\n< quality >\n1. The technical part is sound\n2. The paper does not provide the quantitative methods to measure the disentanglement and sample diversity. The lack of quantitative measure makes the reader not ensure the effectiveness of MFCVAE\n\n< clarity >\n1. The paper is well-written.\n2. It is hard to clarify the quantitative experimental results. There is no quantitative results table.\n3. There exist only J=2 experiments, and additional experimental results for J>2 will be more interesting.\n\n< significance >\n1. The experimental results of MFCVAE are not significant. On MNIST, MFCVAE shows 92.02%, but IMSAT achieves 98.4%.\n2. There is no quantitative diversity metric, and it is hard to ensure that the MFCVAE generates better diverse samples.\n\n[1] Kim, Hyunjik, and Andriy Mnih. \"Disentangling by factorising.\" International Conference on Machine Learning. PMLR, 2018.\n[2] Mathieu, Emile, et al. \"Disentangling disentanglement in variational autoencoders.\" International Conference on Machine Learning. PMLR, 2019.\n[3] Sha, Lei, and Thomas Lukasiewicz. \"Multi-type Disentanglement without Adversarial Training.\" Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "N53aT-vPxl", "reviewer": "Reviewer_bBmr", "summary": "The authors extend the VaDE model (Jiang et al.) to allow for multiple, independent facets i.e. Mixtures of Gaussians. The authors first point out and resolve a technical oversight in derivations related to estimation of $q(c \\mid x)$ of Jiang et al. (VaDE). The authors implement their formulation in an architecture based on ladder VAE to extract multiple clusterings of the data simultaneously. The authors demonstrate their method on 3 different benchmark datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Note: I'm not very familiar with other comparable models learning hierarchical structure in latent variables, and cannot judge the novelty of the contribution too well. I'm willing to revise my score in case I have missed something. \n\nPros\n - The paper is clearly written, and the theoretical portion is well laid out and pleasant to read. The ideas are uncluttered, and the appendices provide enough context for an outsider to the field to follow. \n\nCons:\n1. The authors do not learn more than a handful of clusters in each facet, and do not practically demonstrate the feasibility of the approach in a setting where the number of independent clusters blows up. It might be helpful to elaborate more on the shortcomings of Gumbel-softmax based approaches or even compare this approach with one where the discrete variable is learned through the re-parametrization trick.\n\n2. Many details have been offloaded to the appendices. This is not a major issue since the appendices are well written. For example, it might might help to move schematics and related description for the ladder MFCVAE to the main text since this is the version used for all 3 datasets/ to demonstrate the key points in Sec 4.1-4.4.\n\nOther comments:\n - It's not clear how much do results depend on various hyperparameter choices. In particular, number of components $K$ for each facet, dimensionality of $z$, priors on the distribution across components, are likely central to the interpretability/ apparent disentaglement of the models. It would be great if the authors can address this point.\n\n - Majority of the demonstrations rely on qualitative assessment of the reconstructions, for visual datasets where the factors have some semantic meaning (e.g. color, stroke styles, digit identity etc.), and convolutional neural network layers are known to compose features into images. It would be great if the authors could demonstrate these methods for exploratory settings, for example to make predictions on publicly available tabular, biological datasets that come with rich metadata and often a hierarchical structure. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper and appendices include thoughtful notes for each dataset. I also appreciate the transparency in discussing limitations and experimental details!", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "0UkRZB1BWoX", "reviewer": "Reviewer_VMVb", "summary": "The paper proposed Multi-Facet Clustering Variational Autoencoders, which is an unsupervised model and can learn multiple latent characteristics of the data. They showed how to optimize the ELBO in this setting and implemented the model via Variational Ladder Autoencoder architecture with progressive training. Experimental results demonstrate good empirical performance.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Real-world data usually can be clustered in multiple ways. However, most existing deep learning based clustering algorithms are designed to learn only one such partition. This paper aims to learn multiple levels of abstract characteristics that in combination can form different partitions of data. They assume Mixtures of Gaussians prior on the latent space. The paper extends variational deep embedding to this multi-facet setting and shows how to optimize the ELBO. The experiments are done on datasets of MNIST, 3DShapes, and SVHN. While the experiments seem thorough, the performance comparison to other methods was not clearly presented.\n\nWhile the concept seems to be meaningful, it is unclear to me how practical these (independent) facets are. Not every data set clearly has distinct characteristics like handwritten numbers. During implementation, it’s hypothesized that each layer of the VLAE captures distinct aspects of the data. If so, how many layers are needed? Are more layers better? Is it possible that some layers learn combinations of characteristics? More discussions on these aspects will be appreciated.\n\nThe paper is in general well written and the flow is easy to follow. Much of the experiments are postponed to the appendix, making that section difficult to read.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is well-written and the proposed method shows good performance (quantitative results were shown in the rebuttal).  Correcting a flaw in previous work is a plus.  Reviewer's major concerns, including lack of quantitative results, dependence on hyperparameter choice, etc., have been addressed in the rebuttal. 
Decision: Accept (Poster)