Deep amortized clustering
Authors: Juho Lee
Yoonho Lee
Yee Whye Teh
Venue:  (2020)
Keywords: clustering
amortized inference
meta learning
deep learning

Abstract: We propose a \textit{deep amortized clustering} (DAC), a neural architecture which learns to cluster datasets efficiently using a few forward passes. DAC implicitly learns what makes a cluster, how to group data points into clusters, and how to count the number of clusters in datasets. DAC is meta-learned using labelled datasets for training, a process distinct from traditional clustering algorithms which usually require hand-specified prior knowledge about cluster shapes/structures. We empirically show, on both synthetic and image data, that DAC can efficiently and accurately cluster new datasets coming from the same distribution used to generate training datasets. 

Under review as a conference paper at ICLR 2020
DEEP AMORTIZED CLUSTERING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose deep amortized clustering (DAC), neural architecture which learns to cluster datasets efficiently using a few forward passes. DAC implicitly learns what makes cluster, how to group data points into clusters, and how to count the number of clusters in datasets. DAC is meta-learned using labelled datasets for training. process distinct from traditional clustering algorithms which usu- ally require hand-specified prior knowledge about cluster shapes/structures We empirically show, on both synthetic and image data, that DAC can efficiently and accurately cluster new datasets coming from the same distribution used to generate training datasets.
INTRODUCTION
Clustering is fundamental task in unsupervised machine learning to group similar data points into multiple clusters. Aside from its usefulness in many downstream tasks, clustering is important tool for visualising and understanding the underlying structures datasets, well model for categorisation in cognitive science.
Most clustering algorithms have two basic components how to define cluster and how assign data points those clusters. The former is usually defined using metrics to measure distances between data points, using generative models describing the shapes of clusters. The latter, how to assign data points to the clusters, is then typically optimized iteratively w.r.t. objective functions derived based on the cluster definitions. Note that cluster definitions are user defined, and are reflections of the user's prior knowledge about the clustering process, with different definitions leading to different clusterings. However, cluster definitions used in practice are often quite simple, for example clusters in k-means are defined in terms of la distance to centroids, while Gaussians are a commonly used generative model for clusters in mixture models.
Figure 1: Our model identifies one cluster per iteration (top), allowing it find any number of clusters (bottom).
Recently, advances in deep learning has facilitated the approxima- tion of complex functions in a black-box fashion. One particular application of relevance to the problem of clustering in this paper is that of amortized inference (Gershman & Goodman, 2014; Stuhlmüller al., 2013), where neural networks are trained to predict the states of latent variables given observations in a generative model or probabilistic programme. In the context of learning set-input neural networks (Zaheer al., 2017), Lee et al. (2019) showed that it is possible to amortize the iterative clustering process for a Mixture of Gaussians (MoG), while Pakman et al. (2019) demonstrated that it possible to train neural network to sequentially assign data points to clusters. Both approaches can be interpreted as using neural networks for amortized inference cluster assignments and parameters given a dataset. Note that once neural networks are used for amortized clustering, we can take advantage of their flexibility in working with more complex ways to define clusters. Further, the amortization networks can be trained using generated datasets where the ground truth clusterings are known. This can be interpreted as implicitly learning the definition of clusters underlying the training datasets, such that amortized inference (approximately) produces the appropriate clusterings. In sense this shares similar philosophy as Neural Processes (Garnelo et al., 2018 2018b;a), which meta-leam from multiple datasets to learn prior over functions.

---

Under review as a conference paper at ICLR 2020
In this paper, we build on these prior works and propose Deep Amortized Clustering (DAC). As in prior works, the amortization networks in DAC are trained using generated datasets where the ground truth clusterings are known. Like Lee al. (2019), DAC uses Set Transformer, but differs from Lee et al. (2019) in that it generates clusters sequentially, which enables to produce a varying number of clusters depending
the complexity of the dataset (Fig. 1). Our approach also extends Lee et al. (2019) from MoG to problems with more complex cluster definitions, which are arguably harder to hand specify and casier to meta-learn from data. Our work also differs from Pakman et al. (2019) in that our network processes data points in parallel while Pakman et al. (2019) processes them sequentially, which is arguably less scalable and limits applicability to smaller datasets.
This paper is organized as follows. We begin by describing in Section 2 the permutation permutation-invariant set transformer modules that we use throughout the paper. Section 3, we describe how we implement our core idea of identifying one cluster at time, and describe our framework for clustering, the DAC There are several challenges in solving DAC on complex datasets, and we structured our paper roughly in order of difficulty. We apply DAC to clustering synthetic data (Section 5) and image data (Section 6); some settings required additional components, which we describe when needed.
2 A PRIMER ON SET TRANSFORMER AND AMORTIZED CLUSTERING
In this section, we briefly review the set-input neural network architectures to be used in the paper, and describe how Lee al. (2019) used them to solve amortized clustering for MoG.
2.1 SET TRANSFORMER
The Set Transformer (ST) is a permutation-invariiat set-input neural network that uses self-attetntion operations as building blocks. It utilizes multi-head attention (Vaswani al., 2017) for both encoding elements of a set and decoding encoded features into outputs.
The fundamental building block of ST is the Multihead Attention Block (MAB), which takes two sets X = [z1,.... [I1,...,TT]] x.,) and Y = [yn..... Um] and outputs a set of the same size as X. Throughout this article, we represent sets as matrices where each row corresponds to an element. An MAB is defined as
(I) where r-F(()) is feed-forward layer applied row-wise (i.e., for each element). MMA(XX, computes the pairwise interactions between the elements in X and Y with sparse weights obtained from attention. A Self-Attenton Block (SAB) is simply MAB applied to the set itself: SAB(X) e MAB(X, We can model high-orde interactions among the items in a set by stacking multiple SABs; we denote such stack of T. SABs applied set X as SAB, (X).
MA = H + rFF(H) where H = X + rFF (Muttiiheedd/.A Y)).
To summarize a set into a fixed-length representation, ST uses an operation called Pooling by Multi- head Attention (PMA). A PMA is defined as PMAx(X) MAB(S, X) where $ = [$1,..., sk] are trainable parameters.
Note that the time-complexity of SAB is ( because of pairwise computation. To reduce this, Lee et al. (2019) proposed to use Induced Self- Attention Block (ISAB) defined as
ISAB(X) = MAB(X, X))(I,X)),
where 7 = [i1...., fm 1T are trainable inducing points. ISAB indirectly compares the elements of X through the inducing points, reducing the comelextyyyxity to (nm) Similarly to the SAB, we write , (X) to denote a stack of L ISABs.
2.2 AMORTIZED CLUSTERING WITH SET TRANSFORMER
Lee et al. (2019) presented an example using ST for amortized inference for a MoG. dataset X is clustered by maximizing the likelihood of a component MoG, and ST is used to output the parameters as:
Hx = 1SAB, (X), Ho = PMA, (Hx), (logit , 0,), = (F((S
(3)
2

---

Under review as a conference paper at ICLR 2020
where πyy is the mixing coefficient and 8j = (يy of) the mean and variance for the jth Gaussian component. The network is trained to maximize the expected log likelihood over datasets:
E xp())
Σ log
T logN uy p3,
(4)
j=1
where x is the number of elements in X. Clustering is then achieved by picking the highest posterior probability component for each data point under the MoG with parameters output by the ST.
3 DEEP AMORTIZED CLUSTERING
An apparent limitation of the model described Section 2.2 that it assumes a fixed number of clusters generated from Gaussian distributions. In this section, we describe our method to solve DAC in the more realistic scenario of having a variable number of clusters and arbitrarily complex cluster shapes.
3.1 FILTERING: INFERRING ONE CLUSTER AT TIME
The objective (4) is not applicable when the number of clusters is not fixed nor bounded. A remedy to this to build a set-input neural network that identifies the clusters iteratively and make it to learn "when to stop", similar to Adaptive Computation Time (ACT) for RNNs (Graves, 2016).
One may think of several ways implement this idea (we present an illustrative example that simply augments ACT to ST in Section 5.1). Here we propose to train f to solve a simpler task instead of clustering the entire dataset, foeus on finding one cluster at a time. The task, what we call as filtering, is defined as forward pass through f that takes set X and outputs parameter 0 to describe cluster along with a membership probability vector m E [0, 1]nx where x is the number of elements in X. The meaning of the parameter 0 depends on the specific problem. For example, 0 for MoG is (u, )²), the parameters of a Gaussian distribution. m, represents the probability of x; belonging to the cluster described by 9. To filter out the datapoints that belong to the current cluster, use 0.5 as the threshold to discretize m to a boolean mask vector. The resulting smaller dataset is then fed back into the neural network produce the next cluster and so on.
Minimum Loss Filtering Now we describe how to train the filtering network f. Assume has lix true clusters, and let y E [1....,ks xx be cluster label vector corresponding to the true clustering of X. Then we define the loss function for one filtering iteration producing one 0 and one m as
nx
C(X, y, m. 0)
min
BCE(M; (
Σ logp(z;:0))
(5)
kx
Tix
TIx
where nxx Di= (v==)), BCE(.,) :) the binary cross-entropy loss, and p(z: f) is the density of x under cluster parameterised by This loss encourages to describe the data distribution of cluster, and to specify which datapoints belong this particular cluster. The rationale to take minimum across the clusters is follows. One way to train I to pick a cluster at each iteration is to impose an ordering on the clusters (e.g. order of appearance in some arbitrary indexing of X, or in order of distance to origin), and to train if to follow this order. However, this may introduce unnecessary inductive biases that deteriorates learning. Instead, we let f find the easiest one to identify, thus promoting f to learn its own search strategy. Note that there are kx! equally valid ways to label the clusters in X. This combinatorial explosion makes learning with standard supervised learning objectives for y tricky, but our loss (5) is inherently free from this problem while being invariant to the labelling of clusters.
We use the following architecture for the filtering network f: Section 2:
encode data:
Hx = (X),
decode cluster:
Ho PMAI (H+),
0 rFF((Ht),
(6) The network first encodes X into Hx and extracts cluster parameters 9. Then 9 together with encoded data Hx are further processed to produce the membership probabilities m. We call the filtering network with architecture (6) and trained with objective (5) Minimum Loss Filtering (MLF).
decode mask:
Hm = ISAB (MAB((Xx Ho)).
m = sigmoid((FF((Hm))
3

---

Under review as a conference paper at ICLR 2020
Anchored Filtering An alternative strategy that we found beneficial for harder datasets is to anchor points. Given a dataset X and labels y constructed from the true clustering, we sample an anchor point with index E {I...., Rx f uniformly from X. We parameterize set-input network to take both and a is input, and to output the cluster that contains the anchor point Ia- The corresponding loss function
1 Σ BCE((, nx
C(z, U. a, m,
(m==ia))
log p(a;; f):
Ti alpa
where ja denotes the the true cluster index containing a. The architecture to be trained with this loss can be implemented
encode data:
Hx IS)B,,X),,
Hx MA ha),
decode cluster:
Hg PMA1(HX
0 8= IFF (Ho), sigmoid((PP((H)))
decode mask: Hm (MAB(HXXa Ho)).
(8)
where ha is the row vector of Hx corresponding to the index a. We train (8) by randomly sampling for each step, and thus promoting f to find clusters by comparing each data point the random anchor point. Note that the loss is also free from the label order ambiguity given anchor points. We call this filtering strategy Anchored Filtering (AF).
3.2 BEYOND SIMPLE PARAMETRIC FAMILIES
When each cluster cannot be well described by Gaussian or other simple parametric distributions, we have several choices to learn them. The first is to estimate the densities along with the filtering using neural density estimators such as Masked Autoregressive Flow (MAF) (Papamakarios et al., 2017). Another option is to lower-bound log p(x; 0) by introducing variational distributions, for example using Variational Autoencoder (VAE) (Kingma & Welling, 2014). See Section 5.2 and Section 6.1 for examples. If the density estimation is not necessary, log p(r; f). In other words, instead (5) and (7), we train
choose not to learn
n.
RN
C(X, y. m, 0) = min Σ BCE(M, 1 (m==))) 2(X, = Σ BCB(m, (m)=sa)) (9)
for MLF and AF, respectively. The corresponding architectures are (6) and (8) with parameter estimation branches removed. The DAC trained in this way implicitly learns how to define cluster from the given training datasets and cluster labels. See Section 6.2 and Section 6.3 where we applied this to cluster image datasets.
3.3 DEEP AMORTIZED CLUSTERING
Recall that each step of filtering yields one cluster. To solve DAC, we iterate this procedure until all clusters are found. After each filtering step, we remove from the dataset the points that were assigned to the cluster, and perform filtering again. This recursive procedure is repeated until all datapoints have been assigned to a cluster'. We call the resulting amortized clustering algorithm as DAC DAC learns both data generating distributions and cluster assignment distributions from meta-training datasets without explicit sand-engineering.
4 RELATED WORKS
Deep clustering methods There is a growing interest in developing clustering methods using deep networks for complex data (Yang al., 2016a;b; Xie al., 2015; Li al., 2017; Ji al.. 2018). See Aljalbout et al. (2018) for comprehensive survey on this line of work. The main
In practice, input the entire dataset along with and assign zero attention weight datapoints with
m, = 1. This equivalent to described scheme, hut has the added benefit of heing easy to parallelize across multiple datasets.
Note that DAC with ML.F is stochastic once discretize the memhership probability.
4

---

Under review as a conference paper at ICLR 2020
focus of these methods is to learn a representation of input data amenable to clustering via deep neural networks. Learning representations and assigning data points to the clusters are usually trained alternatively. However, like the traditional clustering algorithms, these methods aim to cluster particular datasets. Since such methods typically learn a data representation using deep neural networks, the representation is prone to overfitting when applied to small datasets.
Learning to Cluster Learning to cluster refers to the task of learning a clustering algorithm from data. Such methods are trained in a set of source datasets and tested on unseen target datasets. Constrained Clustering Networks (Hsu et al., 2017; 2019) follow a two-step process for learning to cluster: they first Iearn a similarity metric that predicts whether a given pair of datapoints belong to the same class, and then optimize neural network to predict assignments that agree with the similarity metric. Centroid Networks (Huang et al., 2019) learn an embedding which is clustered with the Sinkhorn K-means algorithm. While these methods combine deep networks with an iterative clustering algorithm,, our framework is much more efficient as it directly identifies each cluster after one forward pass. Our experiments in Section that our model is orders of magnitude faster than previous works in learning to cluster.
Amortized clustering methods To the best of our knowledge, the only works that consider a similar task to ours is Lee et al. (2019) and Pakman et al. (2019). We refer the reader back to Section 2 for an outline of the amortized clustering framework presented in Lee et al. (2019). Pakman et al. (2019) presented amortized clustering method called Neural Clustering Process (NCP). Given dataset, NCP sequentially computes the conditional probability of assigning the current data point to one of already constructed clusters algorithm for Dirichlet process mixture models (Neal, 2000), but without positing particular priors on partitions, but rather letting the network learn from data. However, the sequential sampling procedure makes the algorithm not parallizable using modern GPUs, limiting its scalability. Furthermore, since the clustering results vary lot w.r.t the sequential processing order, the algorithm needs a sufficient number of random samples to get stable clustering results. We compared our method to NCP on small-scale MoG experiments in Appendix C, and our results support our claim.
one, similar in spirit to the popular Gibbs sampling
EXPERIMENTS ON SYNTHETIC DATASETS
5.1 2D MIXTURE OF GAUSSIANS
We first demonstrate DAC with MLF on 2D MoG datasets with arbitrary number of clusters. We considered two baselines that can handle variable number of clusters: truncated Variational Bayesian Dirichlet Process Mixture Model DVBDPM) (Blei & Jordan, 2006) and the ST architecture (Lee et al., 2019) with ACT-style decoder so that it can produce arbitrary number of clusters. We describe the latter method, ACT-ST, in detail in Appendix A. See Appendix B for detailed experimental setup including data generation process and training scheme. We trained DAC and ACT-ST using random datasets with a random number of data points S Tmmax and clusters S kmux. where we set (Rmax, Ammxs ) = (1000, 4) during training. We tested the resulting model on two scenarios. The first one is to test one same configurations; testing on 1,000 random datasets with (7maax kimax) = (1000, 4). The second one is test on 1,000 random datasets with (7mmxx Amax) (3000, 12) to see whether the amortized clustering methods can generalize to an unseen number of clusters. For both scenarios, we VBDPM each test dataset until convergence from scratch. We used the mean of the variational distributions as the point-estimates of the parameters to compute ligeliiklssoods. Table summarizes the results. DAC works well for both cases, even beating the oracle log-likelihood computed from the true parameters, while ACT-ST fails to work in the more challenging (Amax kimax) = (3000, 12) case (Fig. 2). VBDPM works well for both, but it takes considerabl time converge whereas DAC requires no such optimization to cluster test datasets.
5.2 2D MIXTURE OF WARPED GAUSSIANS
When the parametric form of cluster distribution is not known, we cannot directly compute the log p(x;; 6) term in (5). In this case, we propose to estimate the density p(z; 0) via neural density estimators along with the DAC learning framework. We construct each cluster by first sampling points from 2D unit Gaussian distribution, and then applying random nonlinear transformation on
5

---

Under review as a conference paper at ICLR 2020
Figure 2: (Left) Clustering by ACT-ST, (middle) one step of filtering, (right) clustering by DAC.
Table I: Results on synthetic 2D MoG. We report log-likelihood (LL), clustering accuracies (adjusted Rand index (ARI) (Hubert & Arabie, 1985) and normalized mutual information (NMI), the mean absolute eπror between true lc and estimated k and processing time per dataset. The numbers below (Tmmxx Krmax) are oracle LL values computed by the true parameters, We report the average on runs.
(Teaxxs Kmax)
Algorithm
LL
ARI
NMI
k-MAE
Time [sec]
VBMOG
-0.719 = 0.002
0.971 1: 0.001
0.977 0.001
0.079 +:2 0.003
0.037 1: 0.001
(1000,4)
ACT-ST
-0.721 = 0.008
0,974 + 0.002
0.974
4: 0.001
0.044 +:t 0.003
0.006 + 0.000
-0.693
DAC
-0.692
0.002
0.983 1: 0.001
0.978
0.001
0.120 +: 0.009
0.008 1: 0.000
VBMOG
-1.561
0.001
0.962 +:2 0.000
0.970
0.000
0.435 + 0.010
0,400 +: 0.006
(3000,12) 12)
ACT-ST
-5.278
0.573
0.781 0.008
0.855
0.004
1.993
0.082
0.024 + 0.001
-1.527
DAC
-1.544
0.971 + 0.000
0.974
0.279 0.012 0.021 +
each point. We use MAF (Papamakarios et al., 2017) model p(1; 0) where 0 is context vector that summarizes the information about a cluster. We trained the resulting DAC with MLF using random datasets, and compared to spectral clustering (Shi & Malik, 2000). As shown in Fig. DAC finds and estimates the densities of these nonlinear clusters. See Appendix D for more details and results.
EXPERIMENTS ON REAL DATASETS
6.1 CLUSTERING EMNIST WITH MIXTURE OF NEURAL STATISTICIANS
We may approximate the likelihood p(2; 0) via a VAE (Kingma & Welling, 2014) when the data distribution is too high-dimensional or complex. Instead of directly maximizing the log- lielikkiihood, we maximize a lower-bound on the likelihood, log p(:; 0) 2 [x [log p(x, 2; 0) q(2[2;; 0),, where 8 encodes the context of a cluster and is latent variable that describes based on 6. Neural Statistician (NS) (Edwards & Storkey, 2016) proposed the idea of approximating p(2; 0) using context 0 produced by a network;
thus call this model mixture of NSs. We found that the DAC implemented in this way could cluster well, generalizes to an unseen number of clusters, and generate images conditioned on the cluster context See Appendix E for detailed results.
25.
25
20
Figure 3: Clustering warped Gaussian data with DAC. The model was trained with & E {l....,44 clusters (left) but generalizes to datasets with 12 clusters (right).

---

Under review as a conference paper at ICLR 2020
Table 2: Results for 1,000 test datasets sampled from Embedded ImageNet.
ARI
NMI
k-MAB
Time (scc)
k-means
0.370
0.001
0,514 +: 0.001
0.188 0.003
Spectral (Shi & Malik, 2000)
0.432 0.000
0.568 0.000
0.087
DEC (Xic al., 2015)
0.195
0.326
46.098
KCL (Hsu et al., 2017)
0.201
0,361
13.401
MCL (Hsu et al., 2019)
0.157
0,350
14.646
DACMLE
0.400 0.012
0.527 1. 0.013
2.103 1: 0.160
0.012 0.001
DACAr
0.451 0.014
0.579 0.013
1.805 1: 0.031
0.017
Table 3: Unsupervised cross-task transfer learning on Omniglot. Normalized mutual information (higher is better) is averaged across 20 alphabets (datasets), each of which have between 20 and 47 letters (classes). All values beside DAC were reported Hsu et al. (2019). "k given" means that the true number of clusters given to the model.
Method
NMI(k given)
NMI
K-means (MacQueen et al., 1967)
0.353
0.464
LPNMF (Cai et al., 2009)
0.372
0.498
LSC (Chen & Cai, 2011)
0.376
0.500
CSP (Wang et al., 2014)
0,812
0,812
MPCK-means (Bilenko et al., 2004)
0.871
0.816
KCL (Hsu et al., 2017)
0.889
0.874
MCL (Hsu al., 2019)
0.897
0.893
DACAF (ours)
n/a
0.829
6.2 CLUSTERING EMBEDDED IMAGENET
We applied DAC to cluster the collection of minilmageNet (Vinyals al., 2016) and tieredIma- geNet (Ren et al., 2018). We gathered pretrained 640 dimensional features the images released by Rusu et al. (2018)². We used the training and validation features as training set and test features as test set. The resulting training set contains 620,000 samples from 495 classes, and the test set contains 218,000 samples from 176 classes, with no overlap between training and test classes.
We trained DAC without density estimations (9), using both MLF and AF. We sampled randomly clustered datasets from the training set having (Amax kimax) (100, 4). We then generated 1,000 randomly clustered datasets from the test set with (Nomax, kmmx) (300. 12). We compared DAC to basic clustering algorithms (k-means, spectral clustering), Deep Embedding Clustering (DEC) (Xie et al., 2015), and transfer learning methods (KCL (Hsu et al., 2017) and MCL (Hsu al., 2019)). k-means, spectral, and DEC were trained for each test dataset from scratch. For KCL and MCL, we first trained similarity prediction network using the training set, and used it to cluster each test dataset. For DEC, KCL and MCL, we used fully-connected layers with 256 hidden units and layers for both similarity prediction and clustering network. Note that the size of the test datasets are small mm = 300 < 640), so one can easily predict that DEC, KCL., and MCL would overfit. The algorithms other than ours was given the true number of clusters. The results summarized in Table 2. It turns out that the pretrained features are good enough for the basic clustering algorithms show decent performance. The deep learning based methods failed to learn useful representations due to the small dataset sizes. Ours showed the best clustering accuracies while also consuming the shortest computation time.
6.3 CLUSTERING OMNIGLOT IMAGES
We apply our filtering architecture to the unsupervised cross-task transfer learning benchmark of Hsu et al. (2017; 2019). This benchmark uses the Omniglot dataset (Lake et al., 2015) to measure how well a clustering method can generalize to unseen classes. The Omniglot dataset consists of
https gi thub. wtt. comm/demmiinn nd/1eo.

---

Under review as a conference paper at ICLR 2020
Table Mean absolute error of cluster number (k) estimate and processing time per dataset on the Omniglot benchmark.
k-MAE
Time [sec]
KCL
MCL
DAC
KCL
MCL
DACAF
6.4
5.1
2.
129.3
18.9
124.5 + 14.4
4.3
handwritten characters from 50 different alphabets. Each alphabet consists of several characters, and each character has 20 images drawn by different people. Our problem setup consists of training clustering model using images from the 30 background alphabets, and using each of the 20 alphabets on. We use the same VGG network backbone as Hsu al. (2019) and
as seperate dataset follow their experimental setup.
We show the normalized mutual information (NMI) of DAC along with other methods in Table While previous methods were also evaluated on the easier setting where the true number of clusters is given to the network, this setting is not applicable to DAC. We see that DAC competitive with the state-of-the-an on this challenging task despite requiring orders of magnitude less computation time. While the metrics for DAC were computed after at most 100 forward passes, previous methods all require some sort of iterative optimization. For example, KCL and MCL (Hsu et al., 2017; 2019) 9) required more than 100 epochs of training, for each alphabel to arrive at the cluster assignments in Table 3. This difference in computation requirements is clearly demonstrated in Table 4: DAC requires an average of less than 5 seconds per alphabet whereas KCL and MCL required more than 100. Table 4 additionally shows that in addition being extremely time-efficient, DAC was more accurate in estimating k. This demonstrates the efficacy of our overall structure of identifying one cluster at time.
7 DISCUSSION AND FUTURE WORK
We have proposed DAC, an approach to amortized clustering using set-input neural networks. DAC learns to cluster from data, without the need for specifying the number of clusters or the data generating distribution. It clusters datasets efficiently, using a few forward passes of the dataset through the network.
There are number of interesting directions for future research. The clustering results produced by DAC almost deterministic because we discretise the membership probabilities in the filtering process. It would be interesting to take into account uncertainties in cluster assignments. In the imagenet experiment, we found that we needed a sufficient number of training classes to make DAC generalize to unseen test classes. Training DAC to generalize to unseen image classes with smaller numbers of training classes seem to be a challenging problem. Finally, learning DAC along with state-of-the-art density estimation techniques for images in each cluster is also promising research direction.
REFERENCES
E. Aljalbout, V. Golkov, Y. Siddiqui, M. Strobel, and D. Cremers. Clustering with deep learning: taxonomy and new methods. arXiv: 2018.
Mikhail Bilenko, Sugato Basu, and Raymond Mooney. Integrating constraints and metrie learning in semi-supervised clustering. In Proceedings of the twenty-first international conference on Machine learning, pp. 11. ACM, 2004.
D. M. Blei and M. Jordan. Variational inference for Dirichlet process mixtures. Bayesian Analysis, 1())1221-43,, 2006.
Deng Cai, Xiaofei He, Xuanhui Wang, Hujun Bao, and Jiawei Han. Locality preserving nonnegative matrix factorization. In Twenty- First International Joint Conference on Artificial Intelligence, 2009.
8

---

Under review as a conference paper at ICLR 2020
X. Chen, D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, I. Sutskever, and P. Abbeel. Variational lossy autoencoder. In Proceedings of International Conferences on Learning Represen- tations, 2017.
Xinlei Chen and Deng Cai. Large scale spectral clustering with landmark- representation. In Twenty-Fitt Fiith AAAI Conference on Artificial Intelligence, 2011.
G. Cohen, S. Afshar, J. Tapson, and A. van Schaik. EMNIST: an extension of MNIST to handwritten letters. arXiv preprint arXiv: 172.0070,.. 2017.
H. Edwards and A. Storkey. Towards neural statistician. In Proceedings of International Conference on Learning Representations, 2016.
M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, W. Teh, D. J. Rezende, and S. M Ali Eslami. Conditional neural processes. In Proceedings of International Conference on Machine Learning, 2018a.
M. Garnelo, J. Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. M. Ali Eslami, and Y. W. Teh. Neural processes. ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018b.
M. Germain, K. Gregor, I. Murray, and H. Larochelle. MADE: masked autoencoder for density estimation. In Proceedings of International Conference on Machine Learning. 2015.
S. Gershman and N. D. Goodman. Amortized inference in probabilistic reasoning. In Proceedings of Annual Conference of the Cognitive Science Society, 2014.
A. Graves. Adaptive computation time for recurrent neural networks. arXiv:1603.08983, 2016.
arXiv preprint
Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning cluster in order transfer across domains and tasks. arXiv: 10125 [cs]. November 2017. arXiv: 1711.10125.
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class Classifica- tion without Multi-class Labels. 1XXiv:1901.0054 [cs, stat], January 2019. arXiv: 1901.00544.
Gabriel Huang, Hugo Larochelle, and Simon Lacoste-Julien. Centroid Networks for Few-Shot Clustering and Unsupervised Few-Shot Classification. arXiv:1902.08605 [cs, stat], February 2019. arXiv: 1902.08605.
I. Hubert and P. Arabie. Comparing partitions. Journal of Classification, 2:193-218, 1985.
X. Ji, J. F. Henriques, and A Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. 107.v2180..06633, 2018.
D. P. Kingma and J. L. Ba. Adam: a method for stochastic optimization. In Proceedings of International Conference on Learning Representations, 2015.
D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In Proceedings of International Conference Learning Representations, 2014.
Brenden Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 13326266) 2015.
J. Lee, Y. Lee, J. Kim, A. R. Kosiorck, S. Choi, and Y. W. Teh. Set transformer: framework for attention based connutttiooninvariinn neural networks. In Proceedings of International Conference Machine Learning, 2019.
Li. H. Qiao, Zhang, and X. Xi. Discriminatively boosted image clustering with fully convolu- tional auto-encoders. arXiv: 7703.07890, 2017.
James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pp. 281-297. Oakland, CA, USA, 1967.

---

Under review as a conference paper at ICLR 2020
R. M. Neal. Markov chain sampling methods for Dirchlet process mixture models. Jorunal of Computational and Graphical Statistics, 9(2):249- 265, 2000.
A. Pakman, Y. Wang, C. Mitelut, J. Lee, and L. Paninski. Discrete neural processes. arXiv:1901.00499, 2019.
G. Papamakarios, T. Pavlakou, and Murray Masked autoregressive flow. In Advances Neural Information Processing Systems, 2017.
M. Ren, S. Ravi, E. Triantafillou, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel. Meta-learning for semi-supervised few-shot classification. In Proceedings of International Conference Learning Representaitons, 2018.
A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta- learning with latent embedding optimization. In Proceedings of International Conference on Learning Representations, 2018.
Shi and Malik. Normalized cuts and image segmentation. IEEE Transactions on Putlern Analysis and Machine Intelligence, 22(8), 2000.
A. Stuhlmüller, J. Taylor, and N. D. Goodman. Learning stochastic inverses. In Advances in Neural Information Processing Systems, 2013.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. In Advances in Neural Information Processing Systems, 2017.
O. Vinyals, C. Blundell, Lillicrap, K. Kavukcuoglu, and D. Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, 2016.
Xiang Wang. Buyue Qian, and Ian Davidson. On constrained spectral clustering and its applications. Data Mining and Knowledge Discovery, 28( 28(1):1-30, 2014.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised Deep Embedding for Clustering Analysis. arXiv: 1511.06333 [cs], November 2015. arXiv: 1.06355..
B. Yang, X. Fu, N. D. Sidiropoulos, and M. Hong. Towards k-means -menns-friendly spaces: simultaneous deep learning and clustering. arXiv:1610.00 0.4944, 2016a.
J. Yang. D. Parikh, and D. Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016b.
M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. Salakhutdinov, and A. Smola. Deep sets. In Advances in Neural Information Processing Systems, 2017.
10

---

Under review as a conference paper at ICLR 2020
A
DETAILED DESCRIPTION OF ACT-ST
We first define an adaptive-computationn-in version of PMA,
aPMA(X, k) = MM. sk] .X), $j = PMMM ((. for j = ......
(10)
which enables RNN-like iterative computation by sequentially extending the parameters for PMA.
The clustering network to output variable number of parameters is then defined as
Hx = (X),
g th) = (aBAA k)),
0k sigmoid (meanoorddeea [:, [;)))))), 5k
II jSk
Uks
(logit , - [: 2:1);
(11)
where [:, 1] and [ : are numpy-like notation indexing the columns. 9k is a "stop" variable where 8 > 0.5 means the iteration stops at kth step and continues otherwise.
During training, we utilize the true number of clusters as supervision for training cs. yielding the overall loss function
XXX Σ 1
kine
kmna BCE(Ck, Il ((ockmmet)
Ep(X Rutel
log
(huue) p(1s; Oftene)) T
(12)
1=1
where kirne is the true number of clusters, kmax 2 Kirue maximum number of steps run.
B DETAILS OF MoG EXPERIMENTS
We generated dataset by the following process.
(0.if(0.3mmmm Rmax), 1 Binomial(Hmax - 1, 0.5),
Dir(a
( Lid. Cat(T))
(n,))=1 i.d. Normal((0, T, ,9T,, (a,) iid log (o 0..017),
x; Normal(Py, diag(o for I.
(13)
Both ACT-ST and filtering networks were trained with mmax = 1, 0.0000 and kimax = 4. For each step of training, sampled batch of 100 datasets (sharing the same Unif mmxx) to comprise a tensor of shape 100 x n X 2), and computed the stochastic gradient to update parameters. We trained the networks for 20,000 steps using ADAM optimizer (Kingma & Ba, 2015) with initial learning rate 5 x 10-4. The results in Table are obtained by testing the trained models on randomly generated 1,000 datasets with the same generative process.

---

Under review as a conference paper at ICLR 2020
Figure 4: More clustering results. Top two rows shows ACT-ST applied the datasets having {1,....4) The middle two rows show the same ACT-ST model applied to datasets having k > 4. The bottom two show iterative filtering applied the same datasets.
12

---

Under review as a conference paper at ICLR 2020
Table 5: Comparison of iterative filtering and NCP on 100 random datasets with (Amaax kmax)
(3000, 12). The oracle oog-likelhood is 1.5309, and DAC recorded -1.5640. 5 is the number of samples per dataset used for NCP.
ARI
k-MAE
Time [sec]
DAC
0.9616
0.2800
0.0208
NCP (S =)
0.7947
1.6333
4.1435
NCP (S 10)
0.8955
0.8000
5.8920
NCP (S 50)
0.9098
0.6444
6.7936
C COMPARISON TO NEURAL CLUSTERING PROCESS
We compare DAC to NCP. Due to the sequential nature, the training procedure NCP does not scale to the other experiments we conducted. Instead, we trained NCP for MoG data described in Appendix B with smaller scale having maxa = 100 and kmax = 4. We used the code released by the authors* with default hyperparameters. We measured the clustering performance for 100 random datasets generated with mmaa kmux) = (3000, 12) (Table 5). Both method generalized well w.r.t. the number of data points but filtering did much better in generalizing for the number of clusters k. The performance of NCP for clustering depends heavily on the processing order, so we conducted multiple runs with different random orders and picked the best computed from NCP. Ours outperformed NCP with S = 50 samples per dataset with at least two orders of magnitude faster processing time.
the clustering probability
D DETAILS OF MIXTURE OF MAFS AND WARPED GAUSSIAN EXPERIMENTS
We model the cluster density MAF.
log po;;((;; 0) log)(>))
log 10:1(1)(011111000
log Normal (z) |0, 1) +
Σ
log 0)
d
log Normal On, 44)
(14)
where
U1 = BT, U; T; (2 0)
(15)
o (2 0)
We can efficiently implement this with MADE (Germain al., 2015).
13

---

Under review as a conference paper at ICLR 2020
Table 6: Results on warped Gaussian datasets.
(meax kmax)
Algorithm
LL
ARI
NMI
Á-MAE
Time
[scc]
Spectral
0.845
0.033 0.889 + 0,002
0.103 0.000
(1000,4)
DAC
-1.275 0.015
0.974
0.970 +:2 0.001
0.320 0.035
0.011
(3000,12)
Spectral
0.592
0.001
0.766 1: 0.001
0.572 0.003
DAC
-2.436 1: 0.029 0.923
0.936 0.001 1.345 0.099
0.037
We generated the warped Gaussian datasets by the following generative process.
Unif(0.33 ma. 7max),
Binomial 1, 0.5),
π Dir(a [1, 1),, (u:) Ltd. Cat(T)
F ~ Mo)r 0.8
(az))=1 iid. Normal(O, v2), (b)) Lid. Normal V2)
8: Gy cosT; 0.1 by coST;
l by: sin + 0.1 Oy sinr;
+ #
V / + b2
i.d. ~ (0))
Unif(O, 2m), R; =
cos Py.
- sin 2y: cos Py:
sin Py.
(A1) i.d. Normal(min(k; 4.0][1, 11 I),
i Ra(si, 4." + Aua
(16)
where MoGi(y) (y) denotes the sampling from 1d Mixture of Gaussians with the same parameter distributions as (13).
The filtering network is constructed as (6) where 0 is 128 dimensional vector to be fed into MAF as context vector for a cluster. We implemented log p(z; 8) as a 4 blocks of MAF with MADE (Germain et al., 2015). The filtering network
trained using random datasets with Ammax 1, 000 and kmax = 4, and trained for 20,000 steps with ADAM optimizer. We set initial learning rate as 10 4 Table 6 compares the resulting DAC to spectral clustering (Shi & Malik, 2000). Spectral clustering was ran for each dataset from scratch with true number of clusters given. To give better idea how good is the estimated likeliool values, we trained MAF with same structure (4 blocks of MADE) but without mixture component and cluster context vectors for 100 random test datasets. We trained for 20,000 steps for each dataset using ADAM optimizer with learning rate 10-4, The log- ligliike values estimated with filtering on the same datasets outperforms the one obtained by exhaustively training MAF for each dataset, and got -2.570 for MAF and -2.408 for DAC. This shows that the amortized density estimation works really well.
14

---

Under review as a conference paper at ICLR 2020
C
354
6
Figure 5: More clustering results for warped Gaussian datasets.
15

---

Under review as a conference paper at ICLR 2020
E DETAILS OF MIXTURE OF NEURAL STATISTICIANS AND EMNIST EXPERIMENTS
E.I MIXTURE NEURAL STATISTICIANS
Let X = [21.... nnl be an input set. We construct filtering network as follows.
hene, Encoder(x;) for i = l...,,.., n Hx I ((hene,, ................................ henc, 1),
Hg = (Hx), B = =FFF(Ho),
24 ~ ((2))( hene, 0) for = 1,...,
Rdec,i Decoder(2;, 0) for i = I,..., 2; p(z; (hdec;) for i (m)).
Hm = ISAB, (MAB Hx. Ho)), m
(17)
The log- lig-liielihood for particular cluster Mog(((an (arly: ys = 5}: 0) is then approximate by the variational lower-bound,
(eai/hene,s 8) log p(a, (haeee de,
logp((=;11) = =}:00]
Σ
g(2s/Menc.t 0)
=|
Σ
KLlg(=i|Mennn.)) )llp(z))) (18)
Apps
Once the clustering is done, the likelihood of the whole dataset X can be lower-bounded
log p(:; (... 0k) 2 Ty logp({z/1) (};
(19)
The likelihood lower- bounding for each cluster corresponds to the NS except for that the context vector is constructed with ISAB and PMA, we call this model mixture of NSs.
E.2 EXPERIMENTS ON EMNIST DATA
We trained the model described in (17) to EMNIST (Cohen et al., 2017). We picked "balanced" split, with 47 class / 112,800 training images 880 test images. At each training step. we generated 10 randomly clustered dataset with (Nomax, )max) = (1000, 4) and trained the network by the loss function (5) with the log- likelihoeoood part replaced by (18). For Encoder(ar; and Decoder(z;, 0), we used three layers of multilayer perceptrons. For the variational distribution q(2g/fecee, 0), used Gaussian distribution with parameters constructed by a fully-connecter layer taking [henee 0] as inputs. Following Chen et al. (2017), We used an autoregressive prior distribution constructed by MAF for p(=). For the likelihood distribution, p( hace.i) we used Bernoulli distribution. Each training image stochastically binarized.
Table 7 shows that iterative filtering can decently cluster EMNIST. Fig. 6 shows that given a set of 100 images with 4 clusters, the filtering can correctly identify cluster and learns the generative model to describe it. Fig. shows the clustering results using iterative filtering for 100 images.
Table 7: The results on EMNIST.
(Aaax kmax)
LL/pixel
ARI
k-MAE
(1000,4)
-0.193 0.002 0.887 + 0.005
0.607 1 0.093
(3000,12)
-0.199 0.003 0.728 + 0.019
1,668 + 0.147
F MORE RESULTS FOR OMNIGLOT EXPERIMENTS
We present more detailed comparion of KCL, MCL and ours for each alphabet.
16

---

Under review as a conference paper at ICLR 2020
Table 8: Absolute error of cluster number (k) estimate and time per dataset on the Omniglot benchmark. We show averages the bottom row. Lower better for both metrics.
Absolute Error of k
Time (s)
Alphabet
KCL
MCL
DACAF
KCL
MCL
DACAF
Angelic
20
2
110.6
102.5
3.3
Atemayar Q
26
8
0
5
116.2
115.0
3.5
Atlantean
26
15
10
115.5
112.5
4.1
Aurek Besh
26
2
4
190.2
113.2
2.9
Avesta
26
3
116.3
115.9
3.9
Ge_ez
26
6
116.4
112.8
4.2
Glagolitic
45
0
9
140.4
141.2
4.6
Gurmukhi
45
143.1
144.8
5.1
Kannada
41
11
137.4
138.8
4.5
Keble
26
3
114.4
112.0
3.1
Malayalam
47
0
12
8
146.6
148.3
4.5
Manipuri
40
134.9
135.7
4.3
Mongolian
30
6
122.0
119.6
4.1
Old Church
140.1
142.4
Oriya
46
14
9
144.6
143.9
5,4
Sylheti
28
22
2
117.3
117.2
4.8
Syriac Syriac_Serto
Serto
23
15
1
4
112.5
118.0
4.1
Tengwar
25
16
1
113.0
110.6
4.7
Tibetan
42
0
8
3
139.9
135.7
5.1
ULOG
26
14
114.5
110.8
4.2
Average
6.4
5.1
4.6
129.3
124.3
4.3
17

---

Under feview as a conference paper at ICLR 2020 -
Figure 6. A step filtering for 100 EMNIST images with 4 clusters. (Left) images helong to the cluster identified by the filtering step. (Middle) Images generated by decoding random latent vectors z ~ p(=) passed through the decoder, with cluster context vector extracted from the filtering step. (Right) Images do not belong to the cluster identified the filtering step.
#EHHOP THOHEN H H
H
HEHHM TMMES
WHHHE H LEHOOHER
# HENHE
duster PPPFPerP PpicppD
18

---

Under review as a conference paper at ICLR 2020
3383333303
33333
OD2000D000 D DADDD
Figure 7: Clustering results of 100 EMNIST images with 4 ground-trus truth clusters by iterative filtering. Each block corresponds to a cluster.
19


Reviews:
[{"review_id": "r1gwMlFp_S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a deep amortized clustering framework which learns to cluster data efficiently  based on the combination of set transformer and amortized clustering.    The main motivation is to learn clustering rules from labeled data sets and generalize to new data sets, so as to avoid manually defining clustering criterion. \n\nNot an expert in this domain, I feel that the paper is not easy to read and many intuitions readers might be interested in are not explained very well. For example, the authors mentioned that the method proceeds sequentially, and each step identifies one cluster (the easiest cluster). However, in practical situations clusters might be meaningful only when considered in a global context, and / or under a certain scale, and more discussions are needed on how the proposed method achieves these goals.  Also, how the set transformer extracts  information useful for clustering is very unclear and needs more elaborations. \n\nThe authors used anchor points in harder problems, where the anchor points are uniformly sampled from the input data. One concern is that random sampling may lead to fluctuations in the learning process as well as very close anchor points which can be harmful for clustering. \n\nThe visualization of identified clusters seems a bit misleading. Some very compact clusters seem to be split into halfs (or with fragments of different colors) and does this indicate failed clustering on these simple data sets?\n\nFinally, whether useful rules can be learned for clustering from labeled data is still quite open and authors may want to give some convincing examples of such rules'' for which existing clustering criterion will fail but with learned rules it can be resolved. It looks to be that the result has to do with the clustering structures of the labeled data and how can one be sure that the training data have a similar clustering structure with the to-be-clustered-data? Without answering this basic concerns, the proposed method may be hard to be accepted. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HygfUNzxcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper presents an amortized clustering method, called DAC, which is a neural architecture that allows efficient data clustering using a few forward passes. The proposed method is essentially based on the idea behind set-input neural networks [1], which consists of modeling the interaction between instances within a given dataset. Compared with the previous work [1], the main difference is that DAC does not need to specify the number of clusters, as in the case of Bayesian nonparametrics, making it more flexible for clustering complex datasets. It is empirically shown that DAC can efficiently and accurately cluster new datasets coming from the same distribution for both synthetic and image data.\n\nStrengths:\nOverall, I think the paper is well written and the relationship to previous works is well described. The empirical results seem promising, especially in terms of computational efficiency. The authors conduct some experiments on relatively large datasets, such as miniImageNet and tiereImageNet, which is indeed crucial for the practical applications of the proposed model.\n\nWeaknesses:\n- I think this is a good paper, but my major concern is the limited theoretical contribution, given the fact that this work is mainly based on set-input neural networks introduced in Ref. [1]. I would like the authors to clarify a bit more the novelty of the paper.\n- The authors claim that DAC can process data points in parallel while Ref. [2] uses a sequential sampling procedure. However, there does not seem to be sufficient details on how to parallelize the proposed algorithm.\n- As shown in Figs. 1 and 4, it seems that some clusters are split into two or three fragments. I think this simply means the failure of the proposed method on synthetic data.\n- As also mentioned in the discussion on page 8, it would be important to consider uncertainties in cluster assignments, as already done in Ref. [2]. I would recommend the authors to provide some insight on how to take the cluster assignment uncertainty into account within current model.\n\nAt the moment, I recommend a weak reject as the technical contribution of the paper seems rather limited, but I could be open to increasing my score if my concerns are addressed.\n\nReferences:\n[1] J. Lee, Y. Lee, J. Kim, A. R. Kosiorek, S. Choi, and Y. W. Teh. Set transformer: a framework for attention-based permutation-invariant neural networks. In Proceedings of International Conference on Machine Learning, 2019.\n[2] A. Pakman, Y. Wang, C. Mitelut, J. Lee, and L. Paninski. Discrete neural processes. ArXiv:1901.00409, 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkersYSG9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Overview]\n\nIn this paper, the authors proposed a new clustering method called deep amortized clustering (DAC). Inspired by Lee et al 2019, the authors exploited a transformer to gather the contextual information across different dataset points and then predict the cluster label for each data point. The main difference from Lee et al is that the proposed DAC sequentially estimate the cluster labels for the data points and thus more flexible to estimate the number of clusters in the whole dataset. Based on the proposed DAC method, the authors evaluated the performance on both unsupervised clustering and supervised clustering tasks. it turns out the proposed method has achieved better or comparable performance to previous work on various datasets while hold less computational cost. \n\n[Pros]\n\n1. In this paper, the authors extended the clustering method in Lee et al to a new method called Deep Amortized Clustering (DAC) for data clustering. In this new method, the number of clusters can be unknown at the beginning and the model itself will sequentially cluster the data points into different groups until are data points have been assigned to some clusters. This is an interesting method in that it does not need to specify the number of clusters at the beginning, and thus become more flexible.\n\n2. To achieve the DAC, the authors proposed two losses, one is Minimum Loss Filtering (MLF) and one is Anchor Filtering to cope with either multi-gaussian-distributed data points or even harder datasets. Meanwhile, the authors also proposed to estimate the density P(x; \\theta) in the case that the distribution is not knowing in prior.\n\n3. The authors evaluated the proposed method on both synthetic dataset and realistic dataset. On the synthetic dataset, the proposed method is compared with VBDPM and ACT-ST, two methods that can cope with dataset with unknown number of clusters. On the realistic datasets, the authors evaluated on the EMNIST which is of non-MoG distribution. Besides, the authors further evaluated the method on MiniImageNet features and Omniglot dataset, and showcased comparable performance to previous methods but much shorter running  time.\n\n[Cons]\n\n1. Overall, the paper is poorly written and organized. First, the notations in the method section are hard to follow. There are a number of notations which are all capital characters, either representing a function or a method. Second, the whole training process and inference process of the proposed method is not clear to me. How the model is trained on the training set, what are the learnable parameters in the proposed model and what are the settings for the hyper-parameters, etc. Third, it is hard to get the takeaway messages from the experiment sections. The experimental settings for each subsection are not very clearly explained, and the analysis on the experimental results are also vague.\n\n2. In the method, the authors proposed Minimum Loss Filtering (MLF) for clustering with the loss function in Eq(5). During training, the authors use some training data with ground-truth labels to optimize the loss function. However, it is not clear which parameters will be learned in the optimization. Also, after the training, what the exact inference procedure should be is also not clear to me. Overall, it is really hard to me to follow this section on the filtering process. The authors should definitely describe the process more clearly.\n\n3. The experimental results shown in the paper are hard to interpret. First, the setting for each experiment is not clear to me. In Figure 3, it is hard to understand the figures clearly. In table 2 and table 3, some of the clustering methods are deterministic, such as K-means, Spectral clustering, DEC. However, some other clustering methods are learning to clustering methods, such as KCL and MCL. Putting all of the numbers in the same table is confusing and make it hard to compare. I would suggest the authors make a clear distinction between different methods: 1) deep clustering methods which directly cluster on top of the test set; 2) learning to clustering method which learn some parameters on training set and then generalize to test set; 3) amortized clustering, which also learn some parameters on training and then test on the test set with just one forward pass. Splitting the testing results into three group will be helpful to the readers to understand the paper and the proposed method.\n\n4. Another missed part in the model is the ablation study. How sensitive the model is to different training set, e.g., different training dataset size, different number of training clusters, and different hyper-parameters, etc. Without these information, it is hard to know how well robust the proposed DAC method can generalize.\n \n5. Finally, the proposed method was built upon Lee at al 2019, to extend the previous method to a sequential clustering problem. I think a title \"deep amortized clustering\" is a bit misleading and exaggerated on the proposed method.\n\n[Summary]\n\nIn this paper, the authors proposed a new method called Deep Amortized Clustering (DAC) for amortized clustering. Unlike the previous work Lee et al, the proposed DAC sequentially filter the data points from the whole set and construct the clusters gradually. This is a meaningful method in that it can be applied to those data without explicit number of clusters. However, the presentation of the method and experiments make it hard to follow, and thus hard to capture the contributions of the proposed method, and also its capacity. As also mentioned above, I would highly suggest the authors revise the paper so that it can present better the method and the experimental section.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new clustering method, which builds upon the work introduced by Lee et al, 2019 - contextual information across different dataset samples is gathered with a transformer, and then used to predict the cluster label for a given sample. All reviewers agree the writing should be improved and clarified. The novelty is also on the low side, given the previous work by Lee et al. Experiments should be more convincing.

Decision: Reject