Generalised Mutual Information for Discriminative Clustering
Authors: Louis Ohl
Pierre-Alexandre Mattei
Charles Bouveyron
Warith HARCHAOUI
Mickaël Leclercq
Arnaud Droit
Frederic Precioso
Venue: NeurIPS 2022 Accept (2022)
Keywords: Unsupervised learning
Clustering
Deep learning
Information Theory

Abstract: In the last decade, recent successes in deep clustering majorly involved the mutual information (MI) as an unsupervised objective for training neural networks with increasing regularisations. While the quality of the regularisations have been largely discussed for improvements, little attention has been dedicated to the relevance of MI as a clustering objective. In this paper, we first highlight how the maximisation of MI does not lead to satisfying clusters. We identified the Kullback-Leibler divergence as the main reason of this behaviour. Hence, we generalise the mutual information by changing its core distance, introducing the generalised mutual information (GEMINI): a set of metrics for unsupervised neural network training. Unlike MI, some GEMINIs do not require regularisations when training. Some of these metrics are geometry-aware thanks to distances or kernels in the data space. Finally, we highlight that GEMINIs can automatically select a relevant number of clusters, a property that has been little studied in deep clustering context where the number of clusters is a priori unknown.

Generalised Mutual Information for Diseriminative Clustering
Louis Ohl
Pierre-Alexandre Mattei Université Côte d' Azur Inria, CNRS LJAD, Maasai team pierre-alexandre mattei@iniia.ff..mmmmmmmmmm
Université Côte d'Azur
Inria, CNRS I3S, Maasai team
CHU de Québec Research Center Laval University
louis, ohl@inria.ff
Charles Bouveryon
Warith Harchaoui Jellysmack AI Labs Research and Development
Mickael Leclerq CHU de Québec Research Center Laval University
Université Côte d' Azur
Inria, CNRS LJAD, Maasai team
Arnaud Droit
Frederic Precioso Université Côte d' Azur
CHU de Québec Research Center Laval University
Inria, CNRS I3S, Maasai team
Abstract
In the last decade, recent successes in deep clustering majorly involved the mutual information (MI) as an unsupervised objective for training neural networks with increasing regularisations. While the quality of the regularisations have been largely discussed for improvements, little attention has been dedicated MI as clustering objective. In this paper, we first highlight how the maximisation MI does not lead to satisfying clusters. divergence as the main reason of this behaviour. Hence, generalise the mutual information by changing its core distance, introducing the generalised mutual information (GEMINI): set of metrics for unsupervised neural network training. Unlike MI. some GEMINIS do not require regularisations when training. Some these metrics geometry-awan thanks to distances or kernels in the data space. Finally, we highlight that GEMINIs can automatically select a relevant number clusters, a property that has been little studied in deep clustering context where the number of clusters is priori unknown.
relevance
identified the Kullback-Leinn
Introduction
Clustering is a fundamental learning task which consists in separating data samples into several categories, each named cluster. This task hinges on two main questions concerning the assessment of сorrect clustering and the actual number of clusters that may be contained within the data distribution. However, this problem is ill- ill-posed since cluster lacks formal definitions which makes it a hard problem (Kleinberg, 2003).
Model-based algorithms make assumptions about the true distribution of the data as a result some latent distribution of clusters (Bouveyron al., 2019). These techniques able to find the most
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

likely cluster assignment to data points. These models are usually generative, exhibiting an explicit assumption of the prior knowledge on the data.
Early deep models to perform clustering first relied on autoencoders, based on the belief that an encoding space holds satisfactory properties (Xie et al., 2016; Ghasedi Dizaji al., 2017; Ji et al., 2019). However, the draw back of these architectures is that they do not guarantee that data samples which should meaningfully be far apart remain so in the feature space. Early models that dropped decoders notably used the mutual information (MI) (Krause et al., 2010; Hu et al., 2017) as an objective to maximise. The MI can be written in two ways, either as measure of dependency between two variables z and y, e.g. data distribution p(z) and cluster assignment p(y):
I( v) DrL (p(a,, ())(())))
or as an expected distance between implied distributions and the overall data:
[x; v) (P
(2)
with PKi. being the Bullback-Leiible (KI.) divergence. Related works often relied on the notion of MI as measure of coherence between cluster assignments and data distribution (Hjelm et al., 2019). Regularisation techniques were employed to leverage the potential of MI, mostly by specifying model invariances, for example with data augmentation (Ji et al., 2019).
The maximisation of MI thus gave way to contrastive learning objectives which aim at learning stable representations of data through such invariance specifications (Chen et al., 2020; Caron et al., 2020). The contrastive loss maximises the similarity between the features of a sample and its augmentation. while decreasing the similarity with any other sample. Clustering methods also benefited from recent successful deep architectures (Li al., 2021; Tao et al., 2021; Huang et al., 2020) by encompassing regularisations in the architecture. These methods correspond to discriminative clustering where we seek directly infer cluster given the data distribution. Initial methods also focused on alternate schemes, for example with curriculum learning (Chang et al., 2017) to iteratively select relevant data samples for training: for example by alternating K-means cluster assignment with supervised learning using the inferred labels (Caron et al., 2018), or by proceeding to multiple distinct training steps (Van Gansbeke et al., 2020; Dang et al., 2021; Park et al., 2021).
However, most of the methods above rarely discuss their robustness when the number of clusters to find is different from the amount of preexisting known classes. While previous work was essentially motivated by considering MI as a dependence measure, we explore in this paper the alternative definition of the MI as the expected distance between data distribution implied by the clusters and the entire data. We extend it to incorporate cluster-wit wise comparisons of implied distributions, and question the choice of the KI. divergence with other possible statistical distances.
Throughout the introduction of the generalised mutual information (GEMINI), the contributions of this paper are:
A demonstration of how the maxima of MI are not sufficient criteria for clustering. This extends the contribution of (Tschannen al., 2020) the discrete case.
The introduction of a set of metrics called GEMINIs involving different distances between distributions which can incorporate prior knowledge on the geometry of the data. Some of these metrics do not require regularisations.
A highlight of the implicit selection of clusters from GEMINIs which allows to select relevant number of cluster during training.
2 Is mutual information good clustering objective?
We consider in this section a dataset consisting N unlabelled samples D D= J,W, We distinguish two major use cases of the mutual information: one where we measure the dependence between two continuous variables, as is the case in representation learning, and one where the random variable discrete. In representation learning, the goal is to construct continuous representation z extracted from the data & using learnable distribution of parameters 0. In clustering, samples æ are assigned to the discrete variable y through another learnable distribution.

---

2.1 Representation learning
Representation learning consists in finding high-level features 2; extracted from the data æ; order to perform a downstreum lusk, e.g. clustering or classification. MI between and is common choice for learning fcatures(Hjjllm et al., 2019). However, estimating correctly MI between two random variables in continuous domains is often intractable when p(z|z) or p(z x)) is unknown, thus lower bounds are preferred, c.g. variational estimators such as MINE (Belghazi et al., 2018), Ixcu(Van den Oord et al., 2018). Another common choice of loss function train features contrastive losses such as NT-XENT (Chen et al., 2020) where the similarity between the features 2; from data Z; is maximised with the features È from aadta-uegeeeted I; against any other features 2j- Recently, Do et al. (2021) achieved excellent performances in single-stage methods by highlighting the link between the INCR estimator (Van den Oord al., 2018) and contrastive learning losses. Representation learning therefore comes at the cost of a complex lower bound estimator on MI, which often requires data augmentation. Moreover, it was noticed that the MI is hardly predictive of downstream tasks (Tschannen et al., 2020) when the variable y is continuous, i.c. high value of MI does not clarify whether the discovered representations are insightful with regards the target of the downstream task.
2.2 Discriminative clustering
The MI has been first used as an objective for learning discriminative clustering models (Bridle al.. 1992). Associated architectures went from simple logistic regression (Krause et al., 2010) to deeper architectures (Hu et al., 2017: Ji et al., 2019). Beyond architecture improvement, the MI maximisation was also carried with several regularisations. These regularisations include penalty terms such as weight decay (Krause et al., 2010) or Virtual Adversarial Training (VAT, Hu et al., 2017; Miyato et al., 2018b). Data augmentation was further used to provide invariances in clustering. as well as specific architecture designs like auxiliary clustering heads (Ji et al., 2019). Rewriting the MI in terms of entropies:
I(=: = H(y) - H(ylz)
highlights& requirement for balanced clusters, through the cluster entropy term H(yy). Indeed, uniform distribution maximises the entropy. This hints that an unregularised discrete mutual information for clustering can possibly produce uniformly distributed clusters among samples, regardless of how close they could be. We highlight this claim in section 2.3. As an example of regularisation impact: maximising the MI with Eg constraint can be equivalent to soft and regularised K-Means in a feature space (Jabi al., 2019). In clustering, the number of clusters find usually not known in advance. Therefore, an interesting clustering algorithm should be able to find a relevant number of clusters, i.e. perform model selection. However, model selection for parametric deep clustering models is expensive (Ronen et al., 2022). Cluster selection through MI maximisation has been little studied in related works, since experiments usually tasked models to find the (supervised) classes of datasets. Furthermore, the literature diverged towards deep learning methods focusing mainly on images, yet rarely on other type of data such as tabular data (Min et al., 2018).
2.3 Maximising the MI can lead to bad decision boundaries
Maximising the MI directly can be a poor objective: a high MI value is not necessarily predictive of the quality of the features regarding downstream tasks (Tschannen et al., 2020) when y is continuous. We support similar argument for the case where the data x is a continuous random variable and the cluster assignment a categorical variable. Indeed, the MI can be maximised by setting appropriately a sharp decision boundary which partitions evenly the data. This reasoning can be seen in the entropy-bac based formulation of the MI (Eq. 3): any sharp decision boundary minimises the negative conditional entropy, while ensuring balanced clusters maximises the entropy of cluster proportions. Consider for example Figure 1, where mixture of Gaussian distributions with equal variances is separated by a sharp decision boundary. We highlight that both models will have the same mutual information on condition that the misplaced decision boundary of Figure 1b splits evenly the dataset (see Appendix A).
Globally, MI misses the idea in clustering that any two points close to one another may be in the same cluster according to some chosen metric. Hence regularisations are required to ensure this constraint. A sketch of these insights was mentionned by Bridle et al. (1992); Corduneanu and Jaakkola (2002).
3

---

(a) Good decision boundary
(b) Misplaced decision boundary
Figure I: Example of maximised MI for Gaussian mixture p(x) (No, o²) + N o²). It clear that Figure la presents the best decision boundary and posterior between the two Gaussian distributions. Yet, as p(æly) converges to Dirac distribution, the MI difference converges to O.
Extending the mutual information to the generalised mutual information
Given the identified limitations of MI, we now describe the discriminative clustering framework based on our perception of the mutual information. We then detail the different statistical distances we can use to extend MI to the generalised mutual information (GEMINI).
3.1 The discriminative clustering framework for GEMINIs
We change our view on the mutual information by seeing discriminative clustering objective that aims at separating the data distribution given cluster assignments p(zly) from the data distribution p(z) according to the KL divergence:
I(x; v) = By~p(s) [DKL (p(zl/)))l)()))
(4)
To highlight the discriminative clustering design, we explicitly remove our hypotheses on the data distribution by writing Pama( The only part of the model that design is conditional distribution po(y/z)) that assigns cluster y to a sample æ using the parameters 0 (Minka, 2005). This conditional distribution can typically be neural network of adequate design regarding the data, e.g. CNN, simple categorical distribution. Consequently, the cluster proportions are controlled by because po(y) E[po(y|z) and so is the conditional distribution pe (e y) even though intractable. This questions how Eq. (4) can be computed. Fortunately, well-known properties MI can invert the distributions which the KI. divergence is computed (Bridle et al., 1992; Krause al., 2010) via Bayes' theorem:
I(x; v) (x- [DL x) (ppo()))
(5)
which is possible to estimate. Since we highlighted earlier that the KI. divergence in the MI can lead to inappropriate decision boundaries, we are interested in replacing by other distances or divergences. However, changing it in Eq. (5) would focus on the separation of cluster assignments from the cluster proportions which may be irrelevant to the data distribution. We rather alter Eq. (4) to clearly show that we separate data distributions given clusters from the entire data distribution because allows us to take into account the data space geometry.
3.2 The GEMINI
The goal of the GEMINI is to separate data distributions according to an arbitrary distance D, i.c. changing the KL divergence for another divergence or distance the MI. Moreover, we question the evaluation of the distance between the distribution of the data given cluster assumption po (zlv) and the entire data distribution Pasu (z). We argue that is intuitive in clustering to compare the distribution of one cluster against the distribution of another cluster rather than the data distribution. This raises the definition of two GEMINIs, one named one-vs-all (OvA):
1 v) Ey-po(') [D (x) | [x)))
which compares the cluster distributions to the data distribution, and the one-vs-one (OvO) which independently draw cluster assignments Ya and yb (see App. B for OvO justification):
I (x::) Ey, un pp) (p (20 (y. 1lp (o)))))

---

Table 1: Definition of the GEMINI for f-divergences, MMD and the Wasserstein distance. We directly write here the equation that can be optimised to train a discriminative model pa Po(y/z) via stochastic gradient descent since they are expectations over the data.
Name
Equation
KL OvA/MI
(x) [ (p (z) (lpe
KL OvO
ppsm (ax) DKL. (po x)) Ilpo (u))) + DKL.
Squared Hellinger
1
E Epumat (z)) [ poly)
Ps (ular
OvA Squared Hellinger
poty
E prsso (x) V (o V
Pal an
OvO
Peo
OvA
Epen (x) D (po (y (r)
(1))
TV OvO
1 Ppenu (ar) [R Ue Mh (po
Pe (a mx)
Pe (vo br) po(y))
polua)
MMD OvA
Epo(s)
Bx 30 (ppml(()))
(((xe: 2o)
(po (ao Pa(")
2
20 (u)a=)) no (o(1))
Po la )po (Wa (at6)
Eg (x))
Ex xo mmen (2x)
k(Exa xb6)
po (2()))
MMD OvO
po (1b a 1ps (10 (10/220)) (26))
Do (y (a )po (wo 27t))
po po(y6)
po (Ya (yb)
Wasserstein OvA
E B
[W EEin mm/50
Sz Ei N 5zz
Wasserstein OvO
Eyg 86 po(s))
We (Eim m
N Σi m 8x:
There exists other distances than the KL to measure how far two distributions and q are one from the other. We can make a clear distinction between two types of distances, Csiszar's f-divergences (Csiszár, 1967) and integral probability metrics (IPM) (Sriperumbudur et al., 2009). However, unlike f-divergences, IPM-derived distances like the Wasserstein distance or the maximum mean discrepancy (MMD)(Gneiting and Raftery, 2007; Gretton et al., 2012) bring knowledge about the data throughout either distance e or a kernel k: these distances are geometry-awane
R+ R such that f11 = O. This function is applied to evaluate the ratio between two distributions p and q, as in Eq. (8):
f-divergence GEMINIs:
These divergences involve convex function f :
/ p(z) 7(2)
Draiv (p.9) = Ba-g())
(8)
We will focus on three f-divergences: the KL divergence, the total variation (TV) distance and the squared Hellinger distance. While the KI. divergence the usual divergence for the MI, the TV and the squared Hellinger distance present different advantages among f-divergences. First of all, both of them are bounded between and 1. It is consequently easy to check when any GEMINI using those is maximised contrarily to the MI that is bounded by the minimum of the entropies of æ and (Gray and Shields, 1977). When used as distance between data conditional distribution pa (x: ly) and data distribution (4), we can apply Bayes' theorem in order to get estimable equation to maximise, which only involves cluster assignment po(y))) and marginals pe (y) (see Table 1).
The MMD corresponds to the distance between the respective expected embed- ding of samples from distribution p and distribution in a reproducing kernel hilbert space (RKHS) H:
MMD GEMINIs:
MMD(p lg) = ||Ez=p(z) (z)]
where y the RKHS embedding. To compute this distance we can use the kernel trick (Gretton et al., 2012) by involving the kernel function <(a.b) (yp()), (bb)). We use Bayes' theorem to uncover a version of the MMD that can be estimated through Monte Carlo using only the predictions Pe (u( (see Table I).
5

---

Wasserstein GEMINIs:
This distance is an optimal transport distance, defined as:
(c(p. g) ( inf E (z,a~)(E.a)) [c(z,2)] (El((p.))
(10)
where T(p. q) is the set of all couplings between p and q and distance function in X. Computing the Wasserstein distance between two distributions palalyy k and pa pa = k2) difficult in our discriminative context because we only have access to finite set of samples N. To achieve -ass we instead use approximations of the distributions with weighted sums of Diracs:
N Pa Pa = 1b) = Σ
m, 8z. = PN: with mi poly k: k(x,))
(11)
Po = k,2,)
where 6z, is Dirac located on sample location z; E X. The Wasserstein-OvA and -OvO applied to Dirac sums compatible with the emd2 function of the Python optimal transport package (Flamary et al., 2021) which gracefully supports automatic differentation (see Appendix D.3 for convergence to the expectation). All GEMINIs summarised in Table 1, (see Appendix D for derivations).
4 Experiments
For all experiments below, we report the adjusted rand index (ARI) (Hubert and Arabie, 1985), common metric in clustering. This metric is external as it requires labels for evaluation. It ranges from 0, when labels are independent from cluster assignments, to 1, when labels are equivalent to cluster assignments up to permutations. An ART close to 0 equivalent to the best accuracy when voting constantly for the majority class, e.g. 10% on balanced 10-class dataset. Regarding the MMDD and unless specified otherwise. All discriminative models are trained using the Adam optimiser (Kingma and Ba, 2014). We estimate a total 450 hours GPU consumption. (See Appendix I for the details of Python packages for experiments and Appendix F for further experiments regarding model selection). The code available at https: /ttpt/ggtttt/ttt com oshillou//EMIII
we used by default a linear kernel and the Fuclidean distance
4.1 When the MI fails because the modelling
We first took the most simple discriminative clustering model, where each cluster assignment according to the input datum follows categorical distribution:
y/ z; ~ Cat(O), 82 0%)
We generated N 100 samples from a simple mixture of K 3 Gaussian distributions. Each model thus only consists in N K parameters to optimise. This is simplistic way of describing the most flexible deep neural network. We then maximised on the one hand the KL-OvA (MI) and on the other hand the MMD-OvA. Both clustering results can be seen in Figure 2. We concluded that
(a) KL-OvA (MI)
(b) MMD OvA with lincar kernel
Figure 2: Clustering of mixture of 3 Gaussian distributions with MI (left) and a GEMINI (right) using categorical distributions. The MI does not have insights the data shape because of the model, and clusters points uniformly between the clusters (black dots, red triangles and blue crosses) whereas the MMD is aware of the data shape through its kernel.

---

without any function, e.g. neural network, to link the parameters of the conditional distribution with z. the MI struggles find the correct decision boundaries. Indeed, the position of the 2D space plays role and the decision boundary becomes only relevant with regards to cluster entropy maximisation: a uniform distribution between clusters. However, it plays a major role in the kernel of the MMD-GEMINI thus solving correctly the problem.
4.2 Resistance to outliers
(a) KL-OvA (MI) Entropy Map
(b) MMD- -00 Entropy Map (c) Wasserstein-OWv Entropy Map
Figure 3: Entropy maps of the predictions of each MLP trained using GEMINI or the MI. The bottom-left distribution (yellow) is Student-t distribution with degree of freedom that produces samples far from the origin. The Rényi entropy of prediction is highlighted from lowest (red background) to highest (blue background). MI on the left has the most confident predictions overall and the smallest uncertainty around the decision boundary, i.e. high entropy variations.
To prove the strength of using neural networks for clustering trained with GEMINI, we introduced ex- treme samples Gaussian mixtures by replacing a Gaussian distribution with a Student-t distribution for which the degree of freedom is small. We fixed K 4 clusters, 3 being drawn from multivariate Gaussian distributions and the last one from a multivariate Student-t distribution in 2 dimensions for visualisation purposes with degree of freedom (see AppendixE for other parameters and results). Thus, the Student-t distribution has an undefined expectation and produces samples that can be perceived outliers regarding Gaussian mixture owing to its heavy tail. We report the ARIs of multi-layered perceptron (MLP) trained 20 times with GEMINIs in Table 2. The presence of "outliers" leads K-Means and Gaussian Mixture models to fail grasping the 4 distributions when tring find 4 clusters. Meanwhile, GEMINIs perform better. Note that all MMD and Besserstein-OvO-EEN present lower standard deviation for high scores compared to f f-divergence GEMINIs. We attribute these performances to both the MLP that tries to find separating hyperplanes in the data space and the absence of hypotheses regarding the data. Moreover, as mentioned in section 2.3, the usual MI best maximised when its decision boundary presents little entropy Hpa (y/ ()). As neural networks can be overconfident (Guo et al., 2017), MI is likely to yield overconfident clustering by minimizing the conditional entropy. We highlight such behaviour in Figure 3 where the Rényi entropy (Rényi, 1961) associated to each sample the MI (Figure 3a) is much lower, if not compared MMD-OvO and Wasserstein- OvO (figures 3b and 3c). We conclude that Wasserstein- and ALMD-GEMING train neural networks not to be overconfident, hence yielding more moderate distributions po (y)
Table 2: Mean ARI (std) of MLP fitting mixture of 3 Gaussian and Student-t multivariate distributions compared with Gaussian Mixture Models and K-Means. The models try to find 4 at best and the Student-t distribution has 5=1 degree of freedom. We write the ARI for the maximum posteriori of an oracle aware of all parameters of the data.
GMM
MMD
Wasserstein
K-Means
full cov
diagonal cov
Tova MMD
Tuvo MMD
Tova
Zuy
0
0.024 (0.107)
0.922 (0.004)
0.921 (0.007)
0.915 (0.131)
0.922
(0)
(0)
(0.006)
f f-divergences
Oracle
Tova
Tovo H²
Tova
Tovo
0.939
0.723
0.906 (0.103)
0.858 (0.143)
0.904 (0.104)
0.938
0.989
(0.006)
(0.114)
(0.005)
7

---

4.3 Leveraging a manifold geometry
We highlighted that MI be maximised with- out requiring to find the suitable decision bound- ary. Here, we show how the provided distance to the Wasserstein-Ovee GEMINI can leverage appropriate clustering when we have good priori on the data.
(a) Iaa for 2 clusters (MI)
(b) Zw for 2 clusters
Figure 4: Fitting sand-generated moons using the
GEMINI on top of MLP with 2 clusters to find.
The importance of the distance We gener- ated a dataset consisting of two facing moons on which we trained MLP using either the MI the Wasserstein- GEMINI. To construct distance for the Wasserstein distance, we derived distance from the Saryhhll algorithm (Warshall, 1962; Roy. 1959) on a sparse graph describing neighborhoods of samples. This distance describes how many neighbors are in between two samples, further details are provided in appendix H. We report the different decision boundaries in Figure We observe that the insight on the neighborhood provided by our distance helped the MLP to converge to the correct solution with an appropriate decision boundary unlike the MI. Note that the usual Euclidean distance the Wasserstein metric would have converged to a solution similar to the MI. Indeed for 2 clusters, the optimal transport plan has larger value, using a distribution similar to Figure 4a, than in Figure 4b. This toy example shows how insightful metric provided to the Wasserstein distance in GEMINIs can lead correct decision boundaries while only designing discriminative distribution po po(y(z) and distance c.
Not using all clusters In addition, we high- light an interesting behaviour all GEMINIs. During optimisation, it is possible that the model converges to using fewer clusters than the number to find. For example in Figure 5, for 5 clusters, the model can converge to balanced clusters and empty cluster (Figure 5b) unlike MI that produced 5 misplaced clusters (Figure 5a). Indeed, the entropy on the cluster proportion in the MI forces to use the maximum number of clusters.
(a) I for clusters (MI) (b) I for S clusters
Figure 5: The Wasserstein-ovo model with 5 clus- ters eventually found 4 unlike the MI that main- tained 5 clusters.
Table 3: ARI for deep neural network trained with GEMINIs on MNIST for 500 epochs. Models trained either with either 10 clusters to find 15. We indicate in parentheses the number of used clusters by the model after training.
10 clusters
15 clusters
GEMINI
MLP
LeNet-5
MI.P
LeNet-5
OvA
0.320 (10)
0.138 (8)
0.271 (15)
0.136 (12)
KI.
OvO
0.348 (7)
0.123 (4)
0.333 (8)
0.104 (4)
OvA
0.301 (10)
0.207 (6)
0.224 (13)
0.162 (7)
Squared Hellinger
OvO
0.287 (10)
0.161 (6)
0.305 (13)
0.157 (7)
OvA
0.299 (10)
0.171 (6)
0,277 (15)
0.140 (6)
TV
OvO
0.422 (10)
0.161 (9)
0.330 (15)
0.182 (14)
OvA
0.373 (10)
0.382 (10)
0.345 (15)
0.381 (15)
MMD
OvO
0.361 (10)
0.373 (10)
0.364 (15)
0.379 (15)
OvA
0.471 (10)
0.463 (10)
0.390 (15)
0.446 (11)
Wasserstein
OvO
0.450 (10)
0.383 (10)
0,415 (15)
0.414 (15)
K-Means
0.367
0.385

---

4.4 Fitting MNIST
We trained a neural network using either MI or GEMINIs. Following Hu et al. (2017), we first tried with a MLP with one single hidden layer of dimension 1 1200. To further illustrate the robustness of the method and its adaptability to other architectures, we also experimented using a LeNet-5 architecture (LeCun et al., 1998) since it is adequate to the MNIST dataset. We report our results in Table 3. Since we are dealing with a clustering method, we may not know the number of clusters a priori in dataset. The only thing that can be said about MNIST is that there are at least 10 clusters, one per digit. Indeed, writings of digits could differ leading to more clusters than the number of classes. That is why see that the scores of the MMD and Wasserstein GEMINIs are greater than the MI, with the highest performances for the Was We also observe that f-divergence- GEMINN always yield best ARIs. Nonetheless, we observe better performances in the case of the TV GEMINIS owing to its bounded gradient. This results in controlled stepsize when doing gradient descent contrarily KL.- and squared Hellinger-GEMIIN
further tested the same method with 15 clusters to find in Table 3. We first
Notice that the change of architecture from a MLP to a LeNet-5 unexpectedly halves the scores for the f-divergences. We believe this drop is due to the change of notion of neighborhood implied by the network architecture.
4.5 Cifar10 clustering using SIMCLR-derived kernel
To further illustrate the benefits of the kernel or distance provided to GEMINIs, we continue the same experiment in section 4.4. However, we focus this time the CIFARIO dataset. As improved distance, we chose linear kernel and & norm between features extracted from a pretrained SIMCLR model (Chen et al., 2020). We provide results for two different architectures: LeNet-5 and ResNet-18 both trained from scratch on raw images, the latter being a common choice of models in deep clustering literature (Van Gansbeke et al., 2020; Tao et al., 2021). We report the results in Table and provide the baseline of MI. We also write the baselines from related works when not using data augmentations to make fair comparison. Indeed, models trained with GEMINIs do not use data augmentation: only the architecture and the kernel or distance function in the data space plays a role. We observe here that the choice of kernel or distance can be critical in GEMINIs. Indeed, while the Euclidean norm between images does not provide insights on how images of cats and dogs are far as shown by K-Means, features derived from SIMCLR carry much more insight on the proximity of images. This shows that the performances of GEMINIS depend on the quality of distance functions. Interestingly, we observe that for the Resnet-18 18 using SIMCLR features to guide GEMINIs was not as successful as it has been on the LeNet-5. We believe that the ability of this network to draw any decision boundary makes it equivalent a categorical distribution model as Sec. 4.1. Finally, to the best of our knowledge, we are the first to train from scratch standard discriminative neural network on CIFAR raw images without using labels or direct data augmentations, while getting sensible clustering results. However, other recent methods achieve best scores using data augmentations which we do not (Park et al., 2021).
Table 4: ARI score of models trained for 200 epochs on CIFAR10 with different architectures using GEMINIs. The kernel for the MMD is either a linear kernel or the dot product between features extracted from pretrained SIMCLR model. Both the Euclidean norm between images and SIMCLR features are considered for the Wasserstein metric. We report the ARI of related works when not using data augmentation for comparison.** *: scores reported from Li et al., (2021)
No kernel
Linear kernel / l2 norm
SIMCLR (Chen et al., 2020)
Architecture
KI.
Zova
MMD
Ioya
Tovo
Iova
I
Tovo
LeNet-5
0.026 0.008
0.049
0.048
0.043
0.041
0.157
0.145
0.079
0.138
R8
0.047
0.044
0.037
0.036
0.122
0.145
0.052
0.080
KMeans (images/ / SIMCLR)
0.041
0.147
CC (Li et al., 2021)
0.030
IDFD (Tao et al., 2021)
0.060
JULE (Yang et al., 2016)*
0.138

---

5 Conclusion
We highlighted that the choice of distance at the core of MI alter the performances of deep learning models when used as an objective for a deep discriminative clustering. We first showed that MI maximisation does not necessarily reflect the best decision boundary in clustering. We introduced the GEMINI, method which only needs the specification of a neural network and a kernel or distance in the data space. Moreover, we showed how the notion of neighborhood built by the neural network can affect the clustering, especially for MI. To the best of our knowledge, this is the first method that trains single-stage neural networks from scratch using neither data augmentations nor regularisations, yet achieving good clustering performances. We emphasised that GEMINIs are only searching for a maximum number of clusters: after convergence some may be empty. However, we do not have insights to explain this convergence which is part of future work. Finally, we introduced several versions of GEMINIs and would encourage the MMD-OvA or Wasserstein-Ove as default choice, since it proves to both incorporate knowledge from the data using kernel or distance while remaining the less complex than MMD-OvO and Wasserstein- OvO in time and memory. OvO versions could be privileged for fine-tuning steps. Future works could include an optimisation of the time performances of the Wasserstein-Owe to make it more competitive.
Acknowledgements
This work has been supported by the French government, through the 31A Côte d' Azur, Investment in the Future, project managed by the National Research Agency (ANR) with the reference number ANR-19-P3IA-00022 We would also like to thank the France Canada Research Fund (FFCR) for their contribution to the project. This work was partly supported by EU Horizon 2020 project A14Media, under contract 951911. The authors are grateful to the OPAL infrastructure from Université Côte A'zur for providing resources and support.
References
M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein Generative Adversarial Networks. In D. Precup and Y. W. Teh, editors, Proceedings the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214-223. PMLR, Aug. 2017.
S. Arora, R. Ge, Y. Liang, T. Ma, and Y. Zhang. Generalization and Equilibrium in Generative Adver- sarial Nets (GANs). In D. Precup and W. Teh, editors, Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 224-232. PMLR, 06-11 1 Aug 2017
M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm. Mutual Information Neural Estimation. In J. Dy and A. Krause, editors, Proceedings of the 35th Interna- tional Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 531-540. PMLR, 10-15 Jul 2018
P. Billingsley. Convergence of Probability Measures. John Wiley & Sons, 2013.
C. Bouveyron, G. Celeux, T. B. Murphy, and A. E. Raftery. Model-based Clustering and Classification for Data Science: with applications in R. in Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.
J. Bridle, A. Heading, and D. MacKay. Unsupervised Classifiers, Mutual Information and 'Phantom Targets. In J. Moody, S. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems, volume 4. Jorgan-Kaufmang 1992.
M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep Clustering for Unsupervised Learning of Visual Features. In Proceedings of the European Conference Computer Vision (ECCV), pages 132-149, 2018
M. Caron, I. Misra, J. Mairal, P. Goyal, Bojanowski, and A. Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In H. Larochelle, M. Ranzato, R. Hadsell, Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9912-9924. Curran Associates, Inc., 2020.
10

---

J. Chang, L. Wang, G. Meng. S. Xiang, and C. Pan. Deep Adaptive Image Clustering. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5880-5888, 2017. ICCV.2017.626.
doi: 10.1109/
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In International Conference on Muchine Learning, pages 1597-1607. PMLR, 2020.
A. Corduneanu and T. Jaakkola. On Information Regularization. In Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence, pages 151-158, 2002.
I. Csiszár. Information-Type Measures of Difference of Probability Distributions and Indirect Observation. studia scientiarum Mathematicarum Hungarica, 2:229-318, 1967.
Z. Dang, C. Deng, X. Yang, K. Wei, and H. Huang. Nearest Neighbor Matching for Deep Clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13693-13702, 2021.
K. Do, T. Tran, and Venkatesh. Clustering by Maximizing Mutual Information Across Views. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9928-99388 2021.
W. Falcon et al. Pytorch Lightning. Gitllub, Note: http://ghhho...oom//fffttttthnnn lightning, 3, 2019.
R. Flamary, N. Courty, A. Gramfort, M. Z. Alaya, A. Boisbunon, S. Chambon, L. Chapel, A. Corenflos, K. Fatras, N. Fournier, L. Gautheron, N. T. Gayraud, H. Janati, A. Rakotomamoniy, Redko, A. Rolet, A. Schutz, V. Seguy, D. J. Sutherland, R. Tavenard, A. Tong, and T. Vayer. Pot: Python Optimal Transport. Journal of Machine Learning Research, 22(78): 2021.
K. Ghasedi Dizaji, A. Herandi, C. Deng, W. Cai, and H. Huang. Deep clustering via joint convo- lutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE International Conference on Computer Vision, pages 5736-5745, 2017.
T. Gneiting and A. E. Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-378, 2007.
H. Gouk, Frank, B. Pfahringer, and M. Cree. Regularisation Neural Networks by Enforcing Lipschitz Continuity. Machine Learning. 110(2):393-416, Feb. 2021. ISSN 1573-0565. doi:
10011077/s109944-000-52222
R. M. Gray and C. Shields. The Maximum Mutual Information between Two Random Processes. Information and Control, 33(4):273 280, 1977.
A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A Kernel Two-Sample Test. The Journal of Machine Learning Research, 13(1):723773,, 2012. Publisher: JMLR. org.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On Calibration of Modern Ncural Networks. In International Conference Machine Learning, pages 1321-1330. PMLR, 2017.
C. R. Harris, K. Millman, S. J. der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchan
K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant. Array programming with NumPy. Nature, 585(7825):377-36,, Sept. 2020. doi: 10. 101.0338/44566-2000 186-000-2269-22
R. D. Hjelm, A. Fedorov, S. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
K. Grewal, P. Bachman, A. Trischler, and Y. Bengio.
W. Hu, T. Miyato, S. Tokui, E. Matsumoto, and M. Sugiyama. Learning Discrete Representations via Information Maximizing Self-Augmented Training. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages
PMLR, Aug. 2017.
11

---

J. Huang, S. Gong, and X. Zhu. Deep Semantic Clustering by Partition Confidence Maximisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8849-8858, 2020.
L. Hubert and P. Arabie. Comparing partitions. Journal of classification, 2(1):193-218, 1985.
J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science Engineering, 9()): 90-95, 2007. doi: 10. 109/MCSE.
M. Jabi, M. Pedersoli, A. Mitiche, and I. B. Ayed. Deep clustering: On the link between discriminative models and K-means. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2019. doi: 10.11097PAMM22019292000
X. Ji, F. Henriques, and A. Vedaldi. Invariant Information Clustering for Unsupervised Image Classification and Segmentation. In Proceedings the IEEF/CVF International Conference on Computer Vision, pages 9865-9874, 2019.
D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980, 2014.
J. Kleinberg. An Impossibility Theorem for Clustering, In S. Becker, S. Thrun, and K. Obermayer, editors, Advances in Neural Information Processing Systems, volume 15, MIT Press, 2003.
A. Krause, Perona, and R. Gomes. Discriminative Clustering by Regularized Information Maxi- mization. In J. Lafferty, C. Williams. J. Shawe-Taylor R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324 1998.
Y. Li, P. Hu, 7. Liu, D. Peng. J. T. Zhou, and X. Peng. Contrastive Clustering. In 2021 AAAI Conference on Artificial Intelligence (AAAI), 2021.
E. Min, X. Guo, Q. Liu, G. Zhang, J. Cui, and J. Long. A Survey of Clustering With Deep Learning: From the Perspective of Network Architecture. IEEE Access, PP:1-1, 2018. doi: 10.1 A..10//ACCESS.2218.5555..
T. Minka. Discriminative models, not discriminative training. Technical report, Technical Report MSR-TR-2005-144 144, Microsoft Research, 2005.
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral Normalization for Generative Adversar- ial Networks. In International Conference Learning Representations, 2018a
T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1188):1979-1993, 2018b. Publisher: IEBE.
A. B. Owen. Monte Carlo theory, methods and examples. Stanford, 2013.
Park, S. Han, Kim, D. Kim, S. Park, Hong, and M. Cha. Improving Unsupervised Image Clustering With Robust Learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2021.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Seikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825-28360 2011.
12

---

A. Rényi. On measures of entropy and information. In Proceedings the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statisties, volume 4, pages 547-562. University of California Press, 1961.
M. Ronen, S. E. Finder, and O. Freifeld. DeepDPM: Deep Clustering With an Unknown Number of Clusters. In 2022 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9851-9860, 2022. doi: 10.1 10.1099/VPRS22688222222 202220009......00
B. Roy. Transitivité connexité. Comptes Rendus Hebdomadaires Des Séances De L'Académie Des Sciences, 249(2)211-218,, 1959.
B. K. Sriperumbudur, K. Fukumizu, A. Gretton, B. Schölkopf. and G. R. Lanckriet. On integral probability metrress,-ddivergenees and binary classification. arXiv preprint arXiv:0901.2698, 2009.
Y. Tao, K. Takagi, and K. Nakata. Clustering-friendly representation learning via instance dis- crimination and feature decorrelation. In International Conference on Learning Representations, 2021.
M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic. On Mutual Information Maxi- mization for Representation Learning. In International Conference Learning Representations, 2020.
A. Van den Oord, Y. Li, and O. Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv e-prints, pages arXiv-1807, 2018.
W. Van Gansbeke, S. Vandenhende, S. Georgoulis, M. Proesmans, and Van Gool. Scan: Learning to Classify Images without Labels. In European Conference on Computer Vision, pages 268-28 285. Springer, 2020.
C. Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Warshall. A Theorem on Boolean Matrices. Journal of the ACM (JACM), 11):11112, 1962.
J. Xie, R. Girshick, and A. Farhadi. Unsupervised Deep Embedding for Clustering Analysis. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 478-487, New York, New York, USA, June 2016. PMLR.
Yang. D. Parikh, and Batra. Joint Unsupervised Learning of Deep Representations and Image Clusters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5147-5156, 2016.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We clearly state that we focus on mutual information for clustering and how improve it.
(b) Did you describe the limitations your work? [Yes] We describe shortcomings in Sec. 4.5,5 and provide more in App. G.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Yes, we confirm having read the ethics review guidelines and conform to it.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We clearly highlighted that we compute the GEMINI between a continuous variable, namely the data, and discrete variable, namely the cluster assignment. The framework was detailed precisely in Sec. 3.1 and all distances have been introduced.
(b) Did you include complete proofs of all theoretical results? [Yes] Please refer to appendices A, D.3 D for proofs of our claims.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material github will be available upon release of paper and a sample is zipped aside in the supplementary material.
URL.? [Yes] Link to
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Data generation processes are reported in supplementary materials and main parameters are described at the head of Sec.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] When applicable due to multiple runs in table form, e.g. Table
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We mention the beginning of
Sec.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] All packages for the experiments are mentionned in Appendix I
(b) Did you mention the license of the assets? [Yes] We under licenses of MIT (NumPy, POT.), BSD (torch,pandas, seikit-learn) and the python software foundation (matplotlib, python).
(c) Did you include any new assets either the supplemental material or as URL? [Yes] The code once released will contain the specific file losses, -y that will contain only the definitions of GEMINIs, alongside all configurations to reproduce the experiments.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "w166dzJ002R", "reviewer": "Reviewer_XXcZ", "summary": "This paper discussed the drawbacks of applying MI and K-L divergence as the distance metric in clustering problems and then proposed some replacements for the K-L divergence. The authors showed that the proposed approach, called generalised mutual information (GEMINI) can outperform the traditional mutual information based approach by several experiments.\n", "questions": "The authors mentioned “We argue that it is intuitive in clustering to compare the distribution of one cluster against the distribution of another cluster rather than the data distribution.” Could the authors explain more about this statement? Why it is more intuitive?", "limitations": "No.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This presentation of this paper is clear; however, I am quite skeptical about the contribution of this paper due to the following reasons:\n\n1. The choices of the distance metrics (f-divergence, MMD, Wasserstein) seem to be based on some \"intuitive\" reasons, without technical justifications. It is unclear to me why a particular choice of a distance metric, say MMD, can outperform another, say Wasserstein, in certain datasets. \n\n2. The justifications of the distance metric designs are mainly based on the experiments and it seems that the authors only conduct limited experiments, and compare with limited approaches. If there are theoretical justification of the proposed approach, I can accept with only limited experiments; however, this paper did not provide such kinds of analyses, and hence I think the experiments are not enough.\n\n3. Overall, I think there is not enough contribution and novelty by just proposing different distance metrics to replace K-L divergence in MI-based clustering. There needs more analyses to justify or prove that which choice of the distance metric can perform well under which kinds of condition or datasets. Without that, it is not convincing for the reviewer that the proposed approach can be either with theoretical guarantees or useful for general classed of datasets.\n\n4. Typo：\nline 139 —> “which is possible to estimateSince”\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jLnArYIlOOF", "reviewer": "Reviewer_usNx", "summary": "The authors introduce measures of discrimenative distance between distributions that are alternatives to mutual information. These measures in the case of MMD and Wasserstein include notions of distance or similarity which can capture the shape of the data distribution. The authors use the proposed measures as objectives for clustering. Evaluation is conducted on supervised multi-class data sets using adjusted rand index as the measure of performance.\n", "questions": "Figure 1: What does \"as the posterior turns sharper\" mean? This statement is clarified later in the text, but the caption should stand on its own.\n", "limitations": "Nothing to add.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work deepens our understanding of the clustering problem, an ill-defined and challening problem. By identifying a failure model inherent in using mutual information as an objective, this work opens the possibility to design better clustering algorithms. The proposed methods aim to achieve this through the incorporating of the discrimnative clustering hypothesis: clusters are characterized by a discrimnative model $p(y|x)$.\n\nThe MMD and Wasserstein GEMINI include distance information through $k(x,y)$ and $c(x,y)$, respectively. It seems that these regularize the cluster boundaries to be smooth in data space. It would be nice if this intuition could be confirmed/refuted beyond ARI values and supposition. Additionally, while learning the kernel or distance function is (I think) beyond the scope of this paper, it would be good for the authors to discuss what impact the GEMINI approach has on simulaneous kernel learning and clustering methods.\n\nThe authors demonstrate some degree of automatic selection of the number of clusters. But, if this is to be a major selling point of the method one would be inclined to set the $k$ to be very large and let the method pick out the correct value. The authors should report performance of such experiments. E.g. how well does the method do on MNIST when $k$ is set to 50 or 100? \n\nDemonstration on more real-world data sets would be welcome. \n\n\nTypos:\nTable 1 MMD-OVA: the x's should be x_a or x_b. Should the y_a and y_b just be y?\nTable 2 header: two I_W^OVA.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FmCwLdqLmpq", "reviewer": "Reviewer_gtjD", "summary": "This paper aims to solve the problems that arise from using MI in clustering tasks. They generalise the mutual information by changing its \ncore distance, introducing the generalised mutual information (GEMINI): a set of metrics for unsupervised neural network training.", "questions": "Q1. How can we choose which GEMINI method to use?\n\nQ2. What the connections of GEMINI with contrastive learning techniques?", "limitations": "Unclear writing:\n\nLine 72: 'Maximising the MI becomes then interesting...'\n\nLine 73: 'However, estimating correctly MI on two continuous domains is hard...'\n\nLine 119: 'Hence regularisation to ensure this intuition is necessary.'\n\nLine 139: 'possible to estimateSince we'\n\nand many more...", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The paper tackles an important problem.\n2) Well motivated\n3) The weaknesses of MI are well explained.\n4) Good theoretical analysis of GEMINI\n5) #clusters need not be known apriori\n\nWeaknesses:\n1) Writing needs lot of work\n2) Needs more experimental analysis, on more recent, complex datasets\n3) Formatting needs work", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept