Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims to solve the problems that arise from using MI in clustering tasks. They generalise the mutual information by changing its 
core distance, introducing the generalised mutual information (GEMINI): a set of metrics for unsupervised neural network training.
strengths_and_weaknesses: Strengths:
1) The paper tackles an important problem.
2) Well motivated
3) The weaknesses of MI are well explained.
4) Good theoretical analysis of GEMINI
5) #clusters need not be known apriori

Weaknesses:
1) Writing needs lot of work
2) Needs more experimental analysis, on more recent, complex datasets
3) Formatting needs work
questions: Q1. How can we choose which GEMINI method to use?

Q2. What the connections of GEMINI with contrastive learning techniques?
limitations: Unclear writing:

Line 72: 'Maximising the MI becomes then interesting...'

Line 73: 'However, estimating correctly MI on two continuous domains is hard...'

Line 119: 'Hence regularisation to ensure this intuition is necessary.'

Line 139: 'possible to estimateSince we'

and many more...
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper discussed the drawbacks of applying MI and K-L divergence as the distance metric in clustering problems and then proposed some replacements for the K-L divergence. The authors showed that the proposed approach, called generalised mutual information (GEMINI) can outperform the traditional mutual information based approach by several experiments.

strengths_and_weaknesses: This presentation of this paper is clear; however, I am quite skeptical about the contribution of this paper due to the following reasons:

1. The choices of the distance metrics (f-divergence, MMD, Wasserstein) seem to be based on some "intuitive" reasons, without technical justifications. It is unclear to me why a particular choice of a distance metric, say MMD, can outperform another, say Wasserstein, in certain datasets. 

2. The justifications of the distance metric designs are mainly based on the experiments and it seems that the authors only conduct limited experiments, and compare with limited approaches. If there are theoretical justification of the proposed approach, I can accept with only limited experiments; however, this paper did not provide such kinds of analyses, and hence I think the experiments are not enough.

3. Overall, I think there is not enough contribution and novelty by just proposing different distance metrics to replace K-L divergence in MI-based clustering. There needs more analyses to justify or prove that which choice of the distance metric can perform well under which kinds of condition or datasets. Without that, it is not convincing for the reviewer that the proposed approach can be either with theoretical guarantees or useful for general classed of datasets.

4. Typo：
line 139 —> “which is possible to estimateSince”

questions: The authors mentioned “We argue that it is intuitive in clustering to compare the distribution of one cluster against the distribution of another cluster rather than the data distribution.” Could the authors explain more about this statement? Why it is more intuitive?
limitations: No.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors introduce measures of discrimenative distance between distributions that are alternatives to mutual information. These measures in the case of MMD and Wasserstein include notions of distance or similarity which can capture the shape of the data distribution. The authors use the proposed measures as objectives for clustering. Evaluation is conducted on supervised multi-class data sets using adjusted rand index as the measure of performance.

strengths_and_weaknesses: This work deepens our understanding of the clustering problem, an ill-defined and challening problem. By identifying a failure model inherent in using mutual information as an objective, this work opens the possibility to design better clustering algorithms. The proposed methods aim to achieve this through the incorporating of the discrimnative clustering hypothesis: clusters are characterized by a discrimnative model $p(y|x)$.

The MMD and Wasserstein GEMINI include distance information through $k(x,y)$ and $c(x,y)$, respectively. It seems that these regularize the cluster boundaries to be smooth in data space. It would be nice if this intuition could be confirmed/refuted beyond ARI values and supposition. Additionally, while learning the kernel or distance function is (I think) beyond the scope of this paper, it would be good for the authors to discuss what impact the GEMINI approach has on simulaneous kernel learning and clustering methods.

The authors demonstrate some degree of automatic selection of the number of clusters. But, if this is to be a major selling point of the method one would be inclined to set the $k$ to be very large and let the method pick out the correct value. The authors should report performance of such experiments. E.g. how well does the method do on MNIST when $k$ is set to 50 or 100? 

Demonstration on more real-world data sets would be welcome. 


Typos:
Table 1 MMD-OVA: the x's should be x_a or x_b. Should the y_a and y_b just be y?
Table 2 header: two I_W^OVA.

questions: Figure 1: What does "as the posterior turns sharper" mean? This statement is clarified later in the text, but the caption should stand on its own.

limitations: Nothing to add.
review_title: null

Meta-review: metareview: This work presents alternative measures of discriminative distance between distributions that are alternatives to mutual information, addressing the problems that arise from using mutual information in clustering tasks. Drawbacks of applying MI in clustering problems are discussed, and a replacement is proposed by generalizing the mutual information, introducing the generalized mutual information (GEMINI): a set of metrics for unsupervised neural network training. The authors demonstrate that GEMINI can outperform the traditional mutual information-based approach through several experiments.

This is a borderline paper: theoretical justifications and demonstrations on more real-world data sets would improve the paper. However,  the paper addresses an important problem and the results are of interest to the community, and overall the strength overweigh the weaknesses.


confidence: Less certain
Decision: Accept