Deep Clustering with Associative Memories

Authors: Bishwajit Saha, Dmitry Krotov, Mohammed J Zaki, Parikshit Ram
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: deep clustering, associative memories, representation learning, Hopfield networks

Abstract:
Deep clustering -- joint representation learning and latent space clustering -- is a well studied problem especially in computer vision and text processing under the deep learning framework. While the representation learning is generally differentiable, clustering is an inherently discrete optimization, requiring various approximations and regularizations to fit in a standard differentiable pipeline. This leads to a somewhat disjointed representation learning and clustering. Recently, Associative Memories were utilized in the end-to-end differentiable
$\texttt{ClAM}$ clustering scheme (Saha et al. 2023). In this work, we show how Associative Memories enable a novel take on deep clustering, $\texttt{DClAM}$, simplifying the whole pipeline and tying together the representation learning and clustering more intricately. Our experiments showcase the advantage of $\texttt{DClAM}$, producing improved clustering quality regardless of the architecture choice (convolutional, residual or fully-connected) or data modality (images or text).

Decision: reject
Reviewer scores: 6, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730659788563

thread_type:
official_review

<thread_object>
creator: reviewer_z3ff

<content>
summary:
This paper proposes DClAM, a deep clustering method that integrates the associate memory idea of ClAM into the deep learning pipeline. Their method involves fine-tuning an autoencoder model by taking multiple associative memory (AM) steps on encoded embeddings before decoding, which enables joint optimization of representation learning and clustering. The paper shows that DClAM outperforms existing deep clustering methods on eight datasets spanning image and text with multiple architectures.

strengths:
- I think the core idea of integrating AMs into the (deep) autoencoder pipeline via a differentiable loss is novel and interesting. The method is agnostic to architectural choices on the encoder and decoder.
- The paper comprehensively evaluates on diverse datasets, comparing against a range of clustering and deep clustering baselines.

weaknesses:
- There's a large body of work on vector-quantized VAEs [1], which seems related to DClAM in that they're both learning an autoencoder with a latent space with a discrete structure. One obvious difference is that this model isn't a VAE (doesn't consider a prior over latents). Still, I think the learned centers play a similar role to the learned vectors in a VQ-VAE; their objective is essentially yours, with only one AM step. I think the paper should discuss the connection in more detail.
- The paper could benefit from more detailed ablation studies on key hyperparameters. I think reporting sensitivity to (1) the number of AM steps and (2) the number of clusters would give a better sense of the method's tuning requirements. Also, what is the number of AM steps you use? I can't find it in Table 5.
- What does the "restart" hyperparameter mean? Does it mean you ran the entire pipeline with five random seeds and reported the best performance?
- How does this method compare to other deep clustering methods in terms of computational efficiency? Mainly, I'd like to better understand how computationally expensive the AM steps are.

[1] Van Den Oord, Aaron, and Oriol Vinyals. "Neural discrete representation learning." Advances in neural information processing systems 30 (2017).

Minor
- I think it's generally good to have the proposed method's explanation (sec 4) self-contained. A core component (eq 7) is explained separately in section 3; you might consider briefly re-stating this operator in section 4.

questions:
Please see questions in weaknesses section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661789040

thread_type:
official_review

<thread_object>
creator: reviewer_zsfx

<content>
summary:
This study presents DCLAM a deep clustering algorithm that internally uses Clustering with Associative Memory (CLAM) algorithm. The proposed method uses encoder-decoder architecture which is optimized by an associative memory-inspired loss function.  Finally, the method is evaluated in comparison with existing methods using silhouette coefficient (SC) and reconstruction loss over image and text datasets.

strengths:
The research integrates autoencoder and pretraining while optimizing the data in the latent space using the CLAM algorithm. Which is a good extension to CLAM. 
The approach tries to maintain a minimal reconstruction loss during the process of finding clusters.
The paper describes the technical details well which make it reproducible.

weaknesses:
The evaluation process only included SC although the ground truth labels are available for the used datasets. The SC alone can’t be enough especially with its underlying assumptions about the cluster’s distributions.
The reported SCs for all the experiments are constrained by 10% range of change in the reconstruction loss. This is limiting the space for clustering improvement during training. Which is not fair for clustering algorithms, especially those that don’t use the decoder anymore in the clustering process.
It is mentioned in the paper that “applying CLAM in a latent space learned by a pretrained autoencoder is not an effective strategy”. However, the results reported don’t test this hypothesis.
The proposed method requires a choice of three learning rates which are difficult to select over different applications.
The belief mentioned in the evaluation section of using unsupervised metrics contradicts with using the information of number of clusters which is unknown in many unsupervised applications.

questions:
Describing the method to be agnostic to architecture can be easily misunderstood. Because architecture is important according to the type of dataset. What could be deduced is that the method itself is flexible and can be integrated with different autoencoders architectures.
Evaluating the proposed method using ground truth-dependent metrics would make the results more reliable. Mitigating data leakage between training and evaluation can still be done without completely ignoring the ground truth information.
The paper lacks a solid justification why balancing the SC and reconstruction loss is important.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698472269

thread_type:
official_review

<thread_object>
creator: reviewer_agr2

<content>
summary:
The paper applies ClAM (Saha et al. 2023) to various neural architectures (CNN, MLP) and achieves competitive performances on various text and image clustering benchmarks. The main metric of concern is the Silhouette Coefficient (SC), commonly used in clustering literature for unsupervised clustering quality.

strengths:
- Paper is well written and easy to follow
- Very clear motivation, theoretical analysis appear to be correct, core algorithm clearly explained, and experiments are presented thoroughly

weaknesses:
- The work builds up on ClAM (Saha et al., 2023), which may limit its novelty.
- Arguably, since the work is concerned with deep clustering, baseline methods should include traditionally metric learning-based approaches.
- In contemporary literature standards, both the neural network trained in the work and the dataset are small. It is unclear whether the proposed method would generalize to more realistic network sizes and datasets.

questions:
- It seems like the entire (vision) transformer literature is missing from the analysis. This is not a critical concern, but it would be interesting to evaluate DClAM's performance on transformers across image and text tasks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730879605170

thread_type:
official_review

<thread_object>
creator: reviewer_whzj

<content>
summary:
This paper presents DClAM, a deep clustering method that builds on the previous ClAM approach by integrating it with deep clustering techniques, specifically through a deep autoencoder (AE). DClAM is architecture-agnostic, demonstrating strong performance across different autoencoder designs and dataset types. It achieves high clustering quality and low reconstruction loss in both ambient and latent spaces, surpassing previous methods.

strengths:
The paper demonstrates superior results across various benchmarks, highlighting the robustness of the proposed approach. Additionally, the authors conduct experiments across diverse domains and architectural types, effectively showcasing the generalizability of their method. The preliminary section provides a thorough overview of foundational concepts, enhancing the accessibility of the paper. Furthermore, the appendix includes detailed hyperparameter information, essential for validating experiments in this hyperparameter-sensitive area and supporting the reproducibility of the study.

weaknesses:
First and foremost, this paper resembles the CLAM paper strongly, particularly in the introduction section, which feels almost identical. Given that this work relies heavily on CLAM, it is essential to revise and reframe the introduction to clearly distinguish this approach as more than a direct application of CLAM with an autoencoder. Establishing this work's unique contributions will help clarify its originality and value.

While the authors claim that the method does not require $\gamma$ tuning, it does not appear to resolve the underlying parameter sensitivity issue. In fact, it introduces additional hyperparameters that need careful adjustment. The chosen values suggest that previous tuning decisions are not reusable across different datasets and models, requiring high tuning complexity. This is particularly challenging, given that model training for reconstruction must occur simultaneously with the tuning process.

 An essential missing experiment is an ablation study on the latent dimension \( m \). Evaluating this parameter, even for one of the models used, would provide valuable insights into the impact of latent dimensionality on performance.

questions:
How is the number of centers (clusters) chosen? It’s unclear whether this is a fixed number or if it corresponds to the number of classes in the dataset.

Why is reconstruction considered significant in this context? It seems that the assumption of minimal information loss in the latent space is a prerequisite for effective clustering. To my understanding, a low reconstruction loss simply indicates that good clustering may be achievable, rather than serving as a goal in itself.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732598767834

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response (with revised version)

comment:
We sincerely thank all reviewers for their valuable comments, which we
have thoroughly addressed in our responses below, as well as in the revised
manuscript that has been updated on openreview. All changes have been
highlighted in blue in the main text and the Appendix. We summarize our main
changes below.

As requested by reviewer whZj we have revised the introduction and related
work sections, to outline the key differences between DClAM and ClAM and
other deep clustering works. Likewise, we have added the ablation study by
varying the latent dimension $m$ for USPS in Appendix C.4, Fig 10. We have
also clarified that the number of clusters $k$ is not a hyperparameter (end of
sec A.2).

For reviewer AgR2, we note the fundamental differences between DClAM and
ClAM, as now done in the revised introduction and related works section. We
have added the relation to metric learning too.

For reviewer ZsFx, we have added new experiments on the Pareto frontier for
all the hyperparameter configurations for different vision-based datasets in
Appendix C.2, and Figures 5,6,7,8,9. We also show the Pareto optimal
parameters reported for the methods for the best SC and RL, and within the
10% thresholds, as reported in Tables 2,3,4 and Tables 8,9. These results
highlight how we thoroughly optimize the hyperparameters, and how we select
the final Pareto optimal performance values from the Pareto front to be
consistent and fair across all methods. The results clearly indicate that
DClAM offers the best clustering performance in terms of SC, as well as
having low reconstruction loss. It also performs very well on the
supervised NMI metric. In fact, for NMI, DClAM has the best value in 5 out
of the 8 datasets.

Finally, for reviewer Z3Ff, we include a discussion of VQ-VAEs in the related
work section. We also include ablation results on the effect of varying
the number of AM steps $T$ in Appendix C.4 and Figure 11, for both USPS and
FMNIST datasets.

We request the reviewers to reconsider their evaluation of our paper in
light of these revisions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733172014855

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers, Dear Area Chair,

Thanks a lot for taking the time to read our revised manuscript! We are very happy to hear that reviewers whZj and AgR2 are satisfied with our response and have updated their scores. 

There is still one reject score (3) from reviewer ZsFx, and we have not received any response from them regarding our rebuttal and revised manuscript. Since there is less than 24 hours left for us to be able to communicate with reviewers, we kindly ask reviewer ZsFx to consider increasing the numerical score for our submission.

We are thankful to reviewers whZj and AgR2 for raising their scores to 6. With the decision-making process soon underway, if whZj or AgR2 believe our paper is worthy of acceptance, we kindly ask considering a further increase of the score to a "full accept" (8). We remain available to resolve any outstanding issues that would bump our paper from a borderline to an accept in the remaining time dedicated for discussion.

Sincerely, Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733288506666

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Review and Response

comment:
We express our sincere gratitude to the Reviewers and Area Chairs for their dedicated time and constructive feedback. Here is a concise summary of the review and our responses for easy reference.
 
**Reviewer Acknowledgments:**

Our method, $\texttt{DClAM}$, has been recognized for its **Novelty** and **Superior Results**. Key highlights include:

- **Clear Motivation and Novelty:** $\texttt{DClAM}$ comes up with **very clear motivation** with precise **theoretical analysis** and explanation of the core algorithm [AgR2]. The core idea of integrating AMs into the (deep) autoencoder pipeline via a differentiable loss is **novel** and **interesting** [Z3Ff].

- **Superior Results, Generalizability and Reproducibility:** $\texttt{DClAM}$ achieves superior results across various benchmarks with diverse domains and architectural types, effectively showcases the **generalizability** of the method [whZj]. It describes the technical details well which make it **reproducible** [ZsFx]. Moreover, $\texttt{DClAM}$ is **agnostic** to architectural choices on the encoder and decoder that makes it easy to incorporate with any architecture [Z3Ff].

- **Quality of Writing:** The manuscript is praised as **"well written"** and **"easy to follow"** [AgR2], making it easily **accessible** to the field [whZj].
 
**Addressing Weaknesses:**

The reviewers raised concerns regarding the clarification of unique contributions of $\texttt{DClAM}$ [whZj, AgR2], hyperparameter tuning [whZj, ZsFx], more ablation studies [whZj, Z3Ff], generalization [AgR2] and evaluation metrics [ZsFx]. Our responses include:

- *Unique contributions of $\texttt{DClAM}$:* We clarified in this [comment](https://openreview.net/forum?id=Awsb8jhEx3&noteId=cs6uLwJvJB) the unique contributions of $\texttt{DClAM}$ and fundamental differences from $\texttt{ClAM}$ and other deep clustering methods.

- *Ablation studies:* Following suggestions from Reviewers whZj and Z3Ff, we have added the ablation studies by varying the **latent dimension $\textbf{m}$** for USPS and varying the number of **AM steps $\textbf{T}$** for both USPS and FMNIST datasets.

- *HP tuning and generalization:* We clarified [here](https://openreview.net/forum?id=Awsb8jhEx3&noteId=J7GWg2dRZb) that there are two main hyperparameters for $\texttt{DClAM}$, namely inverse temperature ($\beta$), and number of steps of AM ($T$). The number of AM steps ($T$) is usually 10-15. While $\texttt{DClAM}$ requires tuning $\beta$, it is a completely new strategy for deep clustering using associative memories that **outperforms** all the related baseline schemes. We clarified [here](https://openreview.net/forum?id=Awsb8jhEx3&noteId=zFRpWqtA2I) about the generalization of $\texttt{DClAM}$ to any network and dataset size.

- *evaluation metrics:* We clarified [here](https://openreview.net/forum?id=Awsb8jhEx3&noteId=q77ZFDaKHz)  and [here](https://openreview.net/forum?id=Awsb8jhEx3&noteId=ebazIWUcNv) that we also evaluated $\texttt{DClAM}$ on the supervised Normalized Mutual Information (NMI) metric where $\texttt{DClAM}$ **wins** for **5** out of the 8 datasets. We have also discussed why unsupervised metric like Silhouette Coefficient (SC) and reconstruction loss (RL) are necessary to evaluate unsupervised clustering methods and added new experiments on the Pareto frontier for all the hyperparameter configurations for different vision-based datasets to show how we selected best SC and RL across all datasets.

The responses were **acknowledged positively** by reviewer **[whZj](https://openreview.net/forum?id=Awsb8jhEx3&noteId=c0VelNjAMy)** and reviewer **[AgR2](https://openreview.net/forum?id=Awsb8jhEx3&noteId=dF0B4gr0fw)** and they have **increased** their score to **6**. We have not received any responses from reviewer ZsFx and Z3Ff while we have addressed all of their concerns.

**Revision Overview:**

- To enhance clarity, we have revised our introduction and related work sections, to outline the key differences between $\texttt{DClAM}$ and $\texttt{ClAM}$ and other deep clustering works.
- We have added the ablation study by varying the latent dimension $m$ for USPS in Appendix C.4, Fig 10. We have also clarified that the number of clusters $k$ is not a hyperparameter (end of sec A.2).
- We have added new experiments on the Pareto frontier for all the hyperparameter configurations for different vision-based datasets in Appendix C.2, and Figures 5,6,7,8,9.
- We have also included ablation results on the effect of varying the number of AM steps $T$ in Appendix C.4 and Figure 11, for both USPS and FMNIST datasets.

We are thankful for the valuable feedback from the reviewers, and we believe we have thoroughly addressed all the concerns raised by the reviewers in our responses and revised manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a method for deep clustering based on an autoencoder model which applies the gradient descent step of an associative memory clustering method (ClAM) in the latent space of the encoder.  The reviewers are largely borderline, along with one more negative review.  Many reviewers note that the method appears to be largely similar to ClAM with the main distinction being incorporating the ClAM gradient descent operator in the latent space of an autoencoder.  The high level approach of incorporating a clustering metric on the latent space of an autoencoder is a well-known strategy in the deep clustering literature (though it also can often result in ill-posed formulations as well if not done carefully).  Here the authors argue that their method enjoys the advantage of not requiring a balancing between the sum of an autoencoder reconstruction loss and a clustering loss in the latent space as their method instead directly employs the gradient descent operator of ClAM in the latent space to drive latent points towards cluster centers.  However, while this does eliminate the need for balancing a weighting hyperparameters between two losses, one is still required to choose a hyperparameter for the number of time steps (T) used in applying the ClAM gradient descent steps.  The authors also provide an experimental evaluation of their approach which gives reasonable results, but some reviewers note that the datasets used are relatively small by modern standards.

Overall, the reviewers are largely lukewarm on the paper, and I am unfortunately inclined to agree with the critiques of the reviewers and recommend rejection.  Using an autoencoder with a clustering promoting loss or operator in the latent space is well established in the deep clustering literature.  To meet the bar for publication I would expect to see either a well motivated formulation along with strong theoretical analysis as to the merits of the formulation or convincing experiments on large scale datasets.  Here the authors have taken steps in both directions, but neither has led to the reviewers being particularly convinced about the approach.  I would encourage the authors to consider developing more extensive theoretical motivations for their proposed approach or evaluating the approach on larger scale datasets and look forward seeing an improved version of the work in future meetings.