I-Con: A Unifying Framework for Representation Learning

Authors: Shaden Naif Alshammari, John R. Hershey, Axel Feldmann, William T. Freeman, Mark Hamilton
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: representation learning; unsupervised learning; clustering; dimensionality reduction; kmeans; contrastive learning; information theory

Abstract:
As the field of representation learning grows, there has been a proliferation of different loss functions to solve different classes of problems. We introduce a single information-theoretic equation that generalizes a large collection of mod- ern loss functions in machine learning. In particular, we introduce a framework that shows that several broad classes of machine learning methods are precisely minimizing an integrated KL divergence between two conditional distributions: the supervisory and learned representations. This viewpoint exposes a hidden information geometry underlying clustering, spectral methods, dimensionality re- duction, contrastive learning, and supervised learning. This framework enables the development of new loss functions by combining successful techniques from across the literature. We not only present a wide array of proofs, connecting over 23 different approaches, but we also leverage these theoretical results to create state-of-the-art unsupervised image classifiers that achieve a +8% improvement over the prior state-of-the-art on unsupervised classification on ImageNet-1K. We also demonstrate that I-Con can be used to derive principled debiasing methods which improve contrastive representation learners.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1729958868214

thread_type:
official_review

<thread_object>
creator: reviewer_anko

<content>
summary:
This paper provides a unified framework I-Con that can recover many state-of-the-art approaches in various ML tasks as special examples. The main idea behind the I-Con framework is to minimize the KL divergence between two distributions: a "neighborhood" distribution that is given by data and a proposed distribution that encodes some supervisory information, which is similar to the idea behind variational approaches.

strengths:
1. The paper is very well written and well organized, which conducts a comprehensive survey across different fields and insightfully extracts the similarities behind state-of-the-art methods.
2. It is novel to transfer the proposed distribution across domains and the improvement in performance shown in Section 4 is solid.

weaknesses:
1. The connection with variational Bayesian methods can be elaborated, where the main idea is to find a proper proposed distribution q_{\phi}.
2. This paper provides a comprehensive survey, but it would be better if the authors could have a paragraph to compare with other survey papers, e.g.
	- Tschannen, Michael, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. "On mutual information maximization for representation learning." arXiv preprint arXiv:1907.13625 (2019).

questions:
1. [Naming] In terms of I-Con = Information Contrastive learning, I was wondering if it would be more precise to relate the name to "divergence" or "cross-entropy" since the I-Con framework is essentially the KL minimization and nature of "contrastive learning" is not very intuitive without specification of the proposed distribution q_{\phi}.
2. [Experiments] It is interesting to the improvement in performance when transferring the ideas of proposed q_{\phi} across domains. I was wondering if the improvement is also significant beyond the domain of clustering. It would also be good if some related work that transfers the SOTA q_{\phi} across domains could be provided.
3. Please see "weakness" section. I'll be happy to adjust my score if the connection with variation Bayesian methods can be elaborated.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730196790830

thread_type:
official_review

<thread_object>
creator: reviewer_fvtp

<content>
summary:
This paper addresses the challenge of the variety of unsupervised representation learning methods by introducing I-Con, a unified interpretation grounded in a single mathematical framework. The authors provide a theoretical proof that I-Con unifies existing methods by minimizing the KL divergence between the supervisory and parameterized neighborhood distribution. Furthermore, the authors effectively transferred perspectives of various representation learning approaches into a state-of-the-art unsupervised classification network and showed both quantitative and qualitative improvements.

strengths:
- This paper discusses various methods in representation learning in terms for their proposal.
- The proposed framework is well demonstrated throughout the paper including theoretical evidences.
- The proposed method showed significant improvements compared to state-of-the-art unsupervised classification network.

weaknesses:
- The motivation for the need for a unified framework of (unsupervised) representation learning is not clear. As discussed in introduction section, “difficult to understand which particular loss function to choose for a given problem or domain”, does not directly match with the proposed method, as I-CON still contains the choice of $p_{\theta}$ for each existing method and does not provide a suitable loss function for given domain.
- The term ‘I-Con’ is ambiguous, as it is used with two different meanings. In section 3.1, term ‘I-Con’ represents a unified representation learning algorithm that can that can serve as the foundation for various methods. However in section 3.2, it is also used to describe integrated (new) representation learning method, causing confusion in the following results. To avoid confusion, use distinct terms to clearly differentiate between I-Con as a framework and I-Con as a specific method.
- Section 3.3 is not easy to follow, especially the sub-section ‘Adaptive neighborhoods’. Paraphrasing the explanations and providing a more detailed demonstration would enhance clarity for the readers.
- Random seeds are not considered in the experimental results. Providing results of multiple runs (e.g., 5 or 10) with standard variations (or error bars) would give stronger evidence for the claimed significant improvements.

questions:
### Questions
- Theoretical evidences for I-Con across various method are given (appendix). However, are there any experimental results in practical environments (e.g., InfoNCE on ImageNet) that shows the I-CON framework can reproduce (or improve) the original representation learning methods?
- In Figure 3, is there a reason for using different dataset (ImageNet-1K and subset of MNIST)? Why not use ImageNet for both the quantitative and qualitative results (or vice versa)?
- Is I-Con (Ours) stated in Table 2 result of integrated method discussed in Section 3.3, or is it the best among them? Table 3 shows results of each transferred intuitions, and results of I-Con (Ours) stated in Table 2 appears to be the best among them. Referring to the best among them as I-Con (Ours) in Table 2 may not be a fair comparison.
- As stated in Section 4.2, I-Con has claimed to have the benefit of not requiring additional hyper-parameter tuning compared to TEMI and SCAN. However, doesn’t I-Con require hyper-parameter tuning, such as /alpha and n-walks (Figure 4, Table 4), for notable improvements?
- DiNO[1] also showed results of k-NN classifiers[2] in the paper. Could results in Table 3 maintain its gains when using k-NN classifiers[2] as stated in [1]?
### Modifications
- Table 2 in the first line in Section 4.2, do not have a link to it.
- Like in t-SNE this addition helps… → Like in t-SNE, this addition helps…

[1] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´ e J´ egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021.

[2] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730390771162

thread_type:
official_review

<thread_object>
creator: reviewer_ioht

<content>
summary:
The manuscript attempts to build a universal framework for loss functions corresponding to different representation learning methods. It minimizes an integrated KL divergence between two marginal distributions: the supervisory and learned representations. The proposed framework encompass 11 machine learning methods, including dimensionality reduction, contrastive learning, clustering, and supervised learning, among others.

strengths:
One advantage is that it can leverage losses from other domains to enhance a particular domain method within this framework. For instance, in experiments, spectral clustering, t-SNE, debiased contrastive learning, and SCAN were utilized to develop an unsupervised image classification system.

weaknesses:
- It falls short of truly encompassing all representation learning methods, such as autoencoder-based representation learning, masked pretraining-based representation learning, and other general self-supervised representation learning methods. Therefore, the proposed framework does not entirely unify all representation learning methods. I believe that this unified framework is for clustering methods but not a generic representation learning framework.
- The motivation/benefits for unifying representation learning methods claimed in this paper do not entirely convince me personally. Could you give more discussions on whether this unified framework provides advantages beyond facilitating cross-domain insights? For example, does it provide a new method to address all limitations of different representation learning frameworks once?
- Typically, evaluating representation learning methods involves assessing if a model has learned good representations that generalize well across various downstream tasks. However, I did not observe such experiments.
- The choice of data augmentation strategies is crucial in contrastive learning methods. Are data augmentation-based contrastive learning methods also unified within this framework?
- Are there differences in the task purposes between clustering and contrastive learning tasks? My understanding is that clustering aims to discover different clusters, while contrastive learning aims to learn generic representations that can be easily transferred to different downstream tasks. Could you provide further discussion on this?

Additional minor suggestions:
- Figure 1 is not easily read. The cluster in the right part should be labeled as "D)".

questions:
Please see Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730446177473

thread_type:
official_review

<thread_object>
creator: reviewer_xjue

<content>
summary:
The paper presents a novel framework called I-Con, which unifies various machine learning methods, including clustering, contrastive learning, dimensionality reduction, and supervised learning, under a single equation based on the Kullback-Leibler (KL) divergence between two distributions: the supervisory and learned representations. The authors demonstrate that many existing methods are instances of this unified framework, providing a mathematical foundation that enables the transfer of ideas across domains. The paper also presents new state-of-the-art results in unsupervised image classification on the ImageNet-1K dataset, achieving an 8% improvement over previous methods. Multiple proofs and empirical evaluations back these claims, illustrating the broad applicability of I-Con to various learning problems.

strengths:
- Theoretical Contributions: The I-Con framework offers a unified theoretical foundation for a wide range of machine learning methods, showing that several existing approaches can be viewed as specific instances of a general KL divergence minimization problem. This not only provides a deeper understanding of these methods but also opens up the possibility of creating new methods by mixing and matching components.
- Comprehensive Theoretical Justification: The paper provides detailed mathematical proofs, connecting 11 different approaches to the I-Con framework. 
- Cross-Domain Applicability: By unifying disparate learning paradigms (clustering, dimensionality reduction, contrastive learning, etc.), the framework facilitates the transfer of techniques across domains, allowing innovations from one area (e.g., contrastive learning) to improve methods in another (e.g., clustering).
- Empirical Performance: The framework's potential is validated through unsupervised image classification on ImageNet-1K, where the I-Con-based method outperforms state-of-the-art methods, demonstrating its practical significance.

weaknesses:
- Innovation and Breadth: While the authors introduce the I-Con framework, which attempts to unify multiple existing learning methods, the actual novelty and breadth of this framework might be questionable. Many of the connections between the methods discussed have already been noted in previous works, and the core contribution seems to be more about integrating existing ideas rather than offering fundamentally new insights. Actually, most methods and loss functions can be implemented in a likelihood framework, which is equivalent to minimizing KL distance. 
- Limited Guidance According to The Theorem: Several techniques like adaptive neighborhoods, debiased contrastive learning, neighbor propagation are transfered to create a state-of-the-art unsupervised image classification system. However, these methods can be thought out without the I-Con framework as many articles have established. Actually, inspiring or taking ideas from other domains is natural during research. Thus, based on the theorem, it will be more valuable to discuss how to choose the mechanism according to the I-Con framework. 
- Limited Generalization and Insufficient Experimental Validation: While the paper provides some experimental results demonstrating improvements in unsupervised classification, these are mainly focused on clustering on ImageNet-1K. The generalizability of the I-Con framework to other applications and datasets is still underexplored, lacking broader experimental comparisons across a variety of tasks and datasets. 
I would be glad to improve my score if my concerns could be addressed.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732561480428

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We appreciate the reviewers thoughtful and earnest feedback and have used this feedback to improve the work. We also appreciated that reviewers though our paper had “practical significance” (xJUE), “very well written and well organized” (AnKo), “novel” (AnKo), and showed “significant improvement” (FvTp). We have provided a global summary of changes in this revision, as well as a specific rebuttal to each reviewer. We have also uploaded a revised version of the paper with many requested additions and fixes.
## 1- New Series of Experiments Demonstrating that I-Con improves Feature Learning and works on different backbones and datasets.

In a new series of experiments we show that I-Con not only improves clusterers, but can improve contrastive feature learners. As requested by reviewers this experiment also demonstrates that I-Con works on an additional backbone and datasets. We follow the contrastive feature learning experiment put forth by Hu et al. at ICLR 2023. In particular we train a ResNet34 network contrastively on CIFAR10 and evaluate its linear probe and KNN performance on both CIFAR10 and CIFAR100. We include a detailed description of this experimental setting and its results in the Supplement. In summary, we show that our distribution improvement methods like debiasing and replacing gaussian kernels with cauchy kernels dramatically improves feature quality both in terms of linear probe accuracy and KNN accuracy. 

| **Method**                          | **CIFAR10 (in distribution)** |                        | **CIFAR100 (out of distribution)** |                     |
|-------------------------------------|-------------------------------|------------------------|-------------------------------------|---------------------|
|                                     | Linear Probing               | KNN                    | Linear Probing                      | KNN                 |
| **$q_\phi$ is a Gaussian Distribution**                                                                                                 |
| SimCLR [Chen et al., 2020]          | 77.79                        | 80.02                  | 31.82                               | 40.27               |
| DCL [Chuang et al., 2020]           | 78.32                        | 83.11                  | 32.44                               | 42.10               |
| **Our Debiasing $\alpha = 0.2$**    | 79.50                        | 84.07                  | 32.53                               | 43.19               |
| **Our Debiasing $\alpha = 0.4$**    | 79.07                        | 85.06                  | 32.53                               | **43.29**           |
| **Our Debiasing $\alpha = 0.6$**    | 79.32                        | 85.90                  | 30.67                               | 29.79               |
|-------------------------------------|-------------------------------|------------------------|-------------------------------------|---------------------|
| **$q_\phi$ is a Student's t-distribution**                                                                                             |
| t-SimCLR [Hu et al., 2022]          | 90.97                        | 88.14                  | 38.96                               | 30.75               |
| DCL [Chuang et al., 2020]           | Diverges                     | Diverges               | Diverges                            | Diverges            |
| **Our Debiasing $\alpha = 0.2$**    | 91.31                        | 88.34                  | 41.62                               | 32.88               |
| **Our Debiasing $\alpha = 0.4$**    | 92.70                        | 88.50                  | **41.98**                           | 34.26               |
| **Our Debiasing $\alpha = 0.6$**    | **92.86**                    | **88.92**              | 38.92                               | 32.51               |


## 2- Proving that MDS, PCA (Linear Autoencoders), Masked Language Modeling, Spectral Clustering, Mocov3, and Triplet Loss are unified by I-Con

We introduce new theorems that unify 6 additional methods with the I-Con framework. In particular we show that MDS and SNE can be unified in the limit of large gaussian scales, which is a novel theoretical result first proved in this work. As requested by reviewer iHot, this result allows us to integrate linear autoencoders (PCA) into I-Con.  As requested by reviewer iHot we also show masked language modeling can be considered in the I-Con framework. Intuitively this  is a simple corollary of our existing proof connecting Cross Entropy with I-Con, except replacing class embeddings with embeddings for masked tokens. Finally, we show that the commonly used triplet loss for contrastive learning emerges as in the limit of small width gaussians in I-Cons InfoNCE connection. These additional proofs can be found in the supplement.
</content>
</thread_object>

</thread>
<thread>
creation date:
1745898021339

thread_type:
public_comment

<thread_object>
creator: ~aditya_ravuri1

<content>
title:
Questions to authors

comment:
Hello! I was just made aware of this work, and I'm really glad that there is increasingly more interest in probabilistically unifying representation learning algorithms! I have a few questions to the authors. (Also, this is based on an initial read, massive apologies in advance if I've misunderstood certain ideas!)

Very briefly:
  * what random variables are the distribution $p(j|i)$ over?
  * how does this compare to prior work in this direction?

To elaborate a bit further:
 1. There has been prior work in this direction, that achieves a very similar taxonomical end goal as yours. Would love to see a discussion of these and it'd also help one understand how your framework compares, and provide alternative routes to derive those methods! It's super interesting to see connections from other point of views.

    One such work is [1]. In this work, we also unify most classical dimensionality reduction methods as KL minimising algorithms, including UMAP, t-SNE, (dual-P)PCA, MDS, etc. In [2], we offer a different take on UMAP and t-SNE as MAP algorithms, but due to the results from [1], these can be interpreted variationally too.

    In a nutshell, in [1], we show that SNE, t-SNE and UMAP correspond to KL(q||p) minimisation algorithms, with Categorical/Bernoulli distributions on the data-pairs (the random variable in question corresponds to a slice of an adjacency matrix). [6] also find very similar results as these. Also there's been some really cool work trying to understand the nature of UMAP and t-SNE's true loss functions in [7, 8]. 

    Moreover, algorithms such as (dual-probabilistic) PCA, MDS, kPCA, Laplacian Eigenmaps, Locally Linear Embedding, etc. are all cases in ProbDR of a KL minimisation on a covariance/precision matrix (which is the random variable), endowed with a Wishart distribution [1].

    I realise that a KL can be defined in many different ways, over different _things_ (random variables), so perhaps what we talk about are approximations / different representations of the same thing, and a lot of my questions come from viewing these methods from a ProbDR point of view.

    **A question to the authors:** what is the random variable that $p(j|i)$ is defined on? Is this perhaps a discrete matrix-valued random variable that corresponds to data-pair similarity, or are the real-valued elements of $\mathcal X$ endowed with the distributions in Table 1?

 2. Questions on (t-)SNE:
     (Continuing from point 1) Table 1 suggests that the real-valued data points are endowed with a Gaussian distribution, but I struggle to understand Theorem 1 from this light. In theorem 1, it's easier to see p and q as Categorical distributions over a discrete choice (is data point j similar to data point i).

     Apologies if this seems pedantic, but it's I think important to iron this out. I've disagreed with the use of the word ``distribution'' in the original SNE and t-SNE papers, and prefer the terms 'Gaussian kernels' or 't/Cauchy kernels' for the choice of probabilities over categorical random variables, because I do not understand how such probabilities can be derived (and over what) assuming Gaussian data.

     The closest perhaps is Gaussian mixture model. Assuming that each data point is drawn one point at a time from a gaussian mixture:
     $$x_i | z_{\not i} \sim \mathcal N(\sum_{j \neq i} z_j * x_j, \sigma^2),$$
     $$ z \sim Categorical(\mathbf{1}/(n-1)), $$
     Then, I think that the posterior over the latent discrete variables $z$ would be such that:
     $$ \mathbb P(z_j=1 | x) = \exp(-0.5\sigma^{-1}||x_i - x_j ||^2) / \sum_{k \neq i} \exp(-0.5\sigma^{-1}\|x_i - x_k \|^2)$$
     In this case, I'd describe the data points as conditionally drawn from a mixture of Gaussians, and note that (as far as I can see), this does not mean that x_i jointly or even marginally have a Gaussian distribution.

     I believe that this is important, because if the data did jointly or even marginally have a Gaussian distribution, it'd be easy to relate them to Gaussian Process Latent Variable Models and PCA, and I've not seen an easy way to do this.

     Again, please do let me know if I'm missing something as I've always wondered why the original SNE and t-SNE papers refer to the ``distribution'' over the latents/data as Gaussian/Cauchy(t-) distributed.

     Taking a different route, if a random variable $y_i$ were defined as $y_i = \mathcal{I} (\|y_i - y_j \|^2 < \epsilon)$, the probabilities for $y_i$ do depend on the Euclidean distances between the data points, but these formulations are quite ugly and don't result in the nice form seen in t-SNE (here, I use the fact that the squared distances are marginally Gamma distributed). I apologise if this is easy to see, but it's not obvious to me what transformation applied to the real valued gaussian data to a set of categorical random variables results in probabilities of the kind that appear in SNE and t-SNE.

(Continued in the comment)
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a framework that unifies various representation  learning methods,including clustering, contrastive learning, dimensionality reduction, and supervised learning, as a minimization of the KL divergence between two distributions, this covers 15 known techniques.  Authors provide a method for distribution improvement methods such as debiasing and replacing gaussian kernels with cauchy kernels which improve feature quality both in terms of linear probe accuracy and KNN accuracy.Results on imagenet are provided , giving 8% improvement on previous methods.