I-Con: A Unifying Framework for Representation Learning
Authors: Shaden Naif Alshammari
John R. Hershey
Axel Feldmann
William T. Freeman
Mark Hamilton
Venue: ICLR 2025 Poster (2025)
Keywords: representation learning; unsupervised learning; clustering; dimensionality reduction; kmeans; contrastive learning; information theory
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We unify over 23 methods in contrastive learning, dimensionality reduction, spectral clustering, and supervised learning with a single equation.

Abstract: As the field of representation learning grows, there has been a proliferation of different loss functions to solve different classes of problems. We introduce a single information-theoretic equation that generalizes a large collection of mod- ern loss functions in machine learning. In particular, we introduce a framework that shows that several broad classes of machine learning methods are precisely minimizing an integrated KL divergence between two conditional distributions: the supervisory and learned representations. This viewpoint exposes a hidden information geometry underlying clustering, spectral methods, dimensionality re- duction, contrastive learning, and supervised learning. This framework enables the development of new loss functions by combining successful techniques from across the literature. We not only present a wide array of proofs, connecting over 23 different approaches, but we also leverage these theoretical results to create state-of-the-art unsupervised image classifiers that achieve a +8% improvement over the prior state-of-the-art on unsupervised classification on ImageNet-1K. We also demonstrate that I-Con can be used to derive principled debiasing methods which improve contrastive representation learners.

Published as a conference paper at ICLR 2025
I-CON: A UNIFYING FRAMEWORK FOR REPRESENTATION LEARNING
Shaden Alshammari John Hershey? MIT Goooge
Axel Feldmann'
William T. Freeman
Mark Hamilton
Microsoft
https://aka ms/ i-con
Supervisory Signal
Graph Kernel Weights
Uniform over K-Neighbors
Unitorm over Positive Pairs
Cross-Modial Pairs
Uniform over Classes
Date-Label Pairs
Student-T
Identity
SNE with Unifarm Affinities
InfoN [Bachmmn 2018)
CLIP (Radtood 2017n
(+ (Hinton 2002))
Dual t-SNE
Grapph Embeddiiggs
SimCL
Croes
(Chen 2020]
(Khosla 2020)
[Doad 1988)
Learned Representation
X-Sample CL [Bobal 2029]
LGSimCLR R Banon 2023)
MoCova
CMC [Tian 20200
(Chen 2021)
Gaussian g++00
PCA Pwaman 1801
Vi-Reg [Barces 20211
Average Mairgin CLIP
Averaze Margin SupCon
Gaussian
a+0
(Schrort 2010]
Triplint
Triplet CLIP
SupCon
Error rate
t-SimONE
Student-T
1-SNE [Van Moxten 2008)
Doubly 1-SNE
t-SNE Graph Embeddiin
SNE with Uniform Attinibles
1-CLIP
1-SupCan
Hamonic Loss (Book 2025)
t-SimCLR
Uniform on Clusters
Mcans Macqueen 1867)
I--Maan
Normalzzd
DCD [Yonn 2012)
InfoNCE
Supervised Clustering
(Shi 2000)
Clustoring [Duns]
Dimensionality Reduction
Cluster Learning
Unimodal SSL
Muttimodal SSL
Supervised Learning
Interpretation of Gaps
Figure 1: A "periodic" table of representation learning methods unified by the I-Con frame- work. By choosing different types of conditional probability distributions over neighbors, I-Con generalizes over 23 commonly used representation learning methods.
ABSTRACT
As the field of representation learning grows, there has been a proliferation of different loss functions to solve different classes of problems. We introduce a single Eefommtiion-theorede equation that generalizes large collection of mod- ern loss functions in machine learning. In particular, we introduce a framework that shows that several broad classes machine learning methods are precisely minimizing integrated KL divergence between two conditional distributions: the supervisory and learned representations. This viewpoint exposes a hidden information geometry underlying clustering, spectral methods, dimensionality duction, contrastive learning, and supervised learning. This framework enables the development of loss functions by combining successful techniques from across the literature. We not only present a wide array of proofs, connecting over 23 different approaches, but we also leverage these theoretical results create state-of-the- unsupervised image classifiers that achieve +8% improvement over the prior state-of-the-as seff-hheatttt on unsupervised classification on ImageNet- K. We also demonstrate that I-Con can be used to derive principled debiasing methods which improve contrastive representation learners.
INTRODUCTION
Over the past decade the field of representation learning has flourished, with new techniques, archi- tectures, and loss functions emerging daily. These advances have powered state-of-the-ee models in vision, language, and multimodal learning, often with minimal human supervision. Yet the field

---

Published as a conference paper at ICLR 2025
expands, the diversity of loss functions makes it increasingly difficult understand how different methods relate, and which objectives are best suited for a given task.
In this work, we introduce general mathematical framework that unifies wide range of representa- tion learning techniques spanning supervised. unsupervised. and self-supervised approaches under single information- theoreetc objective. Our framework, Information Contrastive Learning (I- Con), reveals that many seemingly disparate methods including clustering, spectral graph theory, contrastive learning, dimensionality reduction, and supervised classification are all special cases of the same underlying loss function.
While prior work has identified isolated connections between subsets of representation learning methods, typically linking only two or three techniques at a time (Sobal et al., 2025; Hu et al., 2023; Yang et al., 2022; Böhm et al., 2023; Balestriero & LeCun, 2022), I-Con is the first framework to unify over 23 distinct methods under single objective. This unified perspective not only clarifies the structure of existing techniques but also provides strong foundation for transferring ideas and improvements across traditionally separate domains.
Using I-Con, we derive new unsupervised loss functions that significantly outperform previous methods on standard image classification benchmarks. Our key contributions are:
We introduce I-Con, a single information-theorriic loss that generalizes several major classes of representation learning.
We prove 15 theorems showing how diverse algorithms emerge as special cases of I-Con.
We use I-Con to design debiasing strategy that improves unsupervised ImageNet- K accuracy by +8%, with additional gains of +3% % on CIFAR- 100 and +2 % on STL-10 in linear probing.
2 RELATED WORK
Representation learning spans a wide range of methods for extracting structure from complex data. We review approaches that I-Con builds upon and generalizes. For comprehensive surveys, see (Le-Khac et al., 2020; Bengio et al., 2013; Weng, 2021).
Feature Learning aims to derive informative diwdeemeniional embeddings using supervisory sig- nals such as pairwise similarities, nearest neighbors, augmentations, class labels, or reconstruction losses. Classical methods like PCA (Pearson, 1901) and MDS (Kruskal, 1964) preserve global struc- ture, while UMAP (Mclnnes et al., 2018) and t-SNE (Hinton & Roweis, 2002; Van der Maaten & Hinton, 2008) focus on local topology by minimizing divergences between joint distributions. I-Con adopts similar divergence-miniiizztioo minimization view.
Contrastive learning approaches such as SimCLR (Chen et al., 2020a), CMC (Tian et al., 2020), CLIP (Radford et al., 2021), and MoCo v3 (Chen et al., 2021) use positive and negative pairs, often built via augmentations or aligned modalities. I-Con generalizes these losses within a unified KL-based framework, highlighting subtle distinctions between them. Supervised classifiers (e.g., ImageNet models (Krizhevsky et al., 2017)) also yield effective features, which I-Con recovers by treating class labels as discrete contrastive points, bridging supervised and unsupervised learning.
Clustering methods uncover discrete structure through distance metrics, graph partitions, or con- trastive supervision. Algorithms like k-Means (Macqueen, 1967), EM (Dempster al., 1977), and spectral clustering (Shi & Malik, 2000) are foundational. Recent methods, including IIC (Ji et al., 2019), Contrastive Clustering (Li al., 2021), and SCAN (Gansbeke al., 2020), leverage invari- ance and neighborhood structure. Seacher-studeng models such as TEMI (Adaloglou et al., 2023) and EMA-based architectures (Chen et al., 2020b) enhance clustering further. I-Con encompasses these by aligning a clustering-induced joint distribution with a target distribution derived from sim- ilarity, structure, or contrastive signals.
Unifying Representation Learning has been explored through connections between contrastive learning and t-SNE (Hu et al., 2023; Böhm et al., 2023), equivalences between contrastive and cross-entropy losses (Yang et al., 2022), and relations between spectral and contrastive methods (Balestriero & LeCun, 2022; Sobal et al., 2025). Other efforts, like Bayesian grammar models (Grosse et al., 2012), offer probabilistic perspectives. Tschannen et al. (Tschannen al., 2019) emphasized estimator and architecture design in mutual information frameworks but stopped short of broader unification.
2

---

Published as a conference paper at ICLR 2025
Learned Representations
Learned Conditional Distribution
Mapper
fa
DRaCpG p Pllati
djj
Supervisory Conditional Distribution
Supervisory Signal
P(C
Ggg. Gaussion, Paired Inputs and Classes, Angmentation Pairs
High-level I-Con architecture.
A] Spatial
Discrete
CJ Cluster
Graph
"idd"
Choske
Gauuadionn Studhen-T
K-NN
Positive Pairs
Cross-Modal 6.8 image--oee
Labels
Neightsore shared cluster membership
Neighbars datermined by graph connectivity
Naightos brwerd dietsnce
g. data sugrertations
R.g. Imaggga Iniedd
calagory protetypea
(b) Illustrative examples distribution families for pe or 9a.
Figure 2: Overview of the I-Con framework. (a) Alignment of learned and supervisory distribu-
tions. (b) Common distribution families in I-Con's formulation.
While prior work links subsets of these methods, I-Con, to our knowledge, is the first to unify supervised, contrastive, clustering, and dimensionality reduction objectives under a single loss. This perspective clarifies their shared structure and opens paths to new learning principles.
METHODS
The I-Con framework unifies multiple representation learning methods under a single loss function: minimizing the average KL divergence between two conditional "neighborhood distributions" that define transition probabilities between data points. This information-theoretiic objective generalizes techniques from clustering, contrastive learning, dimensionality reduction, spectral graph theory, and supervised learning. By varying the construction of the supervisory distribution and the learned distribution, I-Con encompasses a broad class existing and novel methods. We introduce I-Con and demonstrate its ability to unify techniques from diverse areas and orchestrate the transfer of ideas across different domains, leading to a state-of-ther -f-the-r-- unsupervised image classification method.
3.1 INFORMATION CONTRASTIVE LEARNING
Let i, X' be clements of dataset X, with probabilistic neighborhood function p(jli) defining a transition probability. To ensure valid probability distributions, p(jli) 2 0 and JeEx P(jli) 1. We parameterize this distribution by 8 E θ, to create a learnable function (o()).). Similarly, we define another distribution qo (ili) parameterized by Φ E The core I-Con loss function is then:
[ DxL (po(11))las(1i)) JeEx
ff / Po(jli) log Po(jli) (o()))
C(B, x)
In practice, is typically fixed "supervisory" distribution, while 4s is learned by comparing deep network representations, prototypes, or clusters. Figure 2a illustrates this alignment process. The optimization aligns 9e with p. minimizing their KL divergence. Although most existing methods optimize only qs I-Con also allows learning both pa and qa. although one must take care to prevent trivial solutions.

---

Published as a conference paper at ICLR 2025
3.2 UNIFYING REPRESENTATION LEARNING ALGORITHMS WITH I-CON
Despite the incredible simplicity of Equation 1, this equation is rich enough to generalize several existing methods in the literature simply by choosing parameterized neighborhood distributions pa and do shown in Figure 1. We categorize common choices for po and qo in Figure 2a.
Table summarizes some key choices which recreate popular methods from contrastive learn- ing (SimCLR, MOCOv3, SupCon, CMC, CLIP, VICReg), dimensionality reduction (SNE, t-SNE, PCA), clustering (K-Means, Spectral, DCD, PMD, and supervised learning (Cross-Entropp and Har- monic Loss). Due limited space, we defer proofs of each of these theorems to the supplemental material. We also note that Table is not exhaustive, and we encourage the community to explore whether other learning frameworks implicitly minimize Equation 1 for some choice of p and q.
3.2.1 EXAMPLE: SNE, SIMCLR, AND K-MEANS
While I-Con unifies a broad range of methods, we illustrate how different choices of p and recover well-known techniques such SNE, SimCLR, and K-Means. Full details are in the appendix.
SNE as "neighhors remain neighbors." Stochastic Neighbor Embedding (SNE) is classic example. Given Rdxa with points in dimensions, SNE learns a low-dimensional representation é Rmxn typically m < d. To preserve local structure, p(j i) is de- fined by placing a Gaussian around each high- dimensional point and ( i) by placing a Gaussian around d;., Minimizing the average KL divergence between these distributions en- sures that points close in the original space re- main close in the embedded space.
High Dimensional Data
Low Dimensional Embeddings
p(jl
Gaussaiin
Dim
Lookup
(4) Gauasian contered fe(a)))
Aos fa(x)
Dim
SNE model ICon
target dist Gaussian (sigma 21,
learned dist Gaussian (sigma 1),
mapper Embadding (num, embedeinga-lyy dim-m))
(a) SNE (dimensionality reduction)
RGB Images
Embeddings
SimCLR as "augmeentatoons of the same image are neighbors." Contrastive learning methods like SimCLR and SupCon instead use class labels. Here, p(j i) 1 if is an augmentation of i (and 0 otherwise). In the embedding space, {a( | i) is defined via
P010)) - REI,) aase in the same class]
(o(e)))))))) pentieee at (y( on sphwrs
fx(x)
Mapper
(x))
(x
SimCLR model
Gaussian-like distribution based on cosine sim- ilarity. Minimizing their KI. divergence en- courages images from the same scene to cluster together.
target dist Augmentation (aum, views 2)v
learned Gaussian (signwe3. oeLrice looo")
mapper ResNet jembeddf ng (dimed))
(b) SimCLR (contrastive learning)
Features
Cluster Probabilities
K-Means as "points that are close are mem- bers of the same clusters." Clustering- based approaches like K-Means and DCD follow similar recipe. The distribution p(j i) is again Gaussian-based in the original space, while qsli i) reflects whether points are assigned to the same cluster in the learned representation. Minimizing KL divergence aligns these clus- ter assignments with the actual neighborhood structure in the data. Methods like K-Means in- clude an entropy penalty to enforce hard prob- abilistic assignments, as shown Theorem 13, whereas methods like DCD do not include it.
P(II)) = Gaussian
at
Chuster Proh.
Dim
Lookup
(antss)AA = plptepddedd dinted
fa
(x))
(x())
Cluster
KNeans model ICon (
target dist
Gaussian (signs 1),
learned dist Clustaringgni.fomm
mapper Embedding (num enoeddi iga-ll,, din-n)
(c) K- M-Manss (clustering)
Figure 3: Examples of methods special cases of I-Con via different choices of p and g, with corre- sponding code-style configurations.
3.3 CREATING NEW REPRESENTATION LEARNERS WITH I-CON
The I-Con framework unifies various approaches to representation learning under a single mathe- matical formulation and, crucially, facilitates the transfer of techniques among different domains.
4

---

Published as a conference paper at ICLR 2025
Method
Choice of po (5 (1)
Choice of (1)
(A) Dimensionality Reduction
SNE (Hinton & Rowcis, 2002)
Gaussian over learned low dimensional points, do
Gaussian over data points, 2;
exp(-1l0 : - $1113)
Theorem
Dktt exp(- |6 1x
ISSN (Van der Maaten & Hinton, 2008) Corollary
exp( - || - /20) Enzzs exp(- - 1|2: - =xx 113 /2o?)
Cauchy distribution over os
(1 + I| ولاه 113)
Ex + 16 - Φn 12
PCA (Pearson, 1901)
Wide Gaussian on linear projection features. f (z1) exp( 11 (=+) (x+) 1 ( ) 12 /202) 700 Ensi exp( 11 (x;) So (mx )1 2
1[;= 5] 5]
lim
Theorem 2
(B) Contrastive Learning
Gaussian on deep normalized features
InfoNCE Loss (Bachman et al., 2019)
exp ( (x)))) (2x
Theorem 3
Σ (x (xx))
Gaussian on deep leatures (1 neg, sample, o - 0)
Triplet Loss (Schroff ct al., 2015)
exp( ( 1f (-)=)) f 11222) 2
1. Z [ and j are positive pair]
Theorem
exp (- (-11 115 (1
2202)
RE
t-SimCLR, t-SimCNE
Student- on deep Teatures
(1+16) /2) (v)) (v+
(Hu et al., 2023; Böhm et al., 2023)
Corollary 2
3 kkg 160 2x 12 (x+1)/2
VICReg
Wide Gaussian on learned features
without covariance term
exp(-I1s (z, (2,
(Bardes et al., 2021)
lim Enys exp(- (1, (
Theorem 4
SupCon
1
(Khosla et al., 2020)
1 [i and j have same class]
Theorem 6
Gaussian on corresponding embeddings exp(ge(xg) go (uz))
Gaussian on deep normalized features
X-Sample
exp(Sa(us) (xg))
(Sobal et al., 2025)
Theorem?
exp(go (x;) T (ax))
cxp(f((x;) f (xx))
LGSimCLR
1 [ is among zy's le nearest neighbors]
(E) Banani et al., 2023)
2
CMC CLIP
exp(So((s) So(w))) ΣKeV, (xp(f((2)) for (ox))
(Tian et al., 2020)
1 [i.j pos. pairs, V2 # Vyl
Theorem 8
z
(C) Supervised Learning
Supervised Cross Entropy (Good, 1963) Theorem 9
exp (x;) 25)
Indicator over classes
Ekec exp (fo (z;) 2b)
Harmonic Loss
Student-T on deep features and class prototypes
(Back ct ul., 2025)
1 [ belongs to class j]
0 Ekec (z
lim (xx) ( II=
Theorem 10
Masked Lang, Modeling
1
(s ,1)
(Devlin et al., 2019)
# [Context i precedes token j]
Theorem 11
Z
ΣKEO exp (xxs) xbk
(D) Clustering
Probabilistic k-Means
Gaussians on datapoints
(Macqueen, 1967) Theorem 13
Intra-cluster uniform probability
exp( 1 -; -- -; 12 22
Σkgi exp( 1>; 2x )
Spectral Clustering
m
p(fo and fo (zz) ) c)
Gausslans on spectral embeddings
(Ng et al., 2001)
Σ
exp( ( | [=:-=a = 112 22/2x²
Corollary 4
Elsize of cluster c)
Σkyi exp( 11 2x 1 220?)
Intra-cluster uniform probability weighted by degrce
Normalized Cuts (Shi & Malik, 2000)
exp exp(w;; (wa./ / /d,) Σn exp(wii i )x)
p(fo and fo (*r ) c) dj
Gaussians on graph weigths
Theorem
E[degree of cluster c]
Clusteluustr Unitorm Probability o (Jo(z;) and fo fo(2;) in c)
PMI Clustering
(Adaloglou et al., 2023) Theorem 15
[[jiskNNNof
Elsize of cluster c]
Debaised InfoNCE Clustering
Debiased Graph through Uniform Distribution and Neighbor Propagation
Σ
(1 ax x))p( (So (Jo (wx) and So (x,) ) in c)
(ours)
Elsize of cluster c)
N
Table 1: I-Con unifies representation learners under different choices of (o(ii) and qs (jlb).
Proofs of the propositions in this table can be found in the supplement.
5

---

Published as a conference paper at ICLR 2025
Gausaian Distribution
Student-T Distrbution
Uniferm Nearect Neighbors
k
(1+ by )
11 nearest neighbors ofx, Notherwise
exp(l% /2""/
(a) Continuous distance-based distributions control neighborhood width via hyperparameters.
Original Graph
Expanding Neighborhood with Uniform Distribution
Expanding bood with Neighbor Propagation
k
P(1)) (1 a)p(1+++++++++)))
P + p² +.. pk
(b) Graph-based distributions expand neighhorhoods through structural strategies.
Figure 4: Neighborhood adaptation in continuous and discrete settings. (a) Distance-bast distributions modulate neighborhood "width" via parameters such a. (b) Graph-baset approaches modify the connectivity directly, often via random walks or added edges, thereby broadening each node's neighborhood.
For instance, trick from contrastive learning be applicd to clustering -or vice versa. In this paper, we demonstrate how surveying modern representation methods enables the development of clustering and unsupervised classification algorithms that surpass previous performance levels. Specifically, we integrate insights from spectral clustering, t-SNE, and debiased contrastive learning (Chuang al., 2020) to build a fate-of-the--e unsupervised image classification pipeline.
3.3. DEBASING
Debiased Contrastive Learning (DCL) addresses the mismatch caused by random negative sampling in contrastive learning, especially when the number of classes is small. Randomly chosen negatives turn be positives, introducing spurious repulsive forces between similar examples. Chuang et al. (2020) rectify this by subtracting out such false repulsion terms and boosting attractive forces, substantially improving representation quality. However, their method modifies the softmax itself, implying that Ojl: is no longer a genuine probability distribution and making it more difficult to extend the approach clustering or supervised tasks.
Our view, grounded in the I-Con framework, suggests a simpler and more general alternative: rather than adjusting the learned distribution Ojju we incorporate additional "uncertainty" directly into the supervisory distribution p(jli). This preserves Zjii as valid distribution and keeps the method applicable to tasks beyond contrastive learning.
3.3.2 DEBIASING THROUGH A UNIFORM DISTRIBUTION
Our first example adopts a simple uniform mixture:
P01) = (1 (1-a) + N
where the local neighborhood size, and a specifies the degree of mixing. This approach assigns small probability mass & to each "negative" sample, thereby mitigating overconfident allocations. In supervised contexts, this is analogous to label smoothing (Szegedy et al., 2016). In contrast, Chuang al. (2020) adjust the softmax function itself while retaining one-hot labels.
Another way to view this method is through the lens of heavier-tailed or broader distributions. By adding uniform component, minor the idea in t-SNE's Student-t distribution (Van der Maaten & Hinton, 2008), which allocates greater mass to distant points. In both cases, expanding the distri- bution reduces the chance of overfitting narrowly defined set of neighbors.

---

Published as a conference paper at ICLR 2025
Caree
0.
3000
Count
Valleation
54-
13
Debias Confficient
1640
5000
51
256
532
1024
2048
4096
3.0 0.2
641
266
0.0
1.5
0.0
0.2
0A
0.6
0.8
100
Batch Size
Largest Largest Assignment Probability
Largest Assignment Probability
Figure 5: Left: Debiasing cluster learning improves performance on ImageNet- IK across batch sizes. Center: Distribution of maximum predicted probabilities for the biased model (a = 0) show- ing poor calibration, with overconfident predictions. Right: Distribution of maximum predicted probabilities for the debiased model (a 0.4), demonstrating improved probability calibration. Debiased training alleviates optimization stiffness by reducing the prevalence of saturated logits, mitigating vanishing gradient issues, and fostering more robust and well-calibrated learning dynam- ics.
Empirical results Tables 3, Figures 5, and show that this lightweight modification consistently improves performance across various tasks and batch sizes. It also "relaxes" overconfident distri- butions, much like label smoothing in supervised cross entropy, thereby guarding against vanishing gradients.
3.3.3 DEBIASING THROUGH NEIGHBOR PROPAGATION
A second strategy applies graph- based expansions. As shown in Table 1, replacing k-Means' Gaus- sian neighborhoods with degrte-weighted k-nearest neighbors recovers spectral clustering, which known for robust, high-quality solutions. Building on this idea, we train contrastive learners with KNN-based neighborhood definitions. Given the ncarest-neighhor graph, we can further expand by taking longer walks, analogous Word-Graph2 Vec tsNET (Li et al., 2023; Kruiger et al., 2017), a process we term neighbor propagation.
Formally, let P be the conditional distribution matrix whose entries P.j p(æz, x;)) define the probability of selecting zj as a neighbor of z;. Interpreting P as the adjacency matrix of the training data, can smooth it by summing powers of P up to length k:
P P P2 +... pk
We can further simplify this by taking uniform distribution over all points reachable within steps, denoted by:
Po x
0].
where T[.] is the indicator function. This walk-based smoothing broadens the effective neighbor- hood, allowing the model learn from denser supervisory signal.
Tables 3 and 4 confirm that adopting such propagation-basee approach yields significant improve- ments in unsupervised image classification, underscoring the effectiveness of neighborhood expan- sion as debiasing strategy.
4 EXPERIMENTS
In this section, demonstrate that the I-Con framework offers testable hypotheses and practical insights into self-supervised and unsupervised learning. Rather than aiming only for state-of-ther oftt-----heertt performance, goal is to show how I-Con can enhance existing unsupervised learning methods

---

Published as a conference paper at ICLR 2025
by leveraging unified information -heorrrtc approach. Through this framework, we also highlight the potential for cross-pore pross-pollination between techniques in varied machine learning domains, such as clustering, contrastive learning, and dimensionality reduction. This transfer of techniques, enabled by I-Con, can significantly improve existing methodologies and open new avenues for exploration.
We focus our experiments on clustering because it is relatively understudied compared contrastive learning, and there are a variety of techniques that can now be adapted to this task. By connecting established methods such as k-Means, SimCLR, and t-SNE within the I-Con framework, we uncover a wide range of possibilities for improving clustering methods. We validate these theoretical insights experimentally, demonstrating the practical impact of I-Con.
We evaluate the I-Con framework using the ImageNet-IK dataset (Deng et al., 2009), which consists of 1,000 classes and over one million high-resolution images. This dataset is considered one of the most challenging benchmarks for unsupervised image classification due to its scale and complex- ity. To ensure fair comparison with prior works, we strictly adhere to the experimental protocol introduced by (Adaloglou et al., 2023). The primary metric for evaluating clustering performance is Hungarian accuracy, which measures the quality of cluster assignments by finding the optimal alignment between predicted clusters and ground truth labels via the Hungarian algorithm (Ji et al., 2019). This approach provides a robust measure of clustering performance in an unsupervised con- text, where direct label supervision is absent during training.
For feature extraction, we utilize the DiNO pre-trained Vision Transformer (ViT) models in three variants: ViT-S/14, ViT-B/14, and ViT-L/14 (Caron et al., 2021). These models are chosen to ensure comparability with previous work and to explore how the I-Con framework performs across varying model capacities. The experimental setup, including training protocols, optimization strategies, and data augmentations, mirrors those used in TEMI to ensure consistency in methodology.
The training process involved optimizing a linear classifier on top of the features extracted by the DiNO models. Each model was trained for 30 epochs, using ADAM (Kingma & Ba, 2017) with batch size of 4096 and an initial learning rate of Ie-3. We decayed the learning rate by a factor of 0.5 every 10 epochs to allow for stable convergence. We do not apply additional normalization to the feature vectors. During training, we applied a variety of data augmentation techniques, in- cluding random re-scaling, cropping. color jittering, and Gaussian blurring, to create robust feature representations. Furthermore, to enhance the clustering performance, we pre-computed global near- est neighbors for each image in the dataset using cosine similarity. This allowed us to sample two augmentations and two nearest neighbors for each image in every training batch, thus incorporating both local and global information into the learned representations. We refer to derived approach as "InfoNCE Clusting" in Table 2. In particular, we use a supervisory neighborhood comprised of augmentations, KNNs (A: = 3), and KNN walks of length 1. We use the "shared cluster likelihood by cluster" neighborhood from k-Means (See table for a more detailed Equation) as our learned neighborhood function to drive cluster learning.
4.1 BASELINES
We compare our method against several state-of-the-art clustering methods, including TEMI, SCAN, IIC, and Contrastive Clustering. These methods rely on augmentations and learned representations, but often require additional regularization terms or loss adjustments, such as controlling cluster size or reducing the weight of affinity losses. In contrast, our "Con-based loss function is self- baf--balnncin and does not require such manual tuning, making a cleaner, more theoretically grounded approach. This allows us to achieve higher accuracy and more stable convergence across three different-sized backbones.
4.2 RESULTS
Table 2 compared the Hungarian accuracy of Debiased InfoNCE Clustering across different DINO variants (ViT-S/14, ViT-B/14, ViT-L/14) and several other modern clustering methods. The I-Con framework consistently outperforms the prior state-of-the-an method across all model sizes. Specif- ically, for the DINO ViT-B/14 and ViT-L/14 models, debiased InfoNCE clustering achieves signif- icant performance gains of +4.5% and +7.8% in Hungarian accuracy compared to TEMI, the prior state-of-the-ar ImageNet clusterer. We attribute these improvements to two main factors:
8

---

Published as a conference paper at ICLR 2025
Method
DiNO ViT-S/14
DiNO ViT-B/14
DiNO ViT-L/14
k-Means
51.84
52.26
53,36
Contrastive Clustering
47.35
55.64
59.84
SCAN
49.20
55.60
60.15
TEMI
56.84
58.62
Debiased InfoNCE Clustering (Ours)
57.8 = 0.26
64.75 + 0.18
67.52 + 0.28
Table 2: Comparison of methods on ImageNet- 1K clustering with respect to Hungarian Accuracy. Debiased InfoNCE Clustering significantly outperforms the prior state-of-the-ard TEMI. Note that TEMI does not report results for ViT-L.
Unlike TEMI SCAN, which require hand-tuned regularizations (e-g., bal- ancing cluster sizes or managing the weight of affinity losses), I-Con's loss function automatically balances these factors without additional regularization hyper-parameter tuning as we are using the exact same clustering kernel used by k-Means. This theoretical underpinning leads to more robust and accurate clusters.
Self-Balancing Loss:
Dooss-Dmmain Insights: I-Con leverages insights from contrastive learning to refine clustering by looking at pairs of images based on their embeddings, treating augmentations and neighbors similarly. This approach, originally successful in contrastive learning, translates well into clustering and leads to improved performance on noisy high-dimensional image data.
4.3 ABLATIONS
We conduct several ablation studies to experimentally justify the architectural improvements that emerged from analyzing contrastive clustering through the I-Con framework. These ablations focus on two key areas: the effect of incorporating debiasing into the target and embedding spaces and the impact of neighbor propagation strategies.
Method
DINO VIT-S/14
DiNO ViT-B/14
DiNO ViT-L/14
Baseline
55.51
63.03
65.70
Debiasing
57.27 0.07
63.72 + 0.09
66.87 + 0.07
KNN Propagation
58.45 0.23
64.87 0.19
67.25 0.21
EMA
57.8 + 0.26
64.75 = 0.18
67.52 + 0.28
Table 3: Ablation study of new techniques discovered through the I-Con framework. We compare ImageNet- 1K clustering accuracy across different sized backbones.
Method
DiNO ViT-S/14
DINO VIT-B/14
DiNO ViT-L/14
Baseline
55.51
63.03
65.72
KNNS
56.43
64.26
65.70
+ 1-walks on KNN
58.09
64.29
65.97
+ 2-walks on KNN
57.84
64.27
67.26
3-walks on KNN
57.82
64.15
67.02
Table 4: Ablation Study on Neighbor Propagation. Adding both KNNs and walks length or on the KNN graph achieves the best performance.
We perform experiments with different levels of debiasing in the target distribution, denoted by the parameter a, and test configurations where debiasing is applied to the target side, both sides (target and learned representations), with the optimal value typically around a = 0.6 to a = 0.8, particularly when applied to both sides of the learning process. This method is similar to how debiasing work in contrastive learning by assuming that each negative sample has a non-zero probability (a/N) being incorrect. Figure shows how changing the value of improves performance across different batch sizes.
none. As seen in Figure 6, adding debiasing improves performance,

---

Published as a conference paper at ICLR 2025
DINO
DINO
Large
as Thrge: Dissriuutoon es Bloth burions
67.0
57.5
63.8
66.8.
Validation Accuracy
$7.0
66.6-
66.4
63.4
66.2-
56.0
65.8
63.0
65.6-
0.4 Debiassed
0.6
Deblased
Figure 6: Effects of increasing the debias weight on the supervisory neighborhood (blue line) and both the learned and supervisory neighborhood (red line). Adding some amount of debiasing helps in all cases, with double debiasing yielding the largest improvements.
In a second set of experiments, shown Table 4, we examine the impact neighbor propagation strategies. We evaluate clustering performance when local and global neighbors are included in the contrastive loss computation. Neighbor propagation, especially at small scales (s = and = 2), significantly boosts performance across all model sizes, showing the importance of capturing local structure the embedding space. Larger neighbor propagation values (c.g., & = 3) offer diminishing returns, suggesting that over-propagating neighbors may dilute the information from the nearest, most relevant points. Note that only DiNO-L/14 showed preference for large step size, and this is likely due its higher k-nearest neighbor ability, so the augmented links are correct.
Our ablation studies highlight that small adjustments in the debiasing parameter and neighbor prop- agation can lead notable improvements that achieve state-of-the-art result with simple loss function. Additionally, sensitivity to a and propagation size varies across models, with larger mod- els generally benefiting more from increased propagation but requiring fine-tuning of a for optimal performance. We recommend using ~ 0.6 to a a 0.8 and limiting neighbor propagation to small values for balance between performance and computational efficiency.
5 CONCLUSION
In summary, we have developed I-Con: a single information thhooreticcc equation that unifies broad class of machine learning methods. We provide over 15 theorems that prove this assertion for many of the most popular loss functions used clustering, spectral graph theory. supervised and unsuper- vised contrastive learning, dimensionality reduction, and supervised classification and regression. We not only theoretically unify these algorithms but show that our connections can help us discover new state-of-the-art methods, and apply improvements discovered for a particular method any other method in the class. We illustrate this by creating a new method for unsupervised image clas- sification that achieves +8% improvement over prior art. We believe that the results presented in this work represent just a fraction of the methods that are potentially unify-able with I-Con, and we hope the community can use this viewpoint to improve collaboration and analysis across algorithms and machine learning disciplines.
A Akknowledgments This research was partially sponsored by the Department of the Air Force Arti- ficial Intelligence Accelerator and was conducted under Cooperative Agreement Number FA8750- 19-2-1000, as well as NSF CIF 1955864 (Occlusion and Directional Resolution in Computational Imaging). We also acknowledge support from Quanta Computer. Additionally, would like thank Phillip Isola, Andrew Zisserman, Yair Weiss, Justin Kay, and Shivam Duggal for valuable discus- sions and feedback.
10

---

Published as a conference paper at ICLR 2025
BIBLIOGRAPHY
Nikolas Adalogiou, Felix Michels, Hamza Kalisch, and Markus Kollmann. Exploring the limits of deep image clustering using pretrained models. arXiv preprint arXiv:2303.778%6 2023.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
David Back, Ziming Liu, Riya Tyagi, and Max Tegmark. Harmonic loss trains interpretable ai models. arXiv preprint arXiv:2500.000 06628 2025.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning cover global and local spectral embedding methods. Advances in Neural Information Processing Systems, 35:26671-266666 2022.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varrane-iivrriinc--vvvvineneee rww.aroaaaaiiieeeggeeerrrrattooi for super-sd learning. arXiv preprint arXiv:2105.04906. 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1825 2013.
David Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: review for statisti- cians. Journal of the American statistical Association,
2017.
Jan Niklas Böhm, Philipp Berens, and Dmitry Kobak. Unsupervised visualization of image datasets using contrastive learning. International Conference on Learning Representations, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021. URL https:
/ www.aii.orggaaaeeeeeeeaaaoeeeeee 2104. 14294.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020b.
Xinlei Chen*, Saining Xie*, and Kaiming He. An empirical study training self-supervised vision transformers. arXiv preprint arXiv:2104.00 02057, 2021.
Ching-Yao Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. De- hat///b///// /2077. 00224.
biased contrastive learning, 2020. URL
Keenan Crane. Clarisse Weischedel. and Max Wardetzky. Geodesics in heat: new approach to
computing distance based on heat flow, ACM Transactions on Graphics (TOG), 32,
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1): 1-22, 1977.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-248255 Icve, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2019. URL https://arxiv.org Abss/8100 04805.
Mohamed El Banani, Karan Desai, and Justin Johnson. Learning visual representations via sngguage-guided sampling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19208-19220, 2023.

---

Published as a conference paper at ICLR 2025
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels, 2020. URL https:
0t5//2005.12222
Irving J Good. Maximum entropy for hypothesis formulation, especially for multidimensional con- tingency tables. The Annals of Mathematical Statistics, pp. 911-934, 1963.
Roger Grosse, Ruslan R Salakhutdinov, William T Freeman, and Joshua B Tenenbaum. Exploiting compositionality explore a large space of model structures. arXiv preprint 1rXiv:1210.4856, 2012.
Geoffrey E Hinton and Sam Roweis. Stochastic neighbor embedding. Advances in neural informa- fion processing systems, 15, 2002.
Tianyang Hu, Zhili Liu, Fengwei Zhou, Wenjia Wang, and Weiran Huang. Your contrastive learn- ing is secretly doing stochastic neighbor embedding. In International Conference on Learning Representations, 2023.
Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9865-9874, 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL http:ps//axxxv..rrgg.o
2. 6980.
Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Johannes F Kruiger, Paulo E Rauber, Rafael Messias Martins, Andreas Kerren, Stephen Kobourov, and Alexandru C Telea. Graph layouts by t-sne. In Computer graphics forum, volume 36, pp- 283-294. Wiley Online Library, 2017.
Joseph B Kruskal. Multidimensional scaling by optimizing goodness of fit to nonmetric hypothe- sis. Psychometrika, 29(1):1-27, 1964.
Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. leee Access, 8:199907-193334, 2020.
Wenting Li, Jiahong Xue, Xi Zhang, Huacan Chen, Zeyu Chen, Feijuan Huang, and Yuanzhe Cai. grapdgrepp An efficient word embedding approach on word co-occurrence graph using random walk technique, 2023. URL https: /ttpiii///rvvvi////aae/aseaammvr
orgg /2001.211200 04312.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive cluster- ing. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 8547-8555, 2021.
J Macqueen. Some methods for classification and analysis of multivariate observations. In Pro- ceedings of 5-th Berkeley Symposium on Mathematical Statisties and UrovebilttyUniiversityy of California Press, 1967.
Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprini arXiv: 1802.03426, 2018.
Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems, 14, 2001.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. Xrviv preprint 1 2018
Karl Pearson. Lii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(1 2(11):559-57, 1901.
12

---

Published as a conference paper at ICLR 2025
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds mutual information. In International Conference on Machine Learning, pp. 5171- 5180. PMLR. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
attps://arxiv org/ absaa /ttp////....../////////333 00020.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face the IEEE conference on computer vision and pattern
recognition and clustering. In Pruceedings the recognition, pp. 815 823, 2015.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions
pattern analysis and machine intelligence, 22(8))888-905, 2000.
Vlad Sobal, Mark Ibrahim, Randall Balestriero, Vivien Cabannes, Diane Bouchacourt, Pictro As- tolfi, Kyunghyun Cho, and Yann LeCun. X-sample contrastive loss: Improving contrastive learn- ing with sample similarity graphs. International Conference on Learning Representations, 2025.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16. pp. 776-794. Springer, 2020.
Michael Tschannen, Josip Djolonga, Paul Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv: 13625, 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Lilian Weng.
Contrastive representation learning, lilianweng May 2021. URL https: //lilianweng . gi thub. www.tth.....oooooooooooo /0/posts/2021-05- 10-.001------11-ooorrttttii.ie
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Uni- fied contrastive learning in image-text-label space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19163-19173, 2022.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A Additional Experiments on Debiasing Feature Learning
15
Proofs for Unifying Dimensionality Reduction Methods
17
C Proofs for Unifying Feature Learning Methods
19
D Proofs for Unifying Clustering Methods
24
I-Con as Variational Method
27
Why do we need to unify representation learners?
28
G How to choose neighborhood distributions for your problem
29
H Comparing I-Con, MLE, and the KL Divergence
30
I On I-Con's Hyperparameters
30
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL EXPERIMENTS ON DEBIASING FEATURE LEARNING
The following experiments aim to test the effect of our debiasing approach feature learning. We followed the experimental setup introduced by Hu al. (Hu et al., 2023). The architecture consisted of a ResNet-34 backbone paired with a two-layer multilayer perceptron (MLP) feature extractor. The MLP included a hidden layer with 512 units and an output layer with 64 units, without batch normalization.
CIFAR-10 & CIFAR-100. The models were trained on the CIFAR-10 dataset for 1000 epochs using the Adam W optimizer with the following hyperparameters: A1 = 0.9, _9 0.999, learning rate of 10-3, a batch size of 1024, and weight decay of 10-5, The learned kernel was either Gaussian or Student's ss t-distribution with degrees of freedom d,
For evaluation, we used two methods: (1) linear probing on the 512-dimensional embeddings from the MLP's hidden layer, and (2) k-nearest neighbors (k 3) classification based on the same embeddings for CIFAR-10 (in-distribution) and CIFAR- 100 -uu-oo-ddistribuiion..
STL-10 & Oxford-IIIT Pet. With similar setup, the models were trained contrastively on STL-10 (in distribution) without labels using the same hyperparameters as in the CIFAR experiments. For evaluation, performed (1) linear probing for the STL-10 classification task and Oxford-IIIT Pet binary classification, and (2) k-nearest neighbors classification based on the same embeddings for STL-10 and Oxford-IIIT Pet with k 10.
Method
CIFAR10 (in distribution)
CIFAR100 (out of distribution)
Linear Probing
KNN
Linear Probing
KNN
qa is Gaussian Distribution
SimCLR (Chen al., 2020a)
77.79
80.02
31.82
40.27
DCL (Chuang et al., 2020)
78.32
83.11
32.44
42.10
Our Debiasing a = 0.2
79.50
84.07
32.53
43.19
Our Debiasing 0.4
79.07
85,06
32.53
43.29
Our Debiasing 0.6
79.32
85.90
30.67
29.79
96 is Student's t-distribution
L-SimCLR(Hu et al., 2023)
90.97
88.14
38.96
30.75
DCL (Chuang et al., 2020)
Diverges
Diverges
Diverges
Diverges
Our Debiasing
0.2
91.31
88.34
41.62
32.88
Our Debiasing a = 0.4
92.70
88.50
41.98
34.26
Our Debiasing 0.6
92.86
88.92
38.92
32.51
Table 5: Contrastive feature learning evaluation results for CIFAR10 and CIFAR 100 datasets with various debasing a factors. Adding some amount of debasing helps raising accuracy in both linear probing and KNN classification.
Method
STL-10 (in distribution)
Oxford-IIIT Pet (out of distribution) Logistic Regression
Linear Probing
KNN
KNN
SimCLR (Chen et al., 2020a)
77.71 74.92
74.80
71.48
DCL (Chuang et al., 2020)
78.32
75.03
74.41
70.22
(s is a Student's t-distribution
t-SimCLR(Hu et al., 2023)
85.11
83.05
83,40
81.41
Our Debiasing a 0.2
85.94
83.15
84.11
81.15
Our Debiasing 0.4
86.13 13
84.14
84.07
84.13
Our Debiasing a 0.6
87.18
83.58
84.51
83.04
Contrastive feature learning evaluation results for STL10 (in distribution) and Oxford- IIIT Pet (out of distribution) with various debasing a factors. Similar to the other experiments, our debasing helps raising accuracy in both linear probing and KNN classification.
Table 6:
15

---

Published as a conference paper at ICLR 2025
(a) STL-10 embeddings for SimCLR & DCL
(b) CIFAR-10 10 embeddings for SimCLR DCL
2-0
(c) CIFARIO embeddings for models trained on with Gaussian distribution
2 0.6
(d) CIFAR1O features for models trained with Student's t-distribution Te
0
0.2
0.4
0.6
(e) STL-10 features for models trained with Student's 's l-distribution
Figure 7: t-SNE visualizations of learned embeddings on CIFARIO and STL10 datasets. (a) and (b) display embeddings from the DCL (Chuang et al., 2020) method before and after applying de- biasing, showing tendency to heavily cluster data points, which may hinder diitt--distri generalization (Hu et al., 2023). (c) and (d) show embeddings with Student's t-distribution, where debiasing factor a enhances clustering and separation, resulting in improved data representation.
16

---

Published as a conference paper at ICLR 2025
B PROOFS FOR UNIFYING DIMENSIONALITY REDUCTION METHODS
We begin by defining the setup for dimensionality reduction methods in the context of I-Con. Let J; E I represent high-dimensional data points, and Ø; E RT represent their corresponding low- dimensional embeddings, where < d. The goal of dimensionality reduction methods, such as Stochastic Neighbor Embedding (SNE) and t-Distributed Stochastic Neighbor Embedding (t-SNE), to learn these embeddings such that neighborhood structures in the high-dimensional space preserved in the low-dimensional space. In this context, the low-dimensional embeddings Φ can be interpreted as the outputs of mapping function fx(z)), where fe is essentially an embedding matrix or look-up table. The I-Con framework is well-suited to express this relationship through KL divergence loss between two neighborhood distributions: one in the high-dimensional space and one in the cow-dimensional space.
Theorem 1. Stochastic Neighbor Embedding (SNE) (Ilinton & Roweis, 2002) is an instance of the I-Con framework.
Proof. This is one of the most straightforward proof's in this paper, essentially based on the definition of SNE. The target distribution (supervised part), described by the neighborhood distribution in the high-dimensional space, is given by:
exp [-||, Iz 22202
po(jlt)
Σ Aett exp
(- ||:; TAA 112 22
while the learned low-dimensional neighborhood distribution is:
tb 112) Eneer exp (- 16 - Φ& 2)
exp (- 10:
go(ilt)
The objective is to minimize the KL divergence between these distributions:
Po (j(j
-= Σ Dea = E Po (j log (jl))
The embeddings 0; are learned implicitly by minimizing L. The mapper is an embedding matrix, as SNE is non-parametric optimization. Therefore, SNE is special case of the I-Con framework, where po(ilV k) and (ili) represent the neighborhood probabilities the high- and low-dimensional spaces, respectively.
Corollary (t-SNE (Van der Maaten Hinton, 2008).. 1-SNE is an instance of the I-Con frame- work.
Proof. The proof is similar the for SNE. While the high-dimensional target distribution po(jlt) remains unchanged, t-SNE modifies the low-dimensional distribution to a Student's t- distribution with one degree of freedom (Cauchy distribution):
(1 |0: o
(ili) = Eny (1 11 2n 1
The objective remains to minimize the KL divergence. Therefore, t-SNE an instance of the I-Con framework.
Proposition Let (x.)) then the following cohesion variance loss
Σ wey (()l)f((xz)) - f2( -
Leohesion-var
an instance of I Con in the special case wij = p(jli) and qu is Gaussian as with a large width cc.
17

---

Published as a conference paper at ICLR 2025
Proof. By using AM-GM inequality, we have
1
(IT k= =M
=4)=
>
(e
2k-1 z6)
k=
which implies that
n
(e (i)2
log Ahe log n 2 log
log
e
2k
2k + (n)
km
Alternatively, this can be written as
log 3h
24 -lo((1)
k=1
Now assume that we have Gaussian Kernel 4s
exp
- (f
(o(i))) = Exe; exp (f1)f((z))) (xx
Therefore, given the inequality of exp-sum that we showed above, we have
log go(i)) [] fo(r))]1
(I) [o(
log Σ exp
Right
S [fo(as)) - fo(xx)11 + na²
(1a(()))
log(n))
(=1ff((x;) (0()))11 n
(1)((())) fo(xx)]13) log(n)
Therefore, the cross entropy (pu, 4o), is bounded by
H (poo (6) - ΣΣ log
Σ p(jli)
1 (-110((s))
1
E1lo((a) - - log(n)
n
n
1
Σ
- Σ (p(li)iffaa.)) folow))?)
-log(n)
0²
7
11
zzk
1
Σ
- -2Vr(XX)
+ log(n)
Φ2
1
Σ
P(11)LF((0))) f -(ee)l(2vVa((x)
+ log(n)
Ccohesion-var + log(n)
On the other hand, the L.H.S. can be upper bounded by using second order bound 1-=11-2++ which implies that
log
n
2h 2
log(1
Σ
n
2k
+
1
2
log(n)
n
k=
k=1
18

---

Published as a conference paper at ICLR 2025
On the other hand, log(1 + u) n2/2, therefore, the
1
1
log
Σ
2
Σ k=1
22 - logg(n)
2
k=1
Therefore, in the limit + oo, the bounds become tighter and the I-Con loss approaches the cohesion variance loss.
Theorem 2. Principal Component Analysis (PCA) is an asymptotic instance of the I-Con.
Proof. By using Proposition
1[i = jl. we have the following expression for When Pyj:
C
Pati Palilff((.)) - - fo(x))11 Σ
(V)r(X)
n ij
Σ
IIfa(wx) - -
(V
Therefore, minimizing is equivalent to maximizing the variance which is the equivalent of the PCA objective. Intuitivily, the KL divergence is asking -11/a(x,) fa(
be the maximum in comparison - (z,) f2 to match the supervisory indicator function, which implies the minimization of the sum of - (1;) - (2,, ||? which is maximizing the variance. If we restrict fø to be linear projection map, then minimizing would be equivalent to PCA.
C PROOFS FOR UNIFYING FEATURE LEARNING METHODS
We now extend the I-Con framework to feature learning methods commonly used in contrastive learning. Let I, Rd be the input data points, and fø( R be their learned feature embed- ding. In contrastive learning, the goal is to learn these embeddings such that similar data points (positive pairs) are close in the embedding space, while dissimilar points (negative pairs) far apart. This setup can be expressed using a neighborhood distribution in the original space, where "neighbors" are defined not by proximity in Euclidean space, but by predefined relationships such as data augmentations or class membership. The learned embeddings fa,))) define a new distribu- tion over neighbors, typically using Gaussian kernel the learned feature space. We show that InfoNCE is a natural instance of the I-Con framework, and many other methods, such as SupCon, CMC, and Cross Entropy, follow from this.
Theorem (InfoNCE (Bachman et al., 2019)). InfoNCE is an instance of the I-Con framework.
Proof. InfoNCE aims maximize the similarity between positive pairs while minimizing for negative pairs in the learned feature space. In the I-Con framework, this can be interpreted as minimizing the divergence between two distributions: the neighborhood distribution in the original space and the learned distribution in the embedding space.
The neighborhood distribution pe (ili) uniform over the positive pairs, defined as:
{ 1 if vy is among the k positive views of di; otherwise.
(o())))
where l: is the number of positive pairs for z;-
The learned distribution qu (jli) based on the similarities between the embeddings fo(x;) and fay)), constrained to unit norm (x:))l 1). Using temperature-sealed Gaussian kernel, this distribution is given by:
exp (folzz) fo(I,)/T) Eneo exp (fo (xx)) So(
go(ili)
19

---

Published as a conference paper at ICLR 2025
where is the temperature parameter controlling the sharpness of the distribution. Since [[ff(z))]] 1, the Euclidean distance between fø(zz) and fo(z,) is 2(f
(2,) ) fo(z))).
The InfoNCE loss can be written in its standard form:
cxp (fa(z)) fz(x))/T)
LintoNCE. = - Σ log Ex exp (T) fo(x))/T)
where j* the index of a positive pair for i. Alternatively, in terms of cross-entropy, the loss becomes:
LInfONCE ox ΣΣ (o(i)) log (o(i)) = H(po,
where H (pe, 9a) denotes the cross-entropy between the two distributions. Since po(jii) is fixed, minimizing the cross-entropy (p) is equivalent minimizing the KL divergence (pelpelg))) By aligning the learned distribution (o()))) with the target distribution (e()),) In- foNCE operates within the I-Con framework, where the neighborhood structure in the original space is preserved in the embedding space. Thus, InfoNCE is a direct instance of I-Con, optimizing the same divergence-based objective.
Corollary 2. 1-SimCLR and 1-SimCNE (Hu et al., 2023; Böhm et al., 2023) are instances of the I-Con framework.
Given the proof of Theorem 3, we can see that t-SimCLR is equivelant by having the same po but to would change from a Gaussian distribution over cosine similarity to Student-T distribution over Euclidean distance.
- (66))))))
(ili) = Enes (1)fo(())) Fa(
Theorem 4. VICReg Bardes et al. (2021) without covariance term is an instance of the I-Con framework.
Given Proposition 1, we know that any loss in the cohesion variance form is instance of I-Con:
C Σ - f0(z)) (X)
we choose Pyje to be an indicator over positive pairs, i and 1+, we obtain
C Σ lfo(z)) fa(z, )lI var((X)
which is the VICReg loss without the covariance term and with an invariance-to -o-aiiinnce term ratio of 1:2. Observe that VICReg does not have negative pairs because it applies an equal repulsion force to all points. This is equivalent to taking - oo in the conditional Gaussian distribution over the embeddings.
Theorem (Triplet Loss (Schroff et al., 2015)). Triplet Loss can be viewed as an instance of the I-Con framework with the following distributions pa pa(j and (il)):
po(j))) = {
x is among the positive views of Dis otherwise,
((lla,)-f(azz)))
exp
4s (ili)
Enys exp
particularly in the special case where only two neighbors are considered: one positive view and one negative view.
20

---

Published as a conference paper at ICLR 2025
Proof. The idea of this proof was first presented at (Khosla et al., 2020) using Taylor Approximation; however, in this proof we present a stronger bounds for this result. For simplicity, we set o = 1 (the general bounds for other values provided at the end of the proof).
L ---N NI
1 ΣΣ 2olili log Ekes exp (-11fo (x()))
exp (- (ll - (3)
In the special case where each anchor 2; has exactly one positive n; and one negative 2; example, the denominator simplifies to:
Σ leght
exp (-11f((r;)) (xxx) = exp (-11f((z))) - fo(x;) (1 + exp (-11fo(z;)) fa(x;) )1(
Let de = llf( - fx and di [1f((z;) fa(a; ||I Substituting these into the loss function, we obtain:
exp (- exp (-dtt) ) + exp (-d7)
L = N
log
log
N
+xeexp (dī - -dtt)
1 = N
Σ log (1 + exp (dz - -).)..
Recognizing that the expression inside the logarithm is the softplus function, we can leverage its well-known bounds:
max(=, 0) & log (1 exp(=)) I max(z, 0) + log(2).
By letting dt -d,, we substitute into the bounds to obtain:
N Σ max(d; - di 0) - C
N Σ max(d, - , +lo((2)
where the left-hand side is the Triplet loss Lriplet the following bounds:
E, max -d d Therefore, we obtain
C- log(() < Eriiler S L. For general a, the inequality bounds are as follows:
for 0² log(2) < LTriplet S Lor,
where
exp [-Ilfales) (f(ax))) T²
C, = N
qo (il)) log
(olmi)-f((aa))
Exets kezz exp
As approaches 0, Tripler approaches Co.
Theorem 6. The Supervised Contrastive Loss (Khosla et al., 2020) is an instance of the I-Con framework.
Proof. This follows directly from Theorem 3. Define the supervisory and target distributions 4 = exp (fe( fo(z,)/T)
kys Enyr exp (fo(I.) fo(xx)/T)*
1
polj 1)
K, 1 1 and j share the same label),
where fo is the mapping to deep feature space and K; is the number of samples in the class of Substituting these definitions into the I-Con framework recovers the Supervised Contrastive Loss.
21

---

Published as a conference paper at ICLR 2025
Theorem 7. The X-Sample Contrastive Learning Loss (Sobal et al., 2025) is an instance of the I-Con framework.
Proof. Consier the following p distribution over corresponding features (e.g. caption embeddings for images):
exp(go(ar)) g0(r))) Σ exp(go(x;;) (2x)) Rept
where 9 could be either parametric or a non-parametric mapper the corresponding embeddings (x;)). On the other hand, similar to most feature learning methods, the learned distribution is Gaussian over learned embeddings with cosine distance
exp(fo(z)) fa(2,)) Lo 11) = E @exp(f((rt) fe()))
where fe is the mapping to deep feature space.
Theorem 8. Contrastive Multiview Coding (CMC) and CLIP are instances of the I-Con framework.
Proof. Since we have already established that InfoNCE is an instance of the I-Con framework, this corollary follows naturally. The key difference in Contrastive Multiview Coding (CMC) and CLIP is that they optimize alignment across different modalities. The target probability distribution po (jji) can be expressed as:
Po(jli) = =zz 1ii and j are positive pairs and V, * Vgl,
where V, and V, represent the modality sets of and æj. respectively. Here, Pe (jli) assigns uniform probability over positive pairs drawn from different modalities.
The learned distribution do (j))), in this case, is based Gaussian similarity between deep fea- tures, but conditioned on points from the opposite modality set. Thus, the learned distribution defined as:
exp (=))f((z)) (o,))11
go (il
Ekev, exp (-11f((x:) fa(xx)
This formulation shows that CMC and CLIP follow the same principles as InfoNCE but apply them in multiview setting, fitting seamlessly within the I-Con framework by minimizing the divergence between the target and learned distributions across different modalities.
Theorem 9, Cross-Entropy classification is an instance of the I-Con framework.
Proof. Cross-Entrope can be viewed as a special case of the CMC loss, where one "view" corre- sponds the data point features and the other the class logits. The affinity between a data point and class is based on whether the point belongs to that class. This interpretation has been explored in prior work, where Cross Entos-y was shown to be related to the CLIP loss (Yang et al., 2022).
Theorem 10. Harmonic Loss for classification is an instance of the I- I-Con framework.
Proof. This is the equivalent of moving from Gaussian distribution for g(ilt) in Cross-Entrer Entsspy to a Student-T distribution analogs to moving from SNE t-SNE. More specifically, let V the set of data points, C the set of class prototypes, &e be the learned class prototype for class i, and be the harmonic loss degree.
Consider the following p. which is data-label indicator
p(ll belongs to class j]
22

---

Published as a conference paper at ICLR 2025
and the following 9. which is Student-T distribution with 2n 1 degrees for freedom.
lim (1 + [[f)(z))) 0,1l9/((2n 1)02))
Ekec (1+1flof)) (x113/((2n 1) T
be rewritten as
(((2n + |15o (x.) - -
kEC Ckec(((2n 1)02) + (x;) 0: 1x112/)-
As - oc, the loss function approaches
( - I
f- iEC Exec(lifo(:)) 0 112 n
which's i the Harmonic Loss for classification as introduced by 10
Theorem 11. Masked Language Modeling (MLM) (Devlin et al., 2019) loss is an instance of the I-Con framework.
Proof. In Masked Language Modeling, the objective is to predict masked token given its sur- rounding context zę. This setup fits naturally within the I-Con framework by defining appropriate target and learned distributions.
The target distribution po(jlt) is the empirical distribution over contexts and tokens j. defined as:
po(j)i) Z # [Context i precedes token j]
where # [Context precedes token jl counts the number of times token follows context a, in the training corpus and Z is a normalization constant ensuring that Σ, Pa (a( 1
The learned distribution (o(ili) is modeled using the neural network's output logits for token pre- dictions. defined as a softmax over the dot product between the context embedding fo(r;) and the token embeddings Øy:
exp (Jo(zz) 05)
(ilt)
Ekey exp (1:)):).
where fx;) is the embedding of the context H; produced by the model, Øy is the embedding of token j. and V is the vocabulary of all possible tokens.
The MLM loss aims minimize the cross-entropy between the target distribution pe(jli) and the learned distribution (ili))
LMLM ΣΣ po log = H(po, go).
Since in practice, for each context æa only the true masked token ji is considered, the target distri- bution simplifies to:
Pe (ib) 83.12 where 855; is the Kronecker delta function, equal to if j = j and otherwise.
Substituting this into the loss function, the MLM loss becomes:
LMLM
Σ log qo (x
23

---

Published as a conference paper at ICLR 2025
D PROOFS FOR UNIFYING CLUSTERING METHODS
The connections between clustering and the I-Con framework are more intricate compared to the dimensionality reduction methods discussed earlier. To establish these links, first introduce a probabilistic formulation of K-means and demonstrate its equivalence to the classical K-means algorithm, showing that it is zero-gap relaxation. Building upon this, reveal how probabilistic K-means can be viewed as an instance of I-Con, leading to a novel clustering kernel. Finally, we show that several clustering methods implicitly approximate and optimize for this kernel.
Definition 1 (Classical K-means), Let 3 ...,, EN JA,, p2...... 1 Pm E I the cluster centers.
E R denote the data points, and
The objective of classical K-means is to minimize the following loss function:
N >>
Menss
Ie = =c]lar; - -
t=
where c represents the cluster assignment for data point 2;, and is defined as:
( = arg min || - Mell?.
PROBABILISTIC K-MEANS RELAXATION
probabilistic K-means, the cluster assignments are relaxed by assuming that each data point a;; belongs to cluster with probability dic. In other words, di represents the cluster assignments vector for 24
Proposition 2. The relaxed loss function for probabilistic K-means is given by:
N ΣΣ
L Means
Siella; - Mell?
1
and is equivalent to the original K-means objective LLMeans The optimal assignment probabilities die are deterministic, assigning probability I to the closest cluster and O to others.
Proof. For each data point F3, the term Ees @sell=; - Mell? is minimized when the assignment probabilities oie deterministic, i.e.
(1 if arg || M3ll?
die
otherwise.
With these deterministic probabilities, Cpootk-Means simplifies to the classical K-means objective, confirming that the relaxation introduces no gap.
CONTRASTIVE FORMULATION PROBABILISTIC K-MEANS
Definition 2. Let [ be set of data points. Define the conditional probablity qs (s(i)))
g = Σ
dieRie km Qko
where di is the soft-cluster assignments for Z4.
Given (il)), we can reformulate probabilistic K-means as a contrastive loss: Theorem 12. Let {x E R" and [ be the corresponding assignment probabilities.
Define the objective function as:
Σ (z, qo)(l1))..)))
Minimizing C with respect to the assignment probabilities {dict yields optimal cluster assignments equivalent to those obtained by K-means.
24

---

Published as a conference paper at ICLR 2025
Proof. The relaxed probabilistic K-means objective Means is:
N
Probb Meaus
Diella; - Mell
Expanding this, we obtain:
m7
m
Prrob-k -rems Meins
Pic
M 11
Picti
He
The cluster centers Me that minimize this loss are given by:
Σ -i=1 PioT;
Me
Ei=l Die
Substituting He back into loss function, we get:
C= Σ (x. x,) go(il)),
which proves that minimizing this contrastive formulation leads to the same clustering assignments as classical K-means.
Corollary 3. The alternative loss function:
||,; - 2, g Σ
6
yields the same optimal clustering assignments when minimized with respect to fdich
Proof. Expanding the squared norm in the loss function gives:
C
Σ (|=|11 2x; + ||>,|| go (ili).
1
The terms involving ? and
| II? simplify since E3 (o(o) reducing the loss to:
c 2=
Σ xs 2 (o(1)))
which is equivalent to the objective in the previous theorem.
PROBABILISTIC K-MEANS AS AN I-CON METHOD
In the I-Con framework, the target and learned distributions represent affinities between data points based specific measures. For instance, in SNE, these measures are Euclidean distances in high- and low-dimensional spaces, while in SupCon, the distances reflect whether data points belong to the same class. Similarly, of clustering, where two points are considered neighbors they belong to the same cluster. The probability of selecting Ty xz's 's neighbor is the probability that point, chosen uniformly random from x 's cluster, is Zy. More explicitly, let (i())) represent the probability that x, is selected uniformly random from 2, cluster:
define measure of neighborhood probabilities in the context
g = Σ
Diedie
Ek Que
Theorem 13 (K-means as an instance of I-Con). Given data points (N.) define the neighbor- hood probabilities po (ibb) and (ili) as:
Po (jli)
exp
[z xz,
11,11/2200
(o(il) = Σ
Diedio
Σx exp
(-|
x2
km Oko
25

---

Published as a conference paper at ICLR 2025
Let the loss function Le-SNE be the sum of KL divergences between the distributions q6 (i(t) and Po(jli):
Le-SNE Σ
Then,
CN-SE =
20 Prob--k-Meons
1
Σ H (go(-li))
where H(ga(-10) is the entropy of qo(-10).
Proof. For simplicity, assume that 2o = 1. Denote Σx exp (-112; xxll?) by Z, Then we have:
log po (ilb) -1|2; z log Zi.
Let Li be defined as Ej ||ara 2yy1 (o(ii)). Using the equation above, C; be rewritten
Li
Σ | 2, (0(1)))))))
(llo((po()1)) + log(Z)))
(3)
Σ (o(l))) log(po - Σ g0 (j(()) log Z.)
(4)
Σ (((po - (zg(
(5)
H (-)()))), po log(Z,) = (1s (lpo + (-16(-1)) + log
(6)
(7)
Therefore, Crrot-MMeass defined in Corollary 3, can be rewritten as:
LPtob-MMeuns
ΣΣ a - 2111 ao(i)))) EΦ.
(8)
Σ DKL
+ H (10(-10) + log(Zi)
(9)
SNE
Σ H(g constant.
(10)
Therefore,
LO-SNE CProb-KMeans ((go-lii).
If we allow a to take any value, the entropy penalty will be weighted accordingly:
Le.SNE
20 PPrsh-KMeans
1
Σ H(as(-1)).
Note that the relation above up to an additive constant, This implies that minimizing the con- trastive probabilistic K-means loss with entropy regularization minimizes the sum of KL divergences between (-1)) and pe(-lt).
Corollary 4. Spectral Clustering an instance of the I-Con framework.
Proof. From Theorem 13, we know that K-Means clustering can be formulated as instance of the I-Con framework, where the clustering assignments depend on the inner products of the data points.
Spectral Clustering extends this idea by first embedding the data into lower-dimensional space using the top eigenvectors of the normalized Laplacian derived from the affinity matrix A. The affinity matrix A is constructed using similarity measure (e.g., RBF kernel) and encodes the probabilities of assignments between data points. Given this transformation, spectral clustering is instance of I-Con on the projected embeddings.
26

---

Published as a conference paper at ICLR 2025
Theorem 14. Normalized Cuts (Shi & Malik, 2000) is an instance of I-Con.
Proof. The proof for this follows naturally from our work on K-Means analysis. The loss function for normalized cuts defined as:
m cut(Ac: Ae)
NormCuts
vol(A.)
where Ag is subset of the data corresponding cluster c, Ac is its complement, and cut(AAe, Ac) represents the sum of edge weights between A. and Ac, while vol(A.) is the total volume of cluster Ac, i.e., the sum of edge weights within Ac.
Similar to K- K-Means, by reformulating this in contrastive style with soft-assignments, the learned distribution can be expressed using the probabilistic cluster assignments Øie = p(c/2,) as:
m
Dicojedy
9
Σ
Ek=1 Skodk
where dj is the degree of node Tj. and the volume and cut terms can be viewed weighted sums over the soft-assignments of data points to clusters.
This reformulation shows that normalized cuts can be written manner consistent with the I- Con framework, where the target distribution Pe(jlt) and the learned distribution (i(i))) represent affinity relationships based on graph structure and cluster assignments.
Thus, normalized cuts is an instance of I-Con, where the loss function optimizes the neighborhood structure based on the cut and volume of clusters in manner similar to K-Means and its probabilistic relaxations.
Theorem 15. Mutual Information Clustering is instance of I- I-Con.
Proof. Given the connection established between SimCLR, K-Means, and the I-Con framework, this result follows naturally. Specifically, the target distribution (o(ji) (the supervised part) is uniform distribution over observed positive pairs:
if xr is among the l positive views of T4: 1
pa (ol)
o otherwise.
On the other hand, the learned embeddings Φi represent the probabilistic assignments of x; into clusters. Therefore, similar the analysis of the K-Means connection, the learned distribution is modeled as:
go (1b)
Σ
DicRie
Σk=1 Oke
This shows that Mutual Information Clustering can be viewed as method within the I-Con frame- work, where the learned distribution (s(jl)) aligns with the target distribution (jli), completing the proof.
E I-CON AS A VARIATIONAL METHOD
Variational bounds for mutual information are widely explored and have been connected to loss functions such InfoNCE, where minimizing InfoNCE maximizes the mutual information lower bound (Oord et al., 2018; Poole et al., 2019). The proof usually starts by rewriting the mutual information:
I(XY)) = E Ep(ww) log q( v)
+ Ep(s) [DKL (p(zly) Il ((<l))))
p(x)
This expression typically used to derive a lower bound for I(X;Y) The proof usually begins by assuming that p is uniform discrete data points X {x; (cx)) i=1 (i.e., use uniform sampling
27

---

Published as a conference paper at ICLR 2025
for data points). By using the fact that p(x;) N we can write p(z, v) h/(aly). Therefore, the mutual information lower bound becomes
1(X;)) 2 E Ep(a.v) [logg(zlv)] Bp(a,)) [logp(x)]
E Ep(a.v) [logg(zl))] log(AV)
p(aly) log (<>lv)) + log(N)
N
a.yEAxX
1
Σ
Σ
p(aly) (ogg(x/))) log(N))
yEA ZEX
H (p(xlv), ((>lv)) + log(N)
Therefore, maximizing the cross-entropy between the distributions maximizes the mutual infor- mation between samples.
On the hand, Variational Bayesian (VB) methods are fundamental in approximating intractable pos- terior distributions p(= æ) with tractable variational distributions (z).. This approximation is
achieved by minimizing the Kullback-Leibler (KL) divergence between the variational distribution
and the true posterior:
(=)(g(=)llp(z x)) = Rgy log (=(=)
(11)
p(z
xx)
The optimization objective, known as the Evidence Lower Bound (ELBO), is given by:
ELBO (x) [log =)] Exy(s) [log qo(=))-
(12)
Maximizing the ELBO is equivalent minimizing the KL divergence, thereby ensuring that qo closely approximates p(= a:) (Blei et al., 2017).
VB can framed within the I-Con framework by making specific mappings between the variables and distributions. Let i correspond to the data point and correspond to the latent variable 2. We can set the supervisory distribution pa(z xx) to the true posterior p(= x). This allow us to define the learned distribution (= [ x)) to be independent of x. i.e., 2o(= x)) go(2).
Under these settings, the I-Con loss simplifies to:
f DEX
((Φ)
KL (p(z (x)))(((>) dr = Ep(x) [KL(p(> ())l((()))]-
(13)
INTERPRETATION
Global Approximation: In VB, (2(2) serves as a global approximation to the posterior p(z æ) across all data points æ. Similarly, in I-Con, when go(i i) 4o (j), the learned distribution provides a uniform approximation across all i.
Variational Alignment: Both frameworks employ variational techniques to align a tractable distribution ds with intractable supervisory distribution p. This alignment ensures that the learned representations capture essential information from the target distribution.
Framework Generalization: I-Con generalizes VB by allowing go i) to depend on enabling more flexible and data-specific alignments. VB is recovered special case where the learned distribution uniform across all data points.
F WHY DO WE NEED TO UNIFY REPRESENTATION LEARNERS?
I-con not only provides a deeper understanding these methods but also opens up the possibility of creating new methods by mixing and matching components. We explicitly use this property to discover new improvements to both clustering and representation learners. In short, I-Con acts more clearly
like a periodic table of machine learning losses. With this periodic table the implicit assumptions of each method by breaking down modern ML losses into more simple components: pairwise conditional distributions and
28

---

Published as a conference paper at ICLR 2025
One particular example of how this opens new possibilities is with generalized debiasing op- eration. Through our experiments we show adding slight constant linkage between datapoints improves both stability and performance across clustering and feature learning. Unlike prior art, which only applies to specific feature learners, our debiasers can improve clusterers, feature learn- ers, spectral graph methods, and dimensionality reducers.
Finally it allows us to discover novel theoretical connections by compositionally exploring the space, and considering limiting conditions. We I-Con help derive novel theoretical equivalences between K-Means and contrastive learning, and between MDS, PCA, and SNE. Transferring ideas between methods is standard in research, but in our view becomes much simpler to do this if you know methods are equivalent. Previously, it might not be clear how exactly to translate an insight like changing Gaussian distributions to Cauchy distributions the upgrade from SNE T-SNE has any effect on clustering representation learning. In I-Con it becomes clear to see that similarly softening clustering and representation learning distributions can improve performance and debias representations.
G
How TO CHOOSE NEIGHBORHOOD DISTRIBUTIONS FOR YOUR PROBLEM
PARAMETERIZATION OF LEARNING SIGNAL
(Learn a network to transform a data points to representations). Use a para- metric method to quickly represent new datapoints without retraining. Use a parametric method if there is enough "features" in the underlying data to properly learn representa- tion. Use this option with datasets with sparse supervisory signal in order to share learning signal through network parameters.
Parametric:
(Learn representation per data point). Use nonparametric method if datapoints are abstract and don't contain natural features that are useful for mapping. Use this option better optimize the loss of each individual datapoint. Do not use this in sparse supervisory signal regimes (Like augmentation based contrastive learning), as there are not enough links resolve each individual embedding.
Nonparametric:
CHOICE OF SUPERVISORY SIGNAL
Gaussians on distances in the input space: though this is common choice and underlies methods like k-means, with enough data it is almost always better to use k-neighbor distri- butions as they better capture local topology of data. This is the same intuition that used to justify spectral clustering over k-means.
your data can naturally put into a graph instead of just considering Gaussians on the input space we suggest it. This allows the algorithm to adapt local neighborhoods to the data, as opposed considering all points neighborhoods equally shaped and sized. This better aligns with the manifold hypothesis.
K-neighbor graphs distributions:
Contrastive augmentations: When possible, add contrastive augmentations to your graph this will improve performance in cases where quantities of interest (like an image class) are guaranteed to be shared between augmentations.
General kernel smoothing techniques: Use random walks to improve the optimization quality. It connects more points together and in some cases mirrors geodesic distance on the manifold (Crane et al., 2013).
Debiasing: Use this if you think negative pairs actually have a small chance of aligning positively. For small number of classes this parameter scales like the inverse of the num- ber of classes. You can also use this to improve stability of the optimization.
CHOICE OF REPRESENTATION:
Any conditional distribution representations be used, consider what kind of structure you want to learn, tree, vector, cluster, etc. And choose the distribution to be simple and meaningful for that representation.
29

---

Published as a conference paper at ICLR 2025
Discrete:
Use discrete cluster- risceettissttrrrosseeesessesttss if interpretability and discrete structure are important
Continuous Vector: Use a vector representation generic downstream performance is concern as this is bit easier to optimize than discrete variants.
H COMPARING I-CON, MLE, AND THE KL DIVERGENCE
There many connections between KL divergence and maximum likelihood estimation. We high- light the differences between a standard MLE approach and I-Con. short, although I-Con has a maximum likelihood interpretation, its specific functional form allows it unify both unsupervised and supervised methods in a way that elucidates the key structures that are important for deriving new representation learning losses. This is in contrast to the commonly known connection between MLE and KL divergence minimization, which does not focus on pairwise connections between dat- apoints and does provide much insight for representation learners. To this note that the conventional connection between MLE and KL minimization is as follows:
MMLE arg min D (1|(0)),
where the empirical distribution, P. is defined as:
1 P(z)
N Σ 8(x x;),
where 8(x xr4) is the Dirac delta function. The classical KL minimization fits parameterized model family to an empirical distribution. In contrast the I-Con equation:
((0,4) f JEE
DRL. (po(.li) (2a(-11)
Operates conditional distributions and captures "average" KI. divergence instead of sin- gle KL. divergence. Secondly, I-Con explicitly involves computation over neighboring datapoints which does not appear in the aforementioned equation. This decomposition of methods into their ac- tions on their neighborhoods makes many methods simpler to understand, and makes modifications of these methods easier to transfer between domains. It also makes it possible to apply this theory to unsupervised problems where empirical supervisory data does not exist. Furthermore some meth- ods, like DINO, do not share the exact functional form of I-Con, and suffer from various difficulties like collapse which need to be handled with specific regularizers. This shows that I-Con is not just catchall reformulation of MIE,, but is capturing specific functional form shared by several popular learners.
I ON I-CON'S HYPERPARAMETERS
One important way that I-Con removes hyperparameters from existing works is that it does not rely on things like entropy penalties, activation normalization, activation sharpening, or EMA stabiliza- tion to avoid collapse. The loss is self- eeff-balancing in this regard as any way that it can improve the learned distribution to better match the target distribution "fair game". This allows one gen- eralize certain aspects of existing losses like InfoNCE. In I-Con info NCE looks like fixed-width Gaussian kernels mediating similarity between representation vectors. In I-Con it's trivial to gener- alize these Gaussians have adaptive and learned covariances for example. This allows the network to select its own level of certainty in representation learning. If you did this naively, you would need to ensure the loss function doesn't 't cheat by making everything less certain.
Nevertheless I-Con defines a space of methods depending on the choice of and q. The choice of these two distributions becomes the main source of hyperparameters experiments change the structure of the supervisory signal (often p). For example, in a clustering experiment changing p from "Gaussians with respect distance" to "graph adjacency" transforms
explore. In particular our
30

---

Published as a conference paper at ICLR 2025
K-Means into Spectral clustering. It's important to note that K-means has benefits over Spectral clustering in certain circumstances and vice-versa, and there's not necessarily a singular "right" choice for p in every problem. Like many things in ML, the different supervisory distributions provide different inductive biases and should be chosen thoughtfully. We find that this design space makes it easier to build better performing supervisory signals for specific important problems like unsupervised image classification on ImageNet and others.
31


Reviews:
[{"review_id": "YpeCoydRay", "reviewer": "Reviewer_xJUE", "summary": "The paper presents a novel framework called I-Con, which unifies various machine learning methods, including clustering, contrastive learning, dimensionality reduction, and supervised learning, under a single equation based on the Kullback-Leibler (KL) divergence between two distributions: the supervisory and learned representations. The authors demonstrate that many existing methods are instances of this unified framework, providing a mathematical foundation that enables the transfer of ideas across domains. The paper also presents new state-of-the-art results in unsupervised image classification on the ImageNet-1K dataset, achieving an 8% improvement over previous methods. Multiple proofs and empirical evaluations back these claims, illustrating the broad applicability of I-Con to various learning problems.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Theoretical Contributions: The I-Con framework offers a unified theoretical foundation for a wide range of machine learning methods, showing that several existing approaches can be viewed as specific instances of a general KL divergence minimization problem. This not only provides a deeper understanding of these methods but also opens up the possibility of creating new methods by mixing and matching components.\n- Comprehensive Theoretical Justification: The paper provides detailed mathematical proofs, connecting 11 different approaches to the I-Con framework. \n- Cross-Domain Applicability: By unifying disparate learning paradigms (clustering, dimensionality reduction, contrastive learning, etc.), the framework facilitates the transfer of techniques across domains, allowing innovations from one area (e.g., contrastive learning) to improve methods in another (e.g., clustering).\n- Empirical Performance: The framework's potential is validated through unsupervised image classification on ImageNet-1K, where the I-Con-based method outperforms state-of-the-art methods, demonstrating its practical significance.", "weaknesses": "- Innovation and Breadth: While the authors introduce the I-Con framework, which attempts to unify multiple existing learning methods, the actual novelty and breadth of this framework might be questionable. Many of the connections between the methods discussed have already been noted in previous works, and the core contribution seems to be more about integrating existing ideas rather than offering fundamentally new insights. Actually, most methods and loss functions can be implemented in a likelihood framework, which is equivalent to minimizing KL distance. \n- Limited Guidance According to The Theorem: Several techniques like adaptive neighborhoods, debiased contrastive learning, neighbor propagation are transfered to create a state-of-the-art unsupervised image classification system. However, these methods can be thought out without the I-Con framework as many articles have established. Actually, inspiring or taking ideas from other domains is natural during research. Thus, based on the theorem, it will be more valuable to discuss how to choose the mechanism according to the I-Con framework. \n- Limited Generalization and Insufficient Experimental Validation: While the paper provides some experimental results demonstrating improvements in unsupervised classification, these are mainly focused on clustering on ImageNet-1K. The generalizability of the I-Con framework to other applications and datasets is still underexplored, lacking broader experimental comparisons across a variety of tasks and datasets. \nI would be glad to improve my score if my concerns could be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BlaKo3x9tq", "reviewer": "Reviewer_ioHt", "summary": "The manuscript attempts to build a universal framework for loss functions corresponding to different representation learning methods. It minimizes an integrated KL divergence between two marginal distributions: the supervisory and learned representations. The proposed framework encompass 11 machine learning methods, including dimensionality reduction, contrastive learning, clustering, and supervised learning, among others.", "questions": "Please see Weakness.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "One advantage is that it can leverage losses from other domains to enhance a particular domain method within this framework. For instance, in experiments, spectral clustering, t-SNE, debiased contrastive learning, and SCAN were utilized to develop an unsupervised image classification system.", "weaknesses": "- It falls short of truly encompassing all representation learning methods, such as autoencoder-based representation learning, masked pretraining-based representation learning, and other general self-supervised representation learning methods. Therefore, the proposed framework does not entirely unify all representation learning methods. I believe that this unified framework is for clustering methods but not a generic representation learning framework.\n- The motivation/benefits for unifying representation learning methods claimed in this paper do not entirely convince me personally. Could you give more discussions on whether this unified framework provides advantages beyond facilitating cross-domain insights? For example, does it provide a new method to address all limitations of different representation learning frameworks once?\n- Typically, evaluating representation learning methods involves assessing if a model has learned good representations that generalize well across various downstream tasks. However, I did not observe such experiments.\n- The choice of data augmentation strategies is crucial in contrastive learning methods. Are data augmentation-based contrastive learning methods also unified within this framework?\n- Are there differences in the task purposes between clustering and contrastive learning tasks? My understanding is that clustering aims to discover different clusters, while contrastive learning aims to learn generic representations that can be easily transferred to different downstream tasks. Could you provide further discussion on this?\n\nAdditional minor suggestions:\n- Figure 1 is not easily read. The cluster in the right part should be labeled as \"D)\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WsDCeqLcHs", "reviewer": "Reviewer_FvTp", "summary": "This paper addresses the challenge of the variety of unsupervised representation learning methods by introducing I-Con, a unified interpretation grounded in a single mathematical framework. The authors provide a theoretical proof that I-Con unifies existing methods by minimizing the KL divergence between the supervisory and parameterized neighborhood distribution. Furthermore, the authors effectively transferred perspectives of various representation learning approaches into a state-of-the-art unsupervised classification network and showed both quantitative and qualitative improvements.", "questions": "### Questions\n- Theoretical evidences for I-Con across various method are given (appendix). However, are there any experimental results in practical environments (e.g., InfoNCE on ImageNet) that shows the I-CON framework can reproduce (or improve) the original representation learning methods?\n- In Figure 3, is there a reason for using different dataset (ImageNet-1K and subset of MNIST)? Why not use ImageNet for both the quantitative and qualitative results (or vice versa)?\n- Is I-Con (Ours) stated in Table 2 result of integrated method discussed in Section 3.3, or is it the best among them? Table 3 shows results of each transferred intuitions, and results of I-Con (Ours) stated in Table 2 appears to be the best among them. Referring to the best among them as I-Con (Ours) in Table 2 may not be a fair comparison.\n- As stated in Section 4.2, I-Con has claimed to have the benefit of not requiring additional hyper-parameter tuning compared to TEMI and SCAN. However, doesn’t I-Con require hyper-parameter tuning, such as /alpha and n-walks (Figure 4, Table 4), for notable improvements?\n- DiNO[1] also showed results of k-NN classifiers[2] in the paper. Could results in Table 3 maintain its gains when using k-NN classifiers[2] as stated in [1]?\n### Modifications\n- Table 2 in the first line in Section 4.2, do not have a link to it.\n- Like in t-SNE this addition helps… → Like in t-SNE, this addition helps…\n\n[1] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´ e J´ egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021.\n\n[2] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper discusses various methods in representation learning in terms for their proposal.\n- The proposed framework is well demonstrated throughout the paper including theoretical evidences.\n- The proposed method showed significant improvements compared to state-of-the-art unsupervised classification network.", "weaknesses": "- The motivation for the need for a unified framework of (unsupervised) representation learning is not clear. As discussed in introduction section, “difficult to understand which particular loss function to choose for a given problem or domain”, does not directly match with the proposed method, as I-CON still contains the choice of $p_{\\theta}$ for each existing method and does not provide a suitable loss function for given domain.\n- The term ‘I-Con’ is ambiguous, as it is used with two different meanings. In section 3.1, term ‘I-Con’ represents a unified representation learning algorithm that can that can serve as the foundation for various methods. However in section 3.2, it is also used to describe integrated (new) representation learning method, causing confusion in the following results. To avoid confusion, use distinct terms to clearly differentiate between I-Con as a framework and I-Con as a specific method.\n- Section 3.3 is not easy to follow, especially the sub-section ‘Adaptive neighborhoods’. Paraphrasing the explanations and providing a more detailed demonstration would enhance clarity for the readers.\n- Random seeds are not considered in the experimental results. Providing results of multiple runs (e.g., 5 or 10) with standard variations (or error bars) would give stronger evidence for the claimed significant improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ll9aQaonem", "reviewer": "Reviewer_AnKo", "summary": "This paper provides a unified framework I-Con that can recover many state-of-the-art approaches in various ML tasks as special examples. The main idea behind the I-Con framework is to minimize the KL divergence between two distributions: a \"neighborhood\" distribution that is given by data and a proposed distribution that encodes some supervisory information, which is similar to the idea behind variational approaches.", "questions": "1. [Naming] In terms of I-Con = Information Contrastive learning, I was wondering if it would be more precise to relate the name to \"divergence\" or \"cross-entropy\" since the I-Con framework is essentially the KL minimization and nature of \"contrastive learning\" is not very intuitive without specification of the proposed distribution q_{\\phi}.\n2. [Experiments] It is interesting to the improvement in performance when transferring the ideas of proposed q_{\\phi} across domains. I was wondering if the improvement is also significant beyond the domain of clustering. It would also be good if some related work that transfers the SOTA q_{\\phi} across domains could be provided.\n3. Please see \"weakness\" section. I'll be happy to adjust my score if the connection with variation Bayesian methods can be elaborated.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very well written and well organized, which conducts a comprehensive survey across different fields and insightfully extracts the similarities behind state-of-the-art methods.\n2. It is novel to transfer the proposed distribution across domains and the improvement in performance shown in Section 4 is solid.", "weaknesses": "1. The connection with variational Bayesian methods can be elaborated, where the main idea is to find a proper proposed distribution q_{\\phi}.\n2. This paper provides a comprehensive survey, but it would be better if the authors could have a paragraph to compare with other survey papers, e.g.\n\t- Tschannen, Michael, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. \"On mutual information maximization for representation learning.\" arXiv preprint arXiv:1907.13625 (2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)