Unifying Self-Supervised Clustering and Energy-Based Models
Authors: Emanuele Sansone
Robin Manhaeve
Venue: Accepted by TMLR ()

Abstract: Self-supervised learning excels at learning representations from large amounts of data. At the same time, generative models offer the complementary property of learning information about the underlying data generation process. In this study, we aim at establishing a principled connection between these two paradigms and highlight the benefits of their complementarity. In particular, we perform an analysis of self-supervised learning objectives, elucidating the underlying probabilistic graphical models and presenting a standardized methodology for their derivation from first principles. The analysis suggests a natural means of integrating self-supervised learning with likelihood-based generative models. We instantiate this concept within the realm of cluster-based self-supervised learning and energy models, introducing a lower bound proven to reliably penalize the most important failure modes and unlocking full unification. Our theoretical findings are substantiated through experiments on synthetic and real-world data, including SVHN, CIFAR10, and CIFAR100, demonstrating that our objective function allows to jointly train a backbone network in a discriminative and generative fashion, consequently outperforming existing self-supervised learning strategies in terms of clustering, generation and out-of-distribution detection performance by a wide margin. We also demonstrate that the solution can be integrated into a neuro-symbolic framework to tackle a simple yet non-trivial instantiation of the symbol grounding problem.

Published in Transactions on Machine Learning Research (08/2025)
Unifying Self-Supervised Clustering and Energy-Based Models
Emanuele Sansone
emanuellesonsnne@hulleenn
Deportment of Electrical Engineering (ESAT)
KU Leuven
Robin Manhaeve
abin.mmanhacre@@@aavvvee..bb
Department of Computer Science
KU Lewven
Reviewed on OpenReview: https: // openreview. net/ form/f id= NWOkKe662
Abstract
Self-supervised learning excels at learning representations from large amounts data. At the same time, generative models offer the complementary property of learning informa- tion about the underlying data generation process. In this study, we aim establishing a principled connection between these two paradigms and highlight the benefits of their com- plementarity. In particular, we perform analysis of self-supervised learning objectives, elucidating the underlying probabilistic graphical models and presenting a standardized mothodology for their derivation from first principles. The analysis suggests a natural means of int integrating self-supervised learning with likelihoodd-aseed generalive models. We instanti- ate this concept within the realm of cluster- based supe-vupe learning and energy models, introducing a lower bound proven reliably penalize the most important failure modes and unlocking full unification. Our theoretical findings are substantiated through experiments synthetic and wollwwrld data, including SVHN, CIFARIO, and CIFAR100, demonstrat. ing that objective function allows to jointly train a backbone network in a discriminative and generative fashion, consequently outperforming existing self-supervised lcarning strate- gies in terms of clustering, generation and out-of-distribution detection performance by a wide margin. We also demonstrate that the solution can be integrated into a neuro-symbolic framework to tackle a simple yet non-trivial instantiation of the symbol grounding problem. The code publicly available https:/ //github. wtt./ooo./mmeeeeeeeeeee weweemmmnemeeeeie git.
1 Introduction
Self-supervised learning (SSL) has achieved remarkable results in recent years thanks to its ability to learn high-quality representations from large amounts of unlabeled data (Balestriero et al., 2023). At the same time, generative models have provided valuable insights into the unknown generative processes underlying data. The synergy between these two distinct areas of machine learning holds great potential, for example by leveraging knowledge of the underlying generative process learn more robust representations, or by using learned representations synthesize new data. However, a principled theory and methodology that bridges self-supervised learning and generative modeling is missing. In this work, take significant step toward closing this gap by demonstrating, for the first time, the fcasibility of learning self-supervised clustering model in generative manner. To achieve this, we address three key challenges: formulation, integration, and unification.
The first challenge arises from the diverse range of SSL. objectives proposed recent years and the absence of a common perspective. We tackle this by providing probabilistic interpretation of these objectives through their underlying probabilistic graphical models. This perspective reveals common methodology

---

Published in Transactions on Machine Learning Research (08/2025)
for deriving existing objectives from first principles. The resulting probabilistic framework naturally paves the way to connect SST with likelihood-based generative models, thereby addressing the second challenge of integration.
The final challenge, unification, involves reconciling the variety of architectural heuristics commonly used to prevent trivial representations heuristics that often hamper generative capabilities. We overcome this by introducing a novel objective, the GEnerative DIscriminative lower bound (GEDI). We prove that GEDI is guaranteed to avoid the triad of failure modes (Sansone, 2023), including representation collapse, cluster collapse, and the problem of label inconsistency with data angmentations. These guarantees eliminate the need for common architectural heuristics and enable training neural network in both a generative (akin to energy-based models) and discriminative manner (similar to SSL clustering). The proposed unified view ultimately leads to improved performance in terms of clustering, generation, and aut-of-distribution detection compared to standard cluster-basee SSL methods.
We substantiate our theoretical findings through experiments conducted on both loy and real-world datasets. Specifically, results demonstrate that GEDI can achieve significant improvement in terms of clustering performance compared to state-of-the-ard baselines on SVHN, CIFAR-10 and CIFAR-100. Additionally, in the context of generation performance, GEDI can effectively compete with existing mergy-bassdd solutions, whereas tradit ional SST. approaches fall short. Most importantly, we highlight. that the generative nature of GEDI is a crucial aspect that enhances the model's ability to detect oustif-distributtinn data, surpassing purely discriminative baselines in this regard. Finally, we show that GEDI can be easily integrated into a neuro-symbolic framework like DeepProbLog (Manhaeve al., 2018) and leverage its clustering nature learn higher quality symbolic representations when performing symbol grounding (Harnad, 1990; Barsalou, 1999; Manhaeve et al., 2018; 2021; Sansone & Manhaeve, 2022; 2023a;b; Marconato et al., 2023).
The article is structured as follows: In $2, we provide a probabilistic interpretation of three classes of SSL approaches, namely contrastive, negative-free and cluster-based methods. In $3, we showcase the integration of SSL with likelihood-basse generative models. In $4, we take a step further and provide an instantiation of the general framework, thus providing the first principled objective to unify cluster-based and energy-basee models and that is guaranteed to avoid the triad of failure modes. In $5, we review related work on SSL and in $6 we discuss the experimental analysis. Finally, in $7, we conclude by discussing future rescarch directions for SSL.
2 A Probabilistic Formulation of SSL Objectives
proj
proj enc
Figure 1: Different function blocks and variables used throughout the analysis of existing SSL approaches.
Let us introduce the random quantitics used throughout this section to analyze self-supervised learning approaches. We use x E Ω, where Ω is a compact subset of Rdd , to identify observed data drawn independently from an unknown distribution p(x) and x' E Ω to identify observed data drawn independently from a stochastic augmentation strategy T T(x'(2>) We use É RR,,, and R', to identify latent representations in deep network architecture, see Fig. 1) for visualization. Importantly, the first latent representation is obtained through encoding function enc: : Ω + R²,, whereas the second representation is obtained through the composition of enc and a projector function proj I + R², namely g: Ω + RR. Finally, we introduce a categorical variable to identify the final output of the deep network architecture.
We focus our analysis on three main classes of self-supervised learning approaches: 1) contrastive, 2) negative- free and 3) cluster-based methods.' 1 For each, we construct the underlying probabilistic graphical model (cf. Figure 2). This allows us to derive their corresponding objective functions from first principles and
We focus on purely encoder-based techniques and leave the extension to predictive models (e.g. masked autoencoders) for future work.
2

---

Published in Transactions on Machine Learning Research (08/2025)
x
71
n
(a) Contrastive (CT)
(b) Negative-Fres (NF)
(c) Cluster-Basee (CB)
Figure Probabilistic graphical models for the different classes of self-supervised learning approaches. White and grey nodes represent hidden and observed vectors victors/variables respectively. Solid arrows define the generative process, whereas blue dashed arrows identify auxiliary posterior
Index E (!. is used to identify training samples and their corresponding representations.
highlight that all objectives share common structure. Further on, we will use these insights to integrate sclf-supervised learning with generative models.
2.1 Contrastive SSL
We demonstrate that contrastive self-supervised learning can be modeled as instance classification prob- lem, and reveal that the learning objective can be decomposed into two main terms: entropy and discriminative term. For the simplicity of the exposition, we focus one of the main contrastive objec- tives, InfoNCE (den Oord et al., 2018). Same conclusions can be obtained for other objectives such as CPC (O. Henaff, 2020), SimCLR (Chen et al., 2020), ProtoCPC (Lee, 2022), KSCL (Xu et al., 2022) to name a few, as demonstrated in appendices to H. In the following, we introduce the ground truth and model predictive distributions and demonstrate how contrastive learning encourages alignment between the two.
Definition 1 (Constrastive SSL ground truth joint distribution).
p( Vim)
II p(zz)o(ys j)
with yy (I....,nn)
n) and d delto function.
Definition 2 (Constrastive SSL model distribution).
p(Zim Misii )) = Il p(z;))((,)z;;)))
p θ)
1=1 c=tmlp(a:::0
with sim R² IR! - a similarily function, parameters.
lemperature purameter and Q = [x:),}} set of
The ground truth distribution defines underlying instance classifier, where each input æ; is associated to unique natural number identifier through the å function. The following Lemma provides an alternative yet equivalent interpretation of contrastive learning (the proof is given in Appendix B).
Lemma 1. Given Desinitions and 2, marimizing the InfoNCE objective is equivalent to manimize the following log-likelihood lower bound:
Ep (lopee/www.oolppppppp (logp(Tim m1m)} -Hp(IIm))
+ E
log
p(2,)
ceimlg( ((ax
Neg. extropy term
discrimination SSL term Ler
3

---

Published in Transactions on Machine Learning Research (08/2025)
Moreover, the marimization of this lower bound is equivalent to solue instance classification problem.
From the above Lemma, we observe that the contrastive learning objective involves a discriminative and negative entropy term. Only the discriminative SSL term usually optimized, due the fact that the entropy terms does not depend θ.
2.2 Degative-Free SSL
We demonstrate that negative-free SSL enforces two important properties, namely the decorrelation of features in and the invariance w.r.t. data augmentation over Ç. Furthermore, reveal that negative-free SST. can be decomposed into two main parts, similarly to contrastive SSL. We focus the analysis on recent negative-free criterion, namely CorInfoMax (Ozsoy et al., 2022). Similar conclusions can be derived for other negative-free approaches, including Barlow Twins (Zbontar et al., 2021), VicReg (Bardes et al., 2022a;b) and W-MSE (Ermolov et al., 2021) (please refer to Appendix I for further details). In the following, we introduce the ground truth and model distributions and demonstrate how negative-free SSL encourages alignment of their covariance matrices. In effect, this drives the sample covariance matrix toward diagonal form, thereby decorrelating the features.
Definition 3 (Negative-free SSL ground-truth distribution).
(12, 2mm p(=) II
pla. plac 1=1
1), and assume the following conditional inde- Definition 4 (Negativo-free SSL auxiliary model distributions).
with Gaussion priors p(z) = No=00, 1) p(E,) -0 pendencies Mar/ T and p(æi|z)= p(x:).
(( () W[ 0) enc(2 q(2/01nn: 9) (2)
- g)T + 3I, 3 > O chosen to ensure the positive definiteness of and j = 1/n Σx= (0 0) and = (0).
with Σ = E R
We can state the following Lemma with proof provided in Appendix C.
Lemma Given Definitions and 4, mazimizing the CorInfoMor objective (Ozsoy al., 2022) is equiv- alent to marimize the Sollowing log-lik likelihood lower bound:
Eplzimeh (nd) (log((rr:n> I'.n) 2
Hp(21:n)
Eptris (KL() {)llp(()))
Neg. entropy term
discrimination SSL term ExF(
comst
Continuattoo of (xr(9)
where the first Ballback-Leibler term (1 0)]lp(z) = (1) the second term xi:: K LLg(E,)a x/; ())(())) = KT[(N((een(((;)))) enc(213 1) and const being a constant for the optimization over θ.
In other words, The discriminative terms promole two properties. Indeed, the first. term standardizes the sample covariance of the latent representation, thus decorrelating its feature elements, and the the second term promote the invariance to data augmentations. These two properties are common among other negative- free methods, as shown Appendix Notably, the negativo-entropy term in the log likelihoood lower bound is not influenced by the optimization.
4

---

Published in Transactions on Machine Learning Research (08/2025)
2.3 Cluster- Bluter--Basd SSL
We demonstrate that cluster-based SSL admits probabilistic interprelation based on the graphical model in Fig. 2(c), thus revealing that the objective can be decomposed in negative entropy and discriminative term. The analysis focussos on a recent approach SwAV (Caron et al., 2020), but is more generally applicable to other cluster-based approaches, inchuding DeepChuster (Caron et al.; 2018) and SeLA (Asano et al., 2020) name a few. In the following, we introduce the ground truth and model predictive distributions and demonstrate how contrastive learning encourages alignment between the two. Definition 5 (Cluster-Based SSL ground-truth joint distribution).
p(aim, Fian Flm) = II pla:)
0 Gats
with ye E (I. being a categorical variahle identify one of e clusters, p(ui(=;;; 9)
E, S. where
U haxe is the matris? of cluster centers, G 0) ( (a E Raxx is represendation matric and @ [, is the set of parameters.
Definition 6 Celnster-Based SSL auxiliary clustering distribution).
15 II
g x4)
We state the following Lemma (proof is provided in Appendix D)
Lemma 3. Given Defovitions 5 and mazimizing the SwAV objective (Caron et al, 2020) is equivalent to mazimize the following log- egglikelihood lower bound:
Eplainat plo {nd 2
Hp((11))
+oostt
Neg. entropy term
EpteTTecTeN (Ettelle log θ) (u(())
discriminative SSL term Lon(O)
An important aspect of cluster-based SSL is that the categorical variables yia are unobserved. A naive maximization of LCB(O) can lead to obtain trivial solutions like the one corresponding to uniformative prodictions, namely ((bx(={)= = (v))))) = Uniform((1,.... (f,,.,,,, for all i= il..... n. Fortunately, the problem can be avoided and solved exactly using an alternating optimization strategy based on the Sinkhorn-Kuope algorithm, where the alternation occurs between the optimization over the auxiliary distribution and θ, This is indeed the procedure used in several cluster-based SSL approaches, like SwAV (Caron et al., 2020), DeepCluster (Caron et al., 2018) and SeLA (Asano et al., 2020) name few. We will see later that we can obtain different. lower bound for cluster-based SSL which doesn't 't require asymmetries, in the form of alternating optimization, stopping gradients or momentum encoders.
3 Integrating SSL and Likelihood-Basee Generative Models
In all three classes of SSL approaches (ef. Lemmas 3, 6 and 9), the expected data log-likelihook can be lower bounded by the sum of two contribution terms, namely a negative entropy (xp(1 and a discriminative log-likelihodd term, chosen from (CT(O), LCB(O) and LNF(9). A connection generative models emerges by additionally lower bounding the negative entropy term, namely:
(x Eppon..) (lomn.)) (logp(x1..)
We subscripts select and columns. For instance, Usg identify y-th column of matrix
5

---

Published in Transactions on Machine Learning Research (08/2025)
Bp(as) (logp(x;))
flog (x:)) + ([(((s))loo(x)))
B B(::)) pv (x
(1)
CC)(p.r)
where (r) is generative model parameterized by V, Notably, the relation in (1) can be substituted in any of the objectives previously derived for the different SSL classes, thus allowing to integrate generative and SSL models into a single Bayesian framowork. It important to mention that much can be gained by synergistically optimizing the resulting hybrid objectives. For instance, the recent work EBCLR (Kim & Ye, 2022) proposes a specific instantiation of this general idea, by combining energy-based models with contrastive SSL approaches and showcasing the improved discriminative performance. EBCLR can be obtained by observing that the conditional density in Definition can be decomposed into joint and marginal densities (similarly to what is done in (Grathwohl et al., 2020) for supervised classifier):
{
p(y. 0) gdimlo(r).ata))/> ((@)
@=im(alo))atee)///
p(ylx;+) Ek=1 esimlg())..(zaa)/
0(x;0)= -1= dimlgle)ho(og))
(2)
I(0)
I())
defines the energy score of the marginal densily. By choosing (x)x) = p(:; θ) and 2'm -1==== 2²112 Eq. (2), one recovers the exact formulation of EBCLR (Kim & Ye, 2022). Moreover, this integrated framework is not bound to the specific family of energy-based models, rather one could in principle specify any likelihood-basel model (e.g. VAF, diffusion model, normalizing flow) in Eq. (1), consequently giving rise to wide spectrum of possible hybrid solutions. Indeed, some recent works have started to propose solution aiming lowards the joint integration of VA and contrastive SSL (Gatopoulos & Tomezak, 2020; Zhue et al., 2020; Wu et al., 2023; Nakamura et al., 2023). Therefore, our work generalizes these specific views to a broader class of SSL approaches and likelihood based generative models. In the subsequent sections, we are going to demonstrate an instantiation of this general view based SST clustering and energy-based models. Indeed, are going to provide unified theory allowing to learn backbone classifier network to jointly exploit its generative and discriminative properties.
where = -loy e simig((t,)
4 Unifying Self-Supervieed Clustering and Easee Models
Before demonstrating the unification, we provide an alternative lower bound for cluster- basterr-baedddd SSL to the one obtained in Lemma 9, enabling to simplify the architectural design of the neural network, while ensuring the avoidance of the main failure modes. Then, we provide GEnerative and Discriminative (GEDI) Lower Bound.
4.1 Lower Bound for Cluster- Based SSL
We state the following proposition and leave the proof to Section F in the Supplementary Material. Definition 7. Consider the same conditions in Defonition 5, except for choosing identical auailiary and predictive distributions, namely ((y|z) ply|æ; θ) Softmar(U" (x))))))
Note that the choice of identical auxiliary and predictive distributions is key difference with respect to existing cluster-based SSL. For instance, SwAV avoids learning trivial solutions when optimizing the lower bound in Lemma by introducing asymmetric distributions. Specifically, it considers p(y|>; Q) Softmax(U" g(x)/T) and q(yl=') Sinkhom(StopGraa(("" g(x')/T)), where Sinkhorn and StopGrad are two
For the sake notation simplicity, we omit the dependence 0.

---

Published in Transactions on Machine Learning Research (08/2025)
operators performing the Sinkhorn-Knopp algorithm and stopping the gradients, respectively. Therefore, choice constitutes a simplification of the architectural design, but this relaxation requires the development of new objective to guaranteeing the avoidance of the main training failure modes. The following proposition provide an alternative lower bound for cluster-based SSL (the proof is given in Appendix J, whereas the analysis about failure modes is deferred to Section 4.3)
Proposition Given Definition the expected dula log-likelihood for the probabilistic graphical model in Fig. 2(c) can be alternatively lower bounded as follows:
Ep(simat (l (logp(wim. {mm) - -)((11)) Σ (p(asT(aaaa)) ( (CB(p(pld) 0))}
LiNv(()
CE
p(y)), Σ plus u1(z;;0)
+ const
(3)
(pRIOR(
Additionally, the corresponding mazimum value for the last two addends in Eq. (3) is given by the following inequality:"
(INV(O) + LPRION(O) < Hp(y)m))
The above proposition has interesting implications. First of all, by maximizing the discriminative term (i) with respect to O, we enforce two properties, namely: (i) label invariance, as we ensure that the predictive distributions the discriminative model for sample and its augmented version match each other and (ii) confident predictions, as maximizing the cross-entropy forces also to decrease the entropy of these and by maximizing
distributions. Secondly, by choosing a uniform prior, viz. p(y: Uniform([1...... (PRSOR(O) with respect O, ensure to obtain balancod cluster assignment. This is also commonly done by approaches based on optimal transport objectives and corresponding surrogates, lypically empployed in cluster-basee SSL (Cuturi, 2013; Caron et al., 2018; 2020; Amrani et al., 2022).
4.2 GEDI Lower Bound
For the GEDI instantiation, derive the lower bound the expected log-likelihood by exploiting the bound in Eq. (1) and the one in Proposition 1:
Em( (logg(11s1 (1mm θ)} 2
LGEN !
LINv( + LPRIOR(
(5)
Generative term -CE(p.po)
Discriminative terms
additionally decompose the discriminative model p(y|2;; θ) to obtain pv in similar manner to what have already done in Eq. (2), namely:
xy,x;0) = T(0)
Ee UTa(s//T
6 B((
pιz; V
p(z; θ)
(6)
I(O)
r(e)
where E(x; 9) = log Σÿ= Lolat/ We will shortly analyze the properties of the different objective terms in Eq. (5). For the moment, we finalize the GEDI instantiation by devising the corresponding training algorithm.
Herr, that the predictive model p(y|:; 0) has enough capacity achieve the optimal solution. Indeed, recall that CF(p. q) Hp + KL(pllg). Therefore, maximizing -CF(p.g) q) forces have both KT((pll)) and H, 0.

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm 1: GEDI Training.
2im; L. Iters, SGLD and Adam optimizer hyperparameters: Output: Trained model θ;
Input:
For iter = -,.... Iters;
Generate samples from pe using SGLD;
Estimate A VeLGEN(O) using Eq. (7);
Compute 4y0 = VeLINv(O);
Compute 430 = VeLPRIOR(O);
AO + Σi= 8;;;
< Adam maximizing using Δθ;
Return O;
Learning a GEDI model. We can train the GEDI model by jointly maximizing the objective in Eq. (5) with respect to the parameters O through gradient-based strategies. The overall gradient includes the summation of three terms, viz. -VoCEEp, pe), VeLINv(O) and VeLruton(e)) While the last two gradient terms be computed easily by leveraging automatic differentiation, the first one must be computed by exploiting the following identities (obtained by simply substituting Eq. (6) into the definition of cross-entropy and expanding Vel(@)):
-VoCE(p,
Σ E Ep(x))
Ve log
nVe log (())
iml
y=l
n
E Ep(x))
Ve log y=1
Uniglmsi/T
nLE pl)
Ve log y=1
(7)
Importantly, the first and the second expectations in Eq. (7) estimated using the training and the generated data, respectively. To generate data from po, use sampler based on Stochastic Gradient Langevin Dynamics (SGLD), thus following recent best practices to train beergy-based models (Xie et al., 2016; Nijkamp et al., 2019; Du & Mordatch, 2019; Nijkamp et al., 2020). The whole learning procedure is summarized in Algorithm 1.
Computational requirements. When comparing our GEDI instantiation with traditional SSL training, more specifically to SwAV, we observe two main differences in terms of computation. Firstly, our learning algorithm docs not require to run the Sinkhorn-Knopp algorithm, thus saving computation. Secondly, GEDI instantiation requires additional forward and backward passes to draw samples from the energy-based model Pe- However, the number of additional passes through the discriminative model can be limited by the number of SGLD iterations, necessary to generate data (cf. Experiments).
4.3 Analysis of Loss Landscape and the Triad of Failure Modes
Asymmetries have been playing an important role in self-supervised learning in order to avoid trivial solu- tions/failure modes (Wang et al., 2022). Here, we formalize three main failure modes for clsstrr-based SSL. Then, we study the GEDI loss landscape and show that these undesired trivial solutions are not admitted by our objective. This result holds without introducing asymmetries in the optimization procedure and/or or network architecture.
Let's start by defining the most important failure modes, namely:
Definition 8 (Failure Mode 1 Representational Collapse). There crists a constant vector k E Rh such that for all E R4, g(x) k.
Definition 9 (Failure Mode 2 Cluster Collapsc). There erists a cluster j E [1....,cc] of such that for all x Rdd p(y = j|æ; 0) =
Definition 10 (Failure Mode 3 Label Inconsistency). For all possible permutations fl.....ck .c. c), dalaset D {x,, li ()) its permuted version D = (11); and a loss L(O::),
8

---

Published in Transactions on Machine Learning Research (08/2025)
evaluated at one of the two datasets, have that L(@;D) L(); For GEDI, t, = U gZz) and 4 = UUTgi g(x().
In other words, Definition 8 considers the case where the encoder maps (collapses) every input to the same output. Definition 9 considers the situation where the predictive model assigns all samples to the same cluster with high confidence. And Definition 10 considers the case where a hypothetical adversary swaps the predictions made by the model on different pair of inputs. Ideally, we would like to have an objective that does not admit these failure modes.
Now, we state the properties of the loss landscape of GEDI with the following theorem (we leave the proof to Appendix K):
Thcorem 1. Given definitions 8-10, the following statements tells for a particulor loss, which fasture modes are admissible solutions:
a. LGEN(()) admits failure modes and 3.
b. LINV(O) admits foilure modes 1 and 2.
(PRIOR(O) admits failure modes 1 and 8.
Proof. (Sketch) The proof proceeds by applying the definitions of the various failure families (Definitions 8-10), evaluating the loss terms in each case, and verifying whether their values correspond to global minima. Intuitively, this is feasible because each loss term defines a notion of distance between the ground truth and model distributions.
Importantly, Theorem 1 tells us that LGEN (9) can be used to penalize representational collapse, LINV (e) can be used lo ensure that cluster assignments are consistent with data augmentation, while LeRION can be used to penalize cluster collapse. Consequently, by maximizing the objective in Eq, (5), we are guaranteed to Iearn solutions which are non-trivial. A table summarizing all these propertics given below.
Table 1: Summary of loss landscape
Does | penalize ?? Repr. collapse
Clus. collapse
Lab. Inconst.
CaEN(() (9)
Yes
No
No
No
No
Yes
(PRION( Eq. (5)
No
Yes
No
Yes
Yes
Yes
5
Related Work
We organize the related work according to different objective categories, namely contrastive, cluster-based and non-contrastive self-supervised approaches. Additionally, we discuss recent thcoretical results, augmen- tation strategies as well as connections energy-basee models. For an exhaustive overview of self-supervised learning, we invite the interestod reader to chock out two recent surveys (Jing & Tian, 2021; Balestricro al., 2023).
Contrastive objectives and connection to mutual information. Contrastive learning represents important family of self-supervised learning algorithms, which is rooted in the maximization of mutual infor- mation between the data and its latent representation (Linsker, 1988; Becker & Hinton, 1992). Estimating and optimizing mutual information from samples is notoriously difficult (McAllester & Stratos, 2020), espe- cially when dcaling with high-dimensional data. Most recent approaches focus on devising variational lower bounds on mutual information (Barber Agakov, 2004). Indeed, several popular objectives, like the mutual information neural estimation (MINE) (Belghazi et. al., 2018), deep InfoMAX (Hjelm al., 2018), noise

---

Published in Transactions on Machine Learning Research (08/2025)
contrastive estimation (InfoNCE) (den Oord et al., 2018; O. Henaff, 2020; Chen et al., 2020; Lee, 2022; Xu et al., 2022) to name a few, all belong to the family of variational lower bounds (Poole et al., 2019). All these estimators have different properties in terms of bias-variance trade-off (TTschanen et al., 2019; Song & Ermon, 2020). Our work model contrastive learning using an equivalent probabilistic graphical model and a corresponding objective function based on the data log-likelihood, thus providing an alternative view to the principle of mutual information maximization. This similar in spirit to the formulations proposed in the recent works of (Mitrovic et al., 2021; Tomasev al., 2022; Scherr et al., 2022; Xu et al., 2022). Unlike these works, we are able to extend the log- log-likelihood perspective to other families of self-supervised approaches and also to highlight and exploit their connections to energy-based models.
Cluster-based objectives. There are also recent advances in using clustering techniques in representation learning. For example, DeepCluster (Caron et al., 2018) uses k-means and the produced cluster assignments as pseudo-labels to learn the representation. The work in (Huang et al., 2022) introduces an additional regu- larizer for doep clustering, invariant to local perturbations applicd to the augmented latent representations. The work in (Asano et. al., 2020) shows that the pseudo-label assignment can be seen as an instance of the optimal transport problem. SwAV (Caron et al., 2020) proposes to use the Sinkhorn-Knopp algorithm to optimize the optimal transport objective (Cuturi, 2013) and to perform a soft cluster assignment. Finally, contrastive clustering (Li et al., 2021) proposes to minimize the optimal transport objective in a contrastive setting, leveraging both positive and negative samples. Our work provides a simple yet concise formulation of cluster-based self-supervised learning based on the principle of likelihood maximization. Additionally, thanks to the connection with energy- based models, we can perform implicit density estimation and leverage the learnt information to improve the clustering performance.
Negative-free objectives. It's important to mention that new self-supervised objectives have recently emerged (Zbontar et al., 2021; Grill et al., 2020) as way to avoid using negative samples, which are typically required by variational bounds on the mutual information, namely BYOL (Grill et al., 2020), SimSiam (Chen et al., 2020), DINO (Caron et al., 2021), Zero-CL (Zhang ct al., 2021), W-MSE (Ermolov ct al., 2021), Barlow Twins (Zbontar et al., 2021), VICReg (Bardes et al., 2022a) and its variants (Bardes el al., 2022b), MEC (Liu et al., 2022) and CorInfoMax (Ozsoy et al., 2022). DINO proposes to maximize cross-entropy objective to match the probabilistic predictions from two augmented versions of the same image. BYOL, SimSiam, W-MSE consider the cosine similarity between the embeddings obtained from the augmented pair of images. Additionally, W-MSF introduces a hard constraint implemented as a differentiable layer to whiten the embeddings. Similarly, Barlow Twins proposes soft whitening by minimizing the Frobenius norm between the cross-correlation matrix of the embeddings and the identity matrix. Zero-CL pushes forward the idea of whitening the features by also including an instance decorrelation term. VICReg and its variant extend over Barlow Twins by computing the sample covariance matrix, instcad of the correlation one (thus avoiding to use batch normalization), by enforcing an identity covariance and by introducing an additional regularizer term to minimize the mean squared error between the embeddings of the two networks and to promote the invariance of the embeddings. Similarly, the work in (Tomasev et al., 2022) uses an invariance loss function in conjunction to the contrastive InfoNCE objective. MEC maximizes the log-determinant of the covariance matrix for the latent representation, thus promoting maximum entropy under the Gaussian assumption. Additionally, CorInfoMax extends MEC by introducing a term that enforces the representation to be invariant under data augmentation. In Scction 2.3, we can cast the non-contrastive problem as a minimization of the Kullback Leibler divergence between the latent posterior and a standard normal densily prior. Tn essence, our work allows to compactly represent the family of non-contrastive methods using likelihood-based objective.
Additional objectives. Several works have investigated the relation between different families of self- supervised approaches leading to hybrid objective functions (Kügelgen et al., 2021; Garrido et al., 2023). Tn contrast, our work attempts to provide a unified view from a probabilistic perspective and to highlight /exploit its connection to energy- based models.
The main idea of generative and discriminative training originally appeared in the context of Bayesian mixture models (Sansone et al., 2016). Specifically, the authors proposed to jointly learn generative model and cluster data in each class in order to be able to discover subgroups in breast cancer data. Subsequently, the work in (Liu & Abbeel, 2020) pushed the idea forward and apply it to a supervised deep learning
10

---

Published in Transactions on Machine Learning Research (08/2025)
setting. Instead, our work focuses on self-supervised learning and generative models, thus avoiding the need for additional supervision on the categorical variable y- Recently, the work in (Li et al., 2022) uses the maximum coding rate criterion to jointly learn an embedding and cluster it. However, the training proceeds in a multistage fashion. In contrast, our work provides a simple formulation enabling to jointly learn and cluster the embeddings in one shot. Additionally, we can leverage the generative perspective lo further boost the self-supervised learning performance, as demonstrated by the generation and out-of-distributioo detection experiments.
Theory of selfsuperviee learning. Several works have theoretically analysed self-supervised approaches, both for contrastive (Saunshi et al., 2019; Wang & Isola, 2020; Zimmermann et al., 2021; Tosh et al., 2021; HaoChen et al., 2021; Saunshi et al., 2022) and non-contrastive methods (Tian et al., 2021; Kang-Jun et al., 2022; Weng et al., 2022; Wen & Li, 2022), to motivate the reasons for their successes, identify the main underlying principles and subsequently provide more principled/simplifee solutions. Regarding the former family of approaches, rescarchers have (i) identified key properties, such as representation alignment (i.c. feature for positive pairs need to be close to each other) and uniformity (to avoid both representational and dimensional collapse) (Wang & Isola, 2020; Assran, Mahmoud and Caron, Mathilde and Misra, Ishan and Bojanowski, Piotr and Bordes, Florian and Vincent, Pascal and Joulin, Armand and Rabbat, Mike and Ballas, Nicolas, 2022; Assran et al., 2023), (ii) formulated and analyzed the problem using data augmenta- tion graphs (HaoChen et al., 2021) and (iii) examined generalization bounds on the downstream supervised performance (Saunshi et al., 2022; Bao et al., 2022). Regarding the latter family of approaches, the main focus has been devoted to understanding the reasons on why non-contrastive approaches avoid trivial solu- tions. Tn this regard, asymmetries, in the form of stop-gradient and diversified predictors, are sufficient to ensure well-behaved training dynamics (Tian et al., 2021; Weng et al., 2022; Wen & Li, 2022). Importantly, the asymmetries are shown to implicitly constrain the optimization during training towards solutions with decorrelated features (Kang-Jun et al., 2022). Recent works have also looked at identifying connections between contrastive and non-contrastive methods (Dubois et al., 2022: Garrido et al., 2023; Balestriero & LeCun, 2022) to unify the two families. The work in (Dubois et al., 2022) proposes a set of desiderata for representation learning, including large dimensional representations, invariance to data augmentations and the use of least one linear predictor to ensure good performance on linear probe evaluation tasks. The work in (Garrido et al., 2023) analyzes the relations between contrastive and non-contrastive methods, show- ing their similarities and differences. Both families of approaches learn in contrastive manner. However, while contrastive solutions learn by contrasting between samples, non contrastre ones focus on contrasting between the dimensions of the embeddings. The work in (Balestriero & LeCun, 2022) studies the minima in terms of representations for the different loss functions proposed in the two families. The authors are able to show that such minima are equivalent to solutions achieved by spectral methods. This provides additional evidence on the similarities between the approaches and the possibility for their integration.
Our work provides a unifying view of the different classes, allowing to derive several loss functions in a principled manner using variational inference on the data log-likelihood. Additionally, we provide conditions to learn in principled manner, thus avoiding trivial solutions and the use of asymmetrics.
Generative models and self-supervised learning. Recently, works have considered synergies be- tween self-supervised and energy-based models (LeCun, 2022) for the purposes of out-of-distribution de- tection (Hendrycks al., 2019; Winkens al., 2020; Mohseni et al., 2020). This is a common characteristics of energy-based models and indeed our work highlight the explicit connection with self-supervised learning. To the best of our knowledge, there is only one recent work exploring the integration between self-supervised learning approaches and energy models (Kim & Ye, 2022). The authors propose to use an encrgy-based model to learn a joint distribution over the two augmented views for the same data. The resulting objective can be decomposed into a conditional distribution term, leading to a contrastive learning criterion, and a marginal distribution term, leading to an energy-based model criterion. Therefore, the work only considers the integration between contrastive methods and energy- models. In contrast, our work goes a step forward by showing general methodology to integrate generative and SSL approaches. Moreover, we pro- vide new lower bound for the class of cluster-base based approaches that guarantces the avoidance of important failure modes.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Clustering performance in terms of normalized mutual information (NMI) on test set (moons and circles). Higher values indicate better clustering performance. Mean and standard deviations are computed from 5 different runs.
Dataset
JEM
GEDI no prior
GEDI no inv GEDI no gen
GEDI
Moons 0.0010.00
0.00.10.00 0.00+0.00
0.11-0015 0.22+0013
0.9810.00
0.9410.07
Circles
0.00++000
0.83+0.12
1.00+0.01
There has been also recent. interest in combining VA with self-supervised learning (Gatopoulos & Tomezak, 2020; 'Zhue et al., 2020; Wu et al., 2023; Nakamura et al., 2023; Tong et al., 2024). Our formulation is general enough to encompass ELBO-like objectives like the one used in VAEs. Instead of instantiating the generalive term by leveraging an energy-based model, one could proceed to lower bound the entropy term following traditional ways to derive an ELBO. We are not interested pursue this direction, as we aim devising an objective function which can exploit a simple classifier architecture without the need of additional components, such as decoder network.
6 Experiments
We perform experiments to evaluate the namely an energy-based model JEM (Grathwohl al., 2020), which is trained with persistent contrastive divergence Tieleman (2008) to optimize only the generative term of the GEDI Lower Bound (Eq. (5)), and 2 self-supervised baselines, viz. negative-free approach based on Barlow Twins (Zbontar et al., 2021) and cluster-based approach based on SWAV (Caron et al., 2020). which optimize the objectives in Lemma 6 and Lemma 9, respectively. We also compare with a two-stage generative and discriminative solution originally proposed in (Sansone & Manhaeve, 2022), called GEDI two stage. The whole analysis divided into four main experimental settings, the first part provides empirical validation and intuition on the results of Proposition 1 and Theorem the second part based on two synthetic datasets, including moons and circles, compares GEDT against. the above-mentioned baselines, the third part extends the comparisons to real-world data, including SVHN, CIFAR-10 and CIFAR-100, and finally the last part showcases the utility of the GEDI framework to mitigate the symbol grounding problem arising in neuro-symbolic learning on MNIST data (Harnad, 1990; Sansone & Manhaeve, 2023b; Marconato et al., 2023). We use existing code both as a basis to build our solution and also to run the experiments for the different baselines. In particular, we use the code from (Duvenaud et al., 2021) for training energy-based models and the repository from (da Costa et al., 2022) for all self-super approaches. Our code will be publicly released in its entirety upon acceptance. Implementation details well as additional experiments are reported in the Supplementary Material.
performance of GEDI and its competitors,
6.1 Part 1: Empirical Validation of the Theory
(a) GT GT
(b) JEM
(c) no prior
(d) no ine
(e) no gen
(f) GEDI
(g) GT
(h) JEM
(i) no prior
(j) no inv
(k) no gen
(I)
GEDI
Figure 3: Qualitative visualization of the clustering performance for the different strategies on moons (a-f) and circles (g-l) datasets. Colors identify different cluster predictions. Only GEDI and GEDI no gen are able to perform well on both datasets.
12

---

Published in Transactions on Machine Learning Research (08/2025)
CPRIOR
6
0.50
0.80
LPRIOR
LNNV
1.5
Lenion
0.50
LINV
0.25
0.75
0.65
2-
1.0
4,00
0.70
7060
0.00
5000
5000
5030
5000
Iterations
Rerations
terations
tterations
Iterations
lterations
(a) prior
(b) n10 ircv
(c) GEDI
Lprior
LPRIOR
LINV
LpRiOR
5.0
0.50
1.5
0.50
0.65
0.25
0.25
1.0
0.00
30 70
P0.60
0.00
5000
5000
5000
0
5000 Ieertions
5000
5000
Iterations
Reeations
tterations
Iterations
Rerations
(d) no prior
(e) no ine
(1) GEDI
Figure 4: Visualization of the main training losses. As suggested by Eq. 22 in Proposition 1, we can always debug whether the main failure modes in the triad have occurred by checking the minimum value achieved by the losses and compare it with its corresponding global value. Indeed, the global minimum value of each loss is known, that is In(c) for LPRIOR and 0 for LINV. In both moons (a-c) and circles (d-f) the values are ln) = 0.69 and O, respectively.
We consider two well-known synthetic datasets, namely moons and circles. We use multi-layer perceptron (MLP) with two hidden layers (100 neurons each) for enc and one with a single hidden layer (4 neurons) for proj, we choose = 2 and T(x1zz) = NN, with = 0.03 as our data augmentation strategy. We train GEDI for 7k itcrations using Adam optimizer with learning rate 1e - 3. Similarly we ablate the contribution of the different loss terms by training the GEDI Lower Bound using only LGEN (equivalent. to JEM), without LINV (called GEDI no inv) without CPRIOR (called GEDI no prior). Further details about the hyperparameters
available in the Supplementary Material (Section G). We evaluate the clustering performance both quantitatively, by using the Normalized Mutual Information (NMI) score and qualitatively, by visualizing the cluster assignments using different colors.
From the results in Table 2 and Figure 3, we can make the following observations: (i) GEDI no prior and JEM are subject to cluster collapse on both datasets. This expected as failure mode is not penalized during training due to the omission of Crmon: (ii) GEDI no ine subject the problem of label inconsistency. Indeed, the obtained cluster labels are not informative and consistent with the underlying manifold structure of the data distribution. Again, this confirms the result of Theorem 1, as failure mode 3 could be avoided by the use of CINV; (iii) GEDI no gen achieves competitive performance to GEDI despite the absence of LGEN. While in theory the objective function for this approach admits representation collapse, predicted by our Theorem, in practice we never observed such issue. It might be the case that the learning dynamics of gradient- based optimisation are enough to avoid the convergence to this trivial solution. Finally (iv) GEDI is guaranteed to avoid the most important failure modes and therefore solve both tasks. We will see later the benefits of including also the generative term in the optimization.
An important consequence of Proposition 1 is that can use the training losses to debug whether we achieve the global minimum value and therefore avoid the triad of failure modes. Figure 4 showcases this property.
6.2 Part 2: Generatiee/Diccrinnaaave Comparisons on Synthetic Data
We consider the same experimental setting used in part 1 and train JEM, SwAV and GEDI for 7k iterations using Adam optimizer with learning rate le 1e-3. 3. Further details about the hyperparameters are available in the Supplementary Material (Section G). We evaluate the clustering performance both qualitatively, by visualizing the cluster assignments using different colors, and quantitatively, by using the Normalized
13

---

Published in Transactions on Machine Learning Research (08/2025)
Table 3: Clustering performance in terms of normalized mutual information (NMI) on test set (moons and circles). Higher values indicate better clustering performance. Mean and standard deviations are computed from 5 different runs.
Dataset
JEM
Barlow
SwAV
GEDI no gen
GEDI
Gain
Moons 0.00:10.00
0.2210.10
0.76:10.36
0.00
0.9410.07
+0.22
Circles
0.00++0.00
0.13+0.10
0.00++0.00
0.83+0.12
1.00+0.01
+0.87
(a) JEM
(b) Barlow
(c) SwAV
(d) no gen
(e) GEDI
(f) JEM
(g) Barlow
(h) SwAV
(i) gen
(i) GEDI
(k) JEM
Barlow
(m) SwAV
(n) no gen
(o) GEDI
(p) JEM
(q) Barlow
(r) SwAV
(s) gen
(t) GEDI
Figure 5: Qualitative visualization of the clustering performance for the different strategies moons (a-e) and on circles (f-j) datasets. Colors identify different cluster predictions. Only GEDI and GEDI no gen are able to perform well on both datasets. Qualitative visualization of the generative performance for the different strategies on moons (k-o) and on circles (p-t) datasets. Colors identify different cluster predictions. All GEDI approaches (except for no gen perform comparably well to the generative solution JFM).
Mutual Information (NMI) score. Additionally, we qualitatively demonstrate the generative performance of all approaches.
We report all quantitative performance in Table 3. As expected, Barlow fails to solve both tasks due to the different nature to cluster-based approaches. SwAV correctly solves the task for the moons dataset, while it faces difficulty to solve the one based on circles. This is due to the fact that the optimized bound in Eq. (17) includes several trivial solutions. The introduced asymmetries (such stop gradient and the clustering layers) are not enough to rule them out. This is confirmed by visually inspecting the latent representation and observing that the encoder collapses to the identity function. Therefore, the projector head can solve the task by simply learning lincar scparator. As consequence, we observe that half of the labels are permuted across the two manifolds (refer to Figure 5). In contrast, GEDI can recover the true clusters in both datasets without any additional asymmetry and have a guarantee to avoid the triad of failure modes. Morcover, GEDI able to learn a correct density estimator compared to the SSL basclines (cf. Figure 5). This is an important property that can come in handy for out-of-distribution detection, as we will see later in the real-world experiments.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Table Generative and discriminative performance on test set (SVHN, CIFAR-1O, CIFAR-100). Normalized
mutual information (NMI) and Frechet Inception Distance (FID) are used as evaluation metrics for the discriminative and generative tasks, respectively. Higher values of NMI and lower values of FID indicate better performance. Mean and standard deviations reported for 5 different initialization scods.
Task
Method
SVHN
CIFAR-10
CIFAR-100
JEM
0.00=0.00
0.00
0.00=0.000
Barlow
0.20=0.02
0.17=0.04
0.61#0.05
Discriminative
SwAV
0.21=0.01
0.44
0.51+0.21
(NMI)
GEDI no gen
0.27:1004
0.45:10.00
0.87=0.00
GEDI
0.25-10.04
0.44
0.87
JEM
4:1+
223+11
271 87
Barlow
334428
382122
403428
Generative
SwAV
480
4101:31
4204226
(FID)
GEDI no gen
488
40319
435
GEDI
1931:0
2141:3
226199
Table 5: OOD detection in terms of AUROC test set (CIFAR-10, CIFAR-100). Training is performed SVHN.
Dataset
JEM
Barlow
SwAV
GEDI no gen
GEDI
CIFAR-10
0.73
0.17
0.26
0.1
0.80
CIFAR-100
0.72
0.24
0.32
0.15
0.80
6.3 Part 3: Real-world Experiments SVHN, CIFAR-10, CIFAR-100
We consider three well-known computer vision benchmarks, namely SVHN, CIFAR-10 and CIFAR-100. We use simple 8-layer Resnet network for the backbone encoder for both SVHN and CIFAR-10 (around IM parameters) and increase the hidden layer size for CIFAR-100 (around 4.1M parameters) from (Duvenaud et al., 2021). We use MLP with single hidden layer for proj (the number of hidden neurons is double the number of inputs), we choose h = 256 for CIFAR-100 and h = 128 for all other cases. Additionally, we use data augmentation stralegies commonly used in the SSL literature, including color jitter, and gray scale conversion to name a few. We train JEM, Barlow, SwAV, GEDI no gen and GEDI using Adam optimizer with learning rate 1e-4 and batch size 64 for 20, 200 and 200 epochs each respective dataset (SVHN, CIFAR-10 AND CIFAR-100). Further details about the hyperparameters are available in the Supplementary Material (Section T). Similarly to the toy experiments, evaluate the clustering performance by using the Normalized Mutual Information (NMI) score. Additionally, evaluate the generative performance qualitatively using
(a) Barlow
(b) SwAV
(c) GEDI gen
(d) GEDI
Figure 6: Samples generated by the different discriminative strategies CIFAR-10 running Stochastic Langevin Dynamics for 500 iterations.
15

---

Published in Transactions on Machine Learning Research (08/2025)
p(G]x;;)
Symbolic Program
p0;1
xf @)
p O)
proj
proj
proj
enc
8nc
enc
Figure 7: The full model for the neurosymbolic experiment using GEDI.
the Frechet Inception Distance (Heusel et al., 2017) as well the OOD detection capabilitics following the methodology in (Grathwohl et al., 2020).
From Table 4, we observe that GEDI is able to outperform all other competitors by a large margin, thanks to the propertics of both generative and self-supervised models. We observe that the difference gap in clustering performance increases with CIFAR100. This due to a larger size of the backbone used in the CIFAR100 experiments (cf. the size of the latent representation). In terms of generation performance, GEDI is the only approach that compares favorably with JEM. We provide qualitative set of samples generated by the different discriminative models in Figure 6. Last but not least, investigate the OOD detection capabilities of the different methods. Table 11 provides quantitative summary of the performance for a subset of experiments (the complete set is available in Section J). We observe that GEDI is more robust compared to other discriminative baselines, thanks to its generative nature. Overall, these experiments provide real-world evidence on the benefits of the proposed unification and theoretical results.
6.4 Part 4: Tackling A Non- Trivialvial Instantiation of the Symbol Grounding Problem
For the final task, we consider applying the self-supervised learning approach to a neuro-symbolic (NeSy) setting. For this, we borrow an experiment from DeepProbLog (Manhaeve et al., 2018; 2021). In this task, each example consists of three MNIST images such that the value of the last one is the sum of the first This thus be considered minimal neuro-symbolic tasks, it requires minimal reasoning task (a single addition) top of the image classification task. This task only contains positive examples. We use the neural network from Manhaeve et al. (2018) as the neural backbone. We optimize LNESY instead of LINV, as the NeSy loss also forces a clustering of the digits, but is more informed. To calculate LNESY, we use GEDI to classify the images to produce probability distribution over the classes to 9, and we use the inference mechanism from DeepProbLog to calculate the probability that the sum holds as shown in Figure 7. For this setting, this coincides with the Semantic Loss function (Xu et al., 2018). The NeSy constraint loss is defined as:
two, e.g. 3 S
LNESY (0) - (E(L,(((x;))
where x; is now an arbitrary number of images in each data point. For this experiment, these are the three MNIST images x; ax) x?). The probability of the constraint p(c|x;; Q) is defined in terms of set of rules R and set of probabilistic facts F = [fi.... fn) with probabilities p(f} Q). For this setting, there are 30 probabilistic facts, one for each possible classifications of each digit, with their probabilities is thus parameterized by the neural network. The probability of the facts is thus
ply 0/0:: O), -.. plus 9|m;; θ). p(us 0|0;; 0), ... plu: 9|xr +)
16

---

Published in Transactions on Machine Learning Research (08/2025)
Each subset F C F defines a possible world wr = {al RUF F a), i.e. the set of all atoms entailed by the rules and the facts in F. In this setting, this is a total assignment of classes to digits, and the truth value of the constraint. The probability of the constraint is the sum of the probability of each possible world (i.e. classifications of the digits) in which the constraint holds.
P(o[x;;))
Σ DEWP
P(wr|x;; 0)
(8)
Where the probability of the possible world is defined as the product of the probability of each fact that is true in this possible word, and one minus the probability of each fact that does not hold in the possible world.
P(wF(x;;)) II p(f))x;)) feE
9) II (1 (L-P(()(;0)) GEF(F
(9)
It is specifically interesting use case for representation learning, since when only the constraint probability is optimized, the neural network tends to collapse onto the trivial solution of classifying each digit as a 0, as shown in (Manhaove et al., 2018; 2021; Sansone Manhacve, 2023a;b). This is a logically correct solution, but an undesirable solution. Optimizing the uniformity objective should prevent. this collapse. neural network should be able to correctly classify MNIST digits by using the SSL training objective and the logical constraint. Here, the focus is on the small data regime, and see whether the logical constraint is able to provide additional information. We also look at the performance of SwAV when combined with the logical constraint. Further details about the hyperparameters and experimental setup are available in Appendix M. We evaluate the model by mcasuring the accuracy and NMI of the model on the MNIST test datasct for different nunbers of training examples. The results are shown in Table 6. Here, # Ex. examples indicates the number of addition examples, which each have MNIST digits. The results show that, without the NeSy constraint, the mean accuracy is low for all settings. The NMI higher, however, and increases as there is more data available. This is expected, since the model is able to learn how to cluster from the data, but unable to learn how to classify. By including the constraint loss, the accuracy improves, as the model now has information the NMI, as we have additional information on the clustering which is used by the model. Combining SwAV with the logical constraint severely underperforms. It does not sufficiently enforce the uniformity of the predicted labels, which leads to the a local optimum where the predictions are distributed over 2-3 classes. These results show us that the proposed method is beneficial to learn to correctly recognize MNIST images using only weakly-supervised constraint. Furthermore, we show that the proposed method can leverage the constraint to further improve NMT and classification accuracy.
which cluster belongs to which class. Furthermore, it also has a positive effect
Table 6: The median and standard deviation of the accuracy and NMI of GEDI and SwAV on the MNIST test set after training on the addition dataset. We also include results for learning without GEDI (only optimizing the constraint),, and training GEDI without the NeSy constraint.
No GEDI
Only GEDI
SwAV & NeSy
GEDI & NeSy
# Ex.
Acc.
NMI
Acc.
NMI
Acc.
NMI
Acc.
NMI
100
.10 .00
.00 .00
.08 + .01
.22 + .03
.14 .00
.19 + .00
.42 + .02
.35 + .01
1000
.10 + .00
.00 = 00
.09 + .03
.29 + .06
.10 .0
.36 # .00
.96 - .01
#91
10000
.10 - .00
.00 = .00
.11 + .05
.44 .10
.10 .00
.40 .00
.97 .00
93.000
7
Conclusions and Future Research
We have presented a unified perspective on self-supervised clustering and energy-basod models. The cor- responding GEDI lower bound guaranteed to prevent three main failure modes: representation collapse, cluster collapse, and label inconsistency of cluster assignments. By shedding new light the synergies
17

---

Published in Transactions on Machine Learning Research (08/2025)
between self-supervised learning and likelihood-based generative models, we aim to inspire subsequent stud- ies proposing new implementations of our general methodology, such as novel approaches to integrating self-supervised learning and latent variable models. Additionally, we demonstrate that GEDI can better capture the underlying data manifolds and provide more procise predictions compared to cxisting strategios. We believe that. further progress can be achieved by combining sell-supervised learning with other areas of mathematics, such as topology and differential geometry, for instance guarantce a notion of connectedness when dealing with the manifold assumption. Moreover, we show that GEDI can be easily integrated into existing statistical relational reasoning frameworks, opening the door to new neuro-symbolic integrations and enabling the handling of low data regimes, which are currently beyond the reach of existing self-supervised learning solutions. There are several area of improvements and extension that we will target. First and foremost, we plan in the near fulure to apply the existing solution to larger scale datasets, such as Imagenet. Secondly, the current solution assumes that information about the number classes and the class prior is known advance. This might limit the applicability of the proposed solution to real-world cases, such as open-world set tings or scenarios with long tailed distributions. It will be certainly interesting to generalize the framework to deal with such new settings. Finally, it will be possible to extend the proposed framework towards more object-centric representation learning, thus going beyond the traditional object classification problem.
8 Author Contributions and Acknowledgements
The authors would like thank Michael Puthawala for initial discussion on GEDI. ES had the idea, developed the theory around the three key challenges, implemented the solution for the toy and real cases and wrote the paper. R.M implemented and wrote the experiments for the neuro-symbolic setting and also helped by proof-reading the article. This work received funding from the Horizon Europe research and innovation programme (MSCA-GF grant agrecment n' 101 149800) and also from the Flemish Government under the "Onderzoeksprograamn Artificiele Intelligentie (AI) Vlaanderen" programme. The computational resources and services used in this work were provided by the computing infrastructure in the Electrical Engineering Department (PSI group) and the Department of Computer Science (DTAI group) at KU Leuven.
References
A. Alemi, B. Poole, I. Fischer, J. Dillon, R. A. Saurous, and K. Murphy. Fixing Broken ELBO. In ICML, 2018.
E. Amrani, L. Karlinsky, and A. Bronstein. Self-Supervised Classification Network. In ECCV, 2022.
Y. M. Asano, C. Rupprocht, and A. Vedaldi. Self-Labelling via Simultancous Clustering and Representation Learning. In ICLR, 2020.
Mido Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, and Nicolas Ballas. The Hidden Uniform Cluster Prior in Self-Supervised Learning, In ICLR 2023, 2023.
Assran, Mahmoud and Caron, Mathilde and Misra, Ishan and Bojanowski, Piotr and Bordes, Florian and Vincent, Pascal and Joulin, Armand and Rabbat, Mike and Ballas, Nicolas. Masked Siamese Networks for Label-Efficient Learning. In ECCV, 2022.
R. Balestriero and Y. LeCun. Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods. In NewrIPS, 2022.
R. Balestricro, M. Tbrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Y. Tian, et al. A Cookbook of Self-Supervised Learning. Xiv, 2023.
H. Bao, Y. Nagano, and K. Nozawa. On the Surrogate Gap between Contrastive and Supervised Losses. In TCML, 2022.
18

---

Published in Transactions on Machine Learning Research (08/2025)
D. Barber and F. Agakov. The IM Algorithm: A Variational Approach to Information Maximization, In NewrIPS, 2004.
A. Bardes, J. Ponce, and Y. Lecun. VICReg: Variance-lnvariane--ooaa Covariance Regularization for Self- Supervised Learning. In ICLR, 2022a.
A. Bardes, J. Ponce, and Y. LeCun. VICREGL: Self-Supervised Learning of Local Visual Features. In NewrIPS, 2022b.
L. W. Barsalou. Perceptual symbol systems. Behavioral and Brain Sciences, 1999.
S. Boeker and G. Hinton. Self-Organizing Neural Network that Discovers Surfaces in Random-Dot Stere- ograms. Nature, 851(63566:161-163, 1992.
M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm. Mutual Informa- tion Neural Estimation. In JCML, 2018.
M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep Clustering for Unsupervised Learning of Visual Features. In ECCV, 2018.
M. Caron, Misra, Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In NewrIPS, 2020.
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging Properties in Self-Supervised Vision Transformers. In ICCV, 2021.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In JCML, 2020.
M. Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transport. In NeurIPS, 2013.
V. G. T. da Costa, E. Fini, M. Nabi, N Sebe, and E. Ricci. Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning. JMLR, 23(5)):1-6, 2022.
A. Van den Oord, Y. Li, and O. Vinyals. Representation Learning with Contrastive Predictive Coding, In arviv, 2018.
Y. Du and I. Mordatch. Implicit Generation and Generalization in Energy-Based Models. arXiv, 2019.
Y. Dubois, T. Hashimoto, S. Ermon, and P. Liang. Improving Self-Supervised Learning by Characterizing Idealized Representations. In NewrIPS, 2022.
D. Duvenaud, J. Kelly, K. Swersky, M. Hashemi, M. Norousi, and W. Grathwohl. No MCMC for Me: Amortized Samplers for Fast and Stable Training of Energy-Based Models. In ICLR, 2021.
A. Ermolov, A. Siarohin, Sangineto, and N. Sebe. Whitening for Self-Supervised Representation Learning. In ICML, 2021.
Q Garrido, Y. Chen, A. Bardes, L. Najman, and Y. Lecun. On the Duality Between Contrastive and Non-Contrastive Self-Supervised Learning. In ICLR, 2023.
Gatopoulos and M. Tomczak. Self-Supervised Variational Auto-Encoders. In Symposium on Advances in Approzimate Bagesian Inference, 2020.
W. Grathwohl, K.-C. Wang, J.-H. Jacobsen, D. Duvenaud, M. Norouzi, and K. Swersky. Your Classifier is Secretly an Energy Based Model and You Should Treat It Like One. In ICLR, 2020.
J. B. Grill, F. Strub, F. Altché, C. Talloc, H. Richemond, Buchatskaya, Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, al. Bootstrap Your Own Latent: A New Approach to Self Supervised Learning. NewrIPS, 2020.
19

---

Published in Transactions on Machine Learning Research (08/2025)
J. Z. HaoChen, C. Wei, A. Gaidon, and T. Ma. Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. In NewrIPS, 2021.
Stevan Harnad. The Symbol Grounding Problem. Physica D: Nonlinear Phenomena, 1990.
D. Hendrycks, M. Mazeika, Kadavath, and D. Song. Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty. In NewrIPS, 2019.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to Local Nash Equilibrium. In NewrIPS, 2017.
R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio. Tearning Deep Representations by Mutual Information Estimation and Maximization. ICLR, 2018.
7. Huang, J. Chen, J. Zhang, and H. Shan. Learning Representation for Clustering Via Prototype Scattering and Positive Sampling. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
b. Jing and Y. Tian. Self-Supervised Visual Feature Learning with Deep Neural Networks: A Survey. 1EEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
L. Kung-Jun, S. Masanori, and O. Takayuki. Bridging the Gap from Asymmetry Tricks Decorrelation Principles in Non-contrastive Self-Supervised Learning. In NeurIPS, 2022.
B. Kim and J. C. Ye. Energy-Based Contrastive Learning of Visual Representations. In NewrIPS, 2022.
J. Von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello. Self- Supervised Learning with Data Augmentations Provably Isolates Content from Style. In NeurIPS, 2021.
Y. LeCun. A Path Towards Autonomous Machine Intelligence, 2022.
K. Lee. Prototypical Contrastive Predictive Coding. In ICLR, 2022.
Y. Li, P. Hu, Z. Liu, D. Peng, J. T. Zhou, and X. Peng. Contrastive Clustering. In AAAI, 2021.
Z. Li, Y. Chen, Y. LeCun, and Sommer. Neural Manifold Clustering and Embedding. arxiv 2022.
R. Linsker. Self-Organization Perceptual Network. Compuler, 21(3):105 117, 1988.
H. Liu and P. Abbeel. Hybrid Discimiinative Training via Contrastive Learning. In arKiv, 2020.
X. Liu, Z. Wang, Y. Li, and S. Wang. Self-Supervised Learning via Maximum Entropy Coding, In NewrIPS, 2022.
R. Manhaeve, S. Dumancic, A. Kimmig, T. Demeester, and L. De Raedt. DeepProbLog: Neural Probabilistic Logic Programming. In NewrIPS, 2018.
R. Manhaeve, S. Dumancic, A. Kimmig, T. Demeester, and L. De Raedt. Neural probabilistic logic pro- gramming in DeepProbLog. Artifical Intelligence, 298:103504, 2021.
E. Marconato, Teso, and A. Passerini. Neuro-Symbolic Reasoning Shortcuts: Mitigation Strategies and their Limitations. In Xrx, 2023.
D. McAllester and K. Stratos. Formal Limitations on the Measurement of Mutual Information. In AISTATS. 2020.
J. Mitrovic, B. McWilliams, J. Walker, L. Buesing, and C. Blundell. Representation Learning via Invariant Causal Mechanisms. In ICLR, 2021.
S. Mohseni, M. Pitale, J. B. S. Yadawa, and Z. Wang. Self-Supervised Learning for Generalizable Out-of- Distribution Detection. In AAAT, 2020.
20

---

Published in Transactions on Machine Learning Research (08/2025)
H. Nakamura, M. Okada, and T. Taniguchi. Representation Uncertainty in Self-Supervised Learning Variational Inference. In ICCV, 2023.
E. Nalisnick, A. Matsukawa, Y. W. Tch, D. Gorur, and B. Lakshminarayanan. Do deep generative models know what they don't know? In ICLR, 2019.
E. Nijkamp, M. Hill, S. C. Zhu, and Y. N. Wu. Learning non-convergent non-persistent short-run meme toward energy-based model. In NewrIPS, 2019.
E. Nijkamp, M. Hill, T. Han, S.C.. Zhu, and Y. N. Wu. On the Anatomy of MCMC-Based Maximum Likelihood Learning of Encrgy-Based Models. In AAAI, 2020.
Olivier O. Henaff. Efficiif Image Recognition with Contrastive Predictive Coding. In 1CML, 2020.
S. Ozsoy, S. Hamdan, S. Arik, D. Yuret, and A. T. Erdogan. Self-Supervised Learning with an Information Maximization Criterion. In NeurIPS, 2022.
B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker. On Variational Bounds of Mutual Informa- tion. In ICML, 2019.
E. Sansone. The Triad of Failure Modes and Possible Way Out. In NewrIPS SSL Theory and Practice Workshop, 2023.
E. Sansone and R. Manhaeve. GEDI: GEnerative and Discriminative Training for Self-Supervised Learning. arXiv, 2022.
E. Sansone and R. Manhaeve. Learning Symbolic Representations Through Joint GEnerative and Discrim- inative Training. In ICLR HeSy-GeMs Workshop, 2023a.
E. Sansone and R. Manhaeve. Learning Symbolic Representations Through Joint GEnerative and DIserim- inative Training (Extended Abstract). In IJCAI KBCG Workshop, 2023b.
E. Sansone, A. Passerini, and F. G.B. De Natale. Classtering: Joint Classification and Clustering with Mixture of Factor Analyscrs. ECAI, 2016.
N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, and H. Khandeparkar. A Thcoretical Analysis of Contrastive Unsupervised Representation Learning. In ICML, 2019.
N. Saunshi, J. Ash, S. Goel, D. Misra, C. Zhang, S. Arora, S. Kakade, and A. Krishnamurthy. Understanding Contrastive Learning Requires Incorporating Inductive Biases. In ICML, 2022.
F. Scherr, Q. Guo, and Moraitis. Self-Supervised Learning Through Efference Copies. In NewrIPS, 2022.
J. Song and S. Ermon. Understanding the Limitations of Variational Mutual Information Estimators. ICLR, 2020.
Y. Tian, X. Chen, and S. Ganguli. Understanding Self-Supervised Learning Dynamics Without Contrastive Pairs. In ICML, 2021.
T. Tieleman. Training Restricted Boltzmann Machines Using Approximations to the Likelihood Gradient. In ICML, 2008.
N. Tomasev, I. Bica, B. McWilliams, L. Buesing, R. Pascanu, C. Blundell, and J. Mitrovic. Pushing the Limits of Self-Supervised ResNets: Can We Outperform Supervised Learning Without Labels on ImageNet? ICML Pre- Training Workshop, 2022.
S. Tong, X. Dai, Y. Chen, M. Li, Z. Li, B. Yi, Y. LeCun, and Y. Ma. Unsupervised Learning of Structured Representations via Llosed-Loop Transcription. In CPAL, 2024.
C. Tosh, A. Krishnamurthy, and D. Hsu. Contrastive Estimation Reveals Topic Posterior Information Linear Models. JMLR, 2021.
21

---

Published in Transactions on Machine Learning Research (08/2025)
M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic. On Mutual Information Maximization for Representation Learning. In ICLR, 2019.
T. Wang and P. Isola. Understanding Contrastive Representation Learning Through Alignment and Unifor- mity on the Hypersphere. In ICML, 2020.
X. Wang, H. Fan, Y. Tian, D. Kihara, and X. Chen. On the Importance of Asymmetry for Siamese Representation Learning. In CVPR, 2022.
Z. Wen and Y. Li. The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning. In NewrIPS, 2022.
X. Weng, L. Huang, L. Zhao, R. M. Anwer, S. Khan, and F. S. Khan. An Investigation into Whitening Loss for superviid Learning. In NeurIPS, 2022.
J. Winkens, R. Bunel, A. G. Roy, R. Stanforth, V. Natarajan, J. R. Ledsam, P. MacWilliams, P. Kohli, A. Karthikesalingam, S. Kohl, al. Contrastive Training for Improved Out-of-Distribution Detection. Tn arxiv, 2020.
C. Wu, J. Pfrommer, M. Zhou, and J. Beyerer. Generative- Learning for Self-Supervised Latent Representations of 3D Shapes from Multi-Modal Euclidean Input. arXiv, 2023.
J. Xie, Y. Lu,, S. C. Zhu, and Y. N. Wu. A Theory of Generative Convnet. In TCML, 2016.
Xu, H. Xiong, and G. J. Qi. K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations. IEEE Tronsactions on Pattern Analysis and Machine Intelligence, 2022.
J. Xu, Z. Zhang, T. Friedman, Y. Liang, and G. Broeck. Semantic Loss Function for Deep Learning with Symbolic Knowledge. In ICML, 2018.
J. Zbontar, L. Jing, I. Misra, Y. LeCun, and Deny. Barlow Twins: Self-Supervised Learning via Redun- dancy Reduction. In ICML, 2021.
S. Zhang, F. Zhu, Yan, R. Zhao, and X. Yang. Zero-CL: Instance and Feature Decorrelation for Negative- Free Symmetric Contrastive Learning. Tn ICLR, 2021.
Y. Zhue, R. M. M. Renqiang, A. Kadav, and H. Graf. S3VAE: Self-Super Sequential VAE for Representation Disentanglement and Data Generation. In CVPR, 2020.
R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive Learning Inverts the Data Generating Process. In ICML, 2021.
A Appendix
B Proof of Lemma 3
Definition 11 (Restated). Define the ground truth joint distribution for the graphical model in Fig. 2(a) p( Vim) = I1
with yJ (I,....,n) and 8 della function. Moreover, define the model distribution as p(r1s Vi Q) = P( (1:;; Q) with plyilar A) En coimtatee) sim
R 1XR--0 + R being similarity function, 0 temperature parameter and θ 10, (x set of paramelers.
Lemma 4 (Restated). Given Definition 11, marimizing the InfoNCE objective (den Oord et al., 2018) is equivalent to marimize the following log-likelihood objective:
csimig(zs),(t:))>>>>
Eptero (log (logp(pz.... v1m)))
log
(10)
jum p(a,))
k= estm(g(z)))(eaa))f
Neg. entropy term
Predictive term Ler(9)
22

---

Published in Transactions on Machine Learning Research (08/2025)
Moreover, the marimization of this lower bound is equivalent to solue an instance classification problem.
Proof. We first recall the InfoNCE objective (cf. Eq. 10 in (Poole et al., 2019))
log (
INCE = II,,,, p(x)) o(m,))
(11)
2 (2()5))
This objective can be rewritten in the following way:
1 P(, ((a)) ((8,))
log (x),)))
INCE = -
+ +log(n)
k= ex(eie)))
n F III pla, (0 9(x;))
x
log
e (()))))))))
II p(a))
log
(12)
Ek=1 (x(10((1))))
Next, we highlight an important relation between the ground truth and model log-likelihoods:
Pep vin) flog p( Vi:a)]
= Pep(ssm ) (log (log((mimn Vien 0)) + KT[ppxr Viaa) [p([ Vi:i θ) 2 Fel (log (1 Vimi θ)
(13)
In other words, maximizing the term Ep 1 Mimi θ} is equivalent to minimize the K term, thus solving an instance discrimination problem, where the model matches the instance classifier. This proves the last part of the lemma.
Now, we are left show the first part. We can claborate the above lower bound even further by using the definitions at the basis of the Lemma. Indeed,
Eplasmeri.n) (logp(a Miaii θ)} B1I p(e))ols-s) log II (((a))(
II p(x, 16ls (o:
Σ Ep(r.) (o::: 1]z
Ep(a))) (logp(a;)) + Σ (pgg) i|x,; θ)}
- -HH((1m)) +
Epez,
=-H,(#1mm) + ETTT p(a,)
log - 6)
=-H((Iim)) + B p(2,)
log
(14)
Dk=l
1=1
Now, by defining f(x,2) sim(x, 2)/T and substituting into Eq. 14, we can match with Eq. 12, thus concluding the proof.
23

---

Published in Transactions on Machine Learning Research (08/2025)
C Proof of Lemma 6
Definition 12 (Restated). Define the ground truth joint distribution for the graphical model in Fig. 2(6)
as p(z, E1:, Fimm Fimn = p(z) plll:) &i) with Gaussian priors p(z) = (2(20,,),, I), p(E:)
N(G[0,,~11 and assume the following conditional independencies p(r)( Gi) = =<())() and p(zx(z) = p(a:). Moreover, introduce the following auriliary model distributions, that is q(5i, x/) ((rlenn((a)
enc(x/), 1) and ((=/<rm)) = [= 0, E), with = =i= - Tgg((;) g)t + 31, 3
chosen to ensure the positive definiteness 2f Σ and g 1/n Ex=19(z); Finally, define e {0} as the set of parameters.
Lemma 5 (Restated). Given Definition 12, mazimizing the CorInfoMar objective (Ozsoy et al., 2022) is equivalent marimize the following log-likell boglikkelhhood lower bound:
Ep f (logp(/1.m. x1:1)3 2
Hp((imm)
-Ep(zisa) (/lig/////l/////
Neg. entropy term discriminaaiin SSL L p (9)
(l)l(())) const Continuation of (xr(O)
where the first Leibbeck-Leibler term (L(((=ee))lll(z)) = (L(N(= E)I, I)), the second term K (L(( = KI(N(S(len((xa)) - enc(1!), (1 and const being a constant for the optimization over 0.
Proof. We first recall the CorInfoMax objective (cf Eq. 6 in (Ozsoy et al., 2022))
LCorInfoMaz = P.p(ea..) (log [= + Σ (0 - o(x\)1
(15)
with y a positive scalar to weight the two objective terms.
Next, we derive the log- liceliidl lower bound in the Lomma.
Epizan =in {log p(1 In Eptaimatin { log J.J. p(z) II p(x,))))( [oleeee)l( i=1
= Epteeentiinl log S. p(z) II p(a, ((a:))(
n log II (( T (ax)) + Feteern log S p(t)}
1=1
+ Ep(eime2ii log II IPLENT
Exon iml
(1) const + Ep(rin log J pla))
+ Eptermaiin { log { II II
Exm i=l
=(1,(T + const + Eppari log [ (((](1mm)) p(z)
+ Fyltemetii log II ((Eelar,) a(Es 2e))
plEa)
2 -H -H,((1:)) + const + Epporm {s ((=(21m)) log p(2)
24

---

Published in Transactions on Machine Learning Research (08/2025)
II E P( { o(Eslor, aad ol)lo log g(Ei Dis x(r)
p(E.)
i=1
(3)m))) const + Eprcrm { ((2)21)))) log
p(z)
q( ((2(2115))
p(Ei)
+ II Pa Ep( ((Eil1 2/) log (E) x()) iml
()( Ep (XXL(((((mmmlll(e())
Ep (kaa + const
(16)
Now, observe that the first KL term can be rewritten as follows:
Ep(ern) (orena)(K(ae (KLg(E/(m
KL(N(= [=I /))
Tr(E)
log [
Ep(si.))
+ const'
2
1
-p flog [-]] + const"
2
where the last quality holds whenever g has a batch normalization layer in its output, thus making Tr(E) a constant for the optimization.
The second KL term can be rewritten as follows:
(p an) an (3)
B -
7 2
Σ (ot emc(x?)]13) + const."
Substituting both expresssions into Eq. and choosing enc, we recover the CorInfoMax objective up factor 1/2 and an additive constant.
D Proof of Lemma 9
Definition 13 (Restated). Define the ground Iruth joint distribution for the graphicul model in Fig. 2(c) as p( Iimo Vim) = I. (x.r)/// Q) with E {l,...,c} being a categorical variable to identify one of e clusters, M 9) E, ", where
UT Ga/r
Ihxx is the matria of cluster centers,
-= g(xx)) E Bhxn representation matriz and {0, U} is the set of parameters. Moreover, { (z/)-
introduce the auriliary clustering distrilution q(yrml II
We can state the following Lemma (proof provided in Appendix D)
Lemma (Restated). Given Definition 13, mazimizing the SwAV objective (Caron et al., 2020) is equivalent marimize the following log-likellin likelllld lower bound:
Eplar (log. Zma) 2 (Hp(T1m))
#oostt
Neg. entropy term
We subscripts select and columns. For instance, Usy identify y-th column of matrix
25

---

Published in Transactions on Machine Learning Research (08/2025)
E. ((2 (Futeelet) log pog O) (ui
discriminative SSL term Con(O)
Proof. We first recall the SwAV objective (cf. Eq. and in (Caron et al., 2020))
LSwAv(O)= = Σ ((a (B log θ)
LSwAv(Q) = E pplam iin (Tr(QU* G)} + + (
where where Q Q= ((..:l "" is a prediction matrix of size x and Tr(-) is the trace operator for any input mairix.
Next, derive the lieelikeldoood lower bound:
22 Ep(remati ( (log I'mm)} Ep(zanetian) log E[
V1:m 11 log II [(t=)aloo
8:
-HHp(TIm)) + const
+ + EptxsTTee log Ep(ulzn))
3:
(1 + const
(2 Q) ( (x])
+ Σ EptrOT log Σ (lull1))
1=1
2 (T const.
+ EptasTCeclen Ea(ve (rt log plu pluilzm kzy; θ)
(I + const
+ Σ EplesT(allan
{
Eoinloo log p(u) +) (y( }
We observe that the maximization of this lower bound with respect to θ is equivalent to maximize (swAv((). We can also show the equivalence to LSwAY (Q) by expressing the lower bound in vector format, namcly:
n
(2 ( logp(ulz;;)) Ht Hy(vvlzt)
UT, Ga/t UTGGTT Halurla')
Byculaet) log Σ,
UT G.s T
Ep(esyr(aaln) Eeteilat))
((unlla))) const'
Eplesmnti (Tr(QU G)} Ep af... (HQ + const'
CSswAv (Q) const'
(17)
thus concluding the proof.
26

---

Published in Transactions on Machine Learning Research (08/2025)
E Alternative View of Contrastive SSL
Let us focus the analysis on a different graphical model from the one in Section 2.1, involving an input vector a and a latent embedding = (ef. Figure 1). Without loss of generality, we can discard the index and focus on a single observation. We will later extend the analysis to the multi-sample case. Now, consider e/(x.))
the conditional distribution of a given expressed in the form of energy-based model p(æ|zz; θ) T(=;0) where f
f S x 8h-1 + I is score function and O); Jq ef dz is the normalizing factor." 7 Based on this definition, we can obtain the following lower-bound on the data log-likelihood:
Re()) (logp(x))
= (T(p(=))l<(;;)) + plog
m Ep(z) (log p(; })}
2 Ep(raaat)) (logp(r) 0)) ppx)) (K (XL(( 0))}
Ep(e)(at))) ( θ)} (px (logp((
0))
Epcoa (f(z, 2)) (p())(()=) log O(z;0)
= JELBOEB EBM
(18)
where ((2[z) is an auxiliary density induced by a deterministic encoding function g: g + 5h Eq. (18) provides the basic building block derive variational bounds on mutual information (Poole et al., 2019) as well as to obtain several popular contrastive SSL objectives.
ELBO FBM and variational lower bounds mutual information. Our analysis is similar to the one proposed in (Alemi et al., 2018), as relating ELBOgBM to the mutual information. However, while the work in (Alemi et al., 2018) shows that the third line in Eq. (18) can be expressed as combination of an upper and a lower bound on the mutual information and it studies its rate-distortion trade-off, our analysis considers only lower bounds to mutual information and it makes an explicit connection to them. Indeed, several contrastive objectives are based on lower bounds on the mutual information between input and latent vectors (Poole et al., 2019). We can show that ELBO ELBOgBM EBM is oquivalent to these lower bounds under specific conditions for prior p(2; 0) and score function f.
Specifically, given uniform prior p(z; 0) and f(x, 2) = log p(x) + f(z, 2) for all admissible pair and for some arbitrary function f, we obtain the following equivalence (sce Appendix F for the derivation):
ELBOgBM Ep(a) (logp(z)) + (f
- -p(())(=ta) (log Bacar (e (2' (2)) - ---( IUBA(X,
(19)
where HX) = EEp(a) p(x)) and BA Z) refers to the popular Unnormalized Barber and Agakov bound on mutual information. Importantly, other well-known bounds can be derived from TUBA(X, Z) (cf. (Poole et al., 2019) and Appendix F for further details). Consequently, maximizing FLBO with respect to the parameters of ff (viz. 0) is equivalent to maximize lower bound mutual information.
ELBO and InfoNCE (den Oord et al., 2018). Now, we are ready to show the derivation of ((a.))
the popular InfoNCE objective. By specifying paramertit prior p(z;
where r(0) = Ei_ of de, we achieve the following equality (see Appendix G for the derivation):
((0)
I(a:9)
ELBOEBM ETT planes) Σ log I n Σ cf(
play) Ply [(og](()} [()]]
We assume that f is well-behaved function, such that the intogral value ['2;; O) is finite for all c Sh-1 &g(alz) 5(2 g(x)).
9The term non- paraameraeiic refers the fact that the samples can regarded as parameters the prior and therefore their number increases with the number samples. Indeed, we have (0: (m
27

---

Published in Transactions on Machine Learning Research (08/2025)
= INCE(X,2) Z) EIT; pla,) (log [()))
(20)
Notably, ELBOFRM is equivalent to INCE(X, (X, 2) up lo the term - log r()). Maximizing FLBOBBB has the effect to maximize (NCE(X, 2) and additionally to minimize r(@), thus ensuring that the prior p(z; θ) self- normalizes. However, in practice, people only maximize the InfoNCE objective, disregarding the normalizing term.
F Connection Unnormalized Barber Agakov Bound
Firstly, we recall the derivation of the Unnormalized Barber Agakov bound (Poole et al., 2019) for the mutual information Ix,z, adapting to our notational convention. Socondly, we derive the equivalence relation Eq. (19).
1x,2 E Ep(r,)) log p(z|2) p(x)
Ep(x)a(=1)) log p(x)))g(<1=)
p())a(a)=)
log p(z22)
log g(x(2)
(p(e)a(s1e)
E (()))(=()))
(x122)
p(x)
(<112))
Eg(z) ( (LDD p(x)g((1)))
log p(x) S
log ((r222
E ((()((=)))
p(x)
log p(x)ef(a))
En(a)o(aaa)
p(z)
Ep(x))(=11))
log 2(=)
Ep(a)a(=1a) (fxx Epceig(sta) flog Z(z)}
Ep(a))(<)))) (f(x (x)x,,))))))))) =)} -Bpaya((aa) (log Ep(ar) {e (e())))))
(X)
where we have introduced both an auxiliary encoder ((=[x) and an auxiliary decoder defined as g(x(2) = Pla))
2(2) Now, we can use the assumptions of uniform prior p(2; θ) and f(r, 2) p(x) + f(x, 2) to
achieve the following inequalities:s
ELBO Ep(a)a(=le) (f(2,)) + Epp m) log p(2;
T(z;0)
Ep(a)o(=(a) [f(x,))) Ep 8) [log[ θ)}
Bp(a)a(*(a) (logr(z) + f(r, =)} - Bp(a)a(s (log Ep(wy) (e's)))
(10g(((==)) + E Ee =)} - (p(x)g(s)a) (log 2 te (ef(r ()) -X((X) (f En(e) flog (ell''
H(X TURA(X,
F.1 Other Bounds
Notably, (XB Z) cannot be tractably computed due to the computation of Z(2). Therefore, several other bounds have been derived (Poole et al., 2019) to obtain tractable estimators optimization objectives, namely:
28

---

Published in Transactions on Machine Learning Research (08/2025)
1. ITUBA(X, 20,10 (obtained using the inequality logs s s/8s 8 log r/ for all scalar 8,1/ >> 0)
10 which can be used for both optimization and estimation of mutual information
IUBA(X.Z) ppz..) (f 2)} ((())((()=)) flog [()) pll,,)) (f(x,)))
(())
+ log (1=) -1}
E. p(w)((s))))
17(2)
[TUBA(X,
2. INwJ(X, Z), which can be used for both optimization and estimation of mutual information (ob- tained from (XUBA(X, 7) by imposing 772)= e)
Z(z)
}
TrUBA(X, 2) Ep(.,)) (f(,, Bp(()(()aa)
+ log n1())
17())
[((2)
Ep(a,)) (f(,, Ba(a((aaa)
+
Ep(a,s) 2)} 1 (x(o)ttt))(((22)
Epte.a) ((x,2) 2)} Epcerpce)atell) (e (e INWJ(X,2)
(f(2,
G
Derivation of InfoNCE
((ne))
Consider a prior p(z)
where r(0)= f ". Ei_ (f( dz,
[(,,))
((
ELBOEBM = Ep(a)otsta) (f(,, 2)} (a flog (2; θ)} + (lo
Ep(r) (f(1), [ [2; 0)
(lgg Q))
ple (2), =)} - Pete log
1
e (ze.z)
k=1
(log (
Ey(an )lees) (((s))) flog e (01.2)) - Bpcar (o(s)) (o p(e.))aa)))))
log Σ e (wxka)
k=1
Ep (((a)) p(a.,) (log [())]]
((
-ppte..))pess log 1 k= (f(ezee)
Pp(t))((2))))) (. (log (()))
(11121))
log
p(e (log))) [(0)]
k= ef(unnn))
log
Plo. ((0))
n
En ef(ae)z))
(1())))))
Ep(e in (ttppptaaa.......e.......
log EE Ek=l (x
Peter flog (())
INCE (x,c)(X2) E
B p(z,) flog[(e))
10T in TURA stands for tractable.
29

---

Published in Transactions on Machine Learning Research (08/2025)
H Derivation of ProtoCPC: A Lower Bound of InfoNCE
ProtoCPC (Lee, 2022) can be viewed as a lower bound of the InfoNCE objective. To save space, we use notation E to refer to E III p(15,1z)) Therefore, we have that
log (a))))))
INCE= =
Σ k= e
Pe. (1zz) logg((e)) (xx)
log
17
>n k=1 e lompe(es)
E com Pl(m;;) toget
Lean pe(ot) log >e
E
log
1 Ek=1 Cc (o(
11
1=1
Ee pe(ors) loge loge
Les p2(())) log 20 oo 1r
log
k= CE Pel Po(on) loge bay
11
1 1
cm) (o((a)) log Ese, e
e 2 (() foiee
1
log co's
>g Hh (1F)) log e taie tTa
log
>
11
1=
log Dee co 1To
E (x:)) log Piiz TTo
E
log
12 k= C pflize) log 2o
cml PC Po( TTo
log
11 k=1
ous
log
kk=1 Dom Pe (E
e Ee Polo))) ITo
log
E.n Pelei)sea
log
Ee Cureltt
(e.) 22 /To
log
PratoCPC
Specifically, the second equality comes from the fact that student and teacher networks, respectively, and pz(x) is the e-th entry of the vector obtained by applying a softmax on the embedding of the corresponding network. The incquality in the derivation is obtained by applying Jensen's inequality to the denominator. Finally, d² 1 Eb (x())) corresponds to the prototype for class
(x(2x) logr?(z), where 8, / stand for
n
30

---

Published in Transactions on Machine Learning Research (08/2025)
Details about Negative-Fres Methods
We can specily different definitions for LNF(O), namely:
1. Barlow Twins. The approach enforces the cross-correlation matrix to be close to the identity matrix:
(NF(O) -|ICCCrrr(G o A -I|
where || || is the Frobenius norm, A JA (1 -),, I is the identity matrix, is a positive hyperparameter, J is a matrix of ones, CCorr(G, H is the sample cross-correlation matrix,
G =
E Ru and G" = ((( E Ru*h and
g(x:)) = BN(Proj(enc(z:))
Note that BN is a batch normalization layer, Proj is a projection head usually implemented using multi-layer perceptron and enc is the encoder.
W-MSE. This approach similar to Barlow Twins. The main difference lie the fact that sample cross-correlation matrix is enforced to be identity matrix thanks to whitening layer:
LNF(O) -Tr(G G
where Tr(G @ G') computes the trace of the outer product for matrices G [g(21),.... g(z,))² E Rach and G' [o(x(), ,(())))) E Raxh
g(n;;) = L2-Norm (Wott
L2-Norm is a normalization layer based on L2 norm.
VICReg. This approach attempts simplify the architecture of Barlow Twins by introducing an invariance regularization term in the score function, thus avoiding to use a batch normalization
layer:
CxF(O) - -Trr(G - G') (G G')) (nlu(G)+ + vlw(G)++++++ w(G')
where A, v positive hyperparameters and G (o(x1)....,g("")))" E Raxh, The first addend enforces the representation to be invariant to the data augmentation, whereas the other two addends enforee the sample covariance matrix to be di- agonal. Indeed, the second addend forces the diagonal elements of the sample covariance matrices to be unitary, namely:
Raxh and G'
(G)) = Σ max 1 VarlG.)) +
/
where e > 0 is used to avoid numerical issues, Var computes the variance for each column of matrix G. The third addend ensures that the off-diagonal elements of the sample covariance matrix approach to zero:
c(G) |[Cooo(() @ T)l
with Cou( H) = G.G. Therefore, these last two addends have a similar behaviour the score function of Barlow Twins. Finally, the resulting function is simplified.
g(x;) Proj(enc(z:))
31

---

Published in Transactions on Machine Learning Research (08/2025)
J Proof of Proposition 1
Proposition 2. (Restated) Given Definition 7, the expected data log-likelihood for the probabilistic graphical model in Fig. 2(c) can be allernatively lower bounded as follows:
Epla a
(I. (llggppp11 d)m) 2 Hp(xim)
Bp (CElpluldd: θ))
Livv(()
CE
p(u:), Σ p(us Viloz; 0)
+ +onst
(21)
(PRIOR(()
Additionally, the corresponding mazimum volue for the last two addends in Eq. (21) is given by the following inequality:"
(I) + (PRIOR(O) S (y1m)))
(22)
Proof. Let's start to rewrite the likeliieldoodd term.
Ep(zue flog 1in, =1 = Eplarm
log p(x)) To
Mi:m 1=1
= Eepeemit
log II
i=1 y4
(H + const
Σ
Bpan)TEc log Σ p(ullz;
(23)
1=1
Now add the zero quantity log Ey,.. p(y):)) to the previous equation:
} log
Eq. (23) -Hp(11m)) + const + Σ E log Ep(ulze;()) 0)
-log Σ p(y1n))
(24)
Vien
((2)((2) for any given We lower bound the previous equation by exploiting the fact that E. auxiliary discrete distribution g, viz.:
a Ep(as)((s(aa)) log E
Eq. (24) 2 Hp(x)m) const
2/1
+ log Σ p(yum)g(11s))
(25)
Mian
Now, by applying Jensen's incquality to the last two addends in Eq. (25), by defining a(yim) 1 - p[,,l=g θ) and considering Definition 10, we obtain the following lower bound:
Emeuless θ) logr(ul ))
Eq. (24) 2 - Hp(xim)) + const + Σ Ep(ast((a(aa))
Σ M1im
p( log Σ p(u |x,, 9)
(26)
Here, assume that the predictive model plyl:; θ) has enough capacity achieve the optimal solution.
32

---

Published in Transactions on Machine Learning Research (08/2025)
Additionally, by substituting p(y1:)) Ih p(y)) into the above bound, achieve the following equality:
n
Eq. (26) Hp(01m)) + const
B (((ri)T(a(laa))
M
p(u:) log
p(us = Bil 0) ) Σ j=1
(27)
Vi
And by rewriting the last two addends in Eq. (27) using the definition of cross-entropy, we obtain our final result.
Now, we can conclude the proof by looking at the maxima for LINV and LPRIOR- Indeed, we observe that both terms compute a negative ross-entropy between two distributions. By leveraging the fact that C)(p,)) Hp + (pllg) for arbitrary distributions P.q, we can easily see that the maximun of LNNV is attained when the term is 0 (corresponding to minimal entropy and minimal KL), whereas the maximum of LPRSOR is attained when the term is equal to - -Hp(y:) (corresponding to minimal KT).
K Proof of Theorem 1
Proof. The overall strategy prove the statements relies on the evaluation of the loss terms over the three failure modes and checking whether these attain their corresponding maxima.
Let's start by proving statement a and recalling that LG D) =CCF(pppe) Firstly, we test for failure mode (i.c. representational collapse). We observe that for all x E Rdd
(e( y= (
((O)
thus (z( assigns constant mass everywhere. Clearly, pe is different from p. Therefore, -C'(pp pe) < -CE(p,,) p) and failure mode not admissible. Secondly, we test for failure mode 2 (i.e. cluster collapse). We can equivalently rewrite the definition of cluster collapse by stating that there exists j {I,...,ch, such that for all E Rd and for all # j, gf)(() Uzz((z) cc. Additionally, observe that
Ey= (TO(A)/T
(x(x)=
J EEizs Uholo)/T dx
1+ Ertt ((U
Evtre(U e fole))/
10
jeisss)/
1 Eyee ell (e(e)
dr
UTolo)tT f Tgalww dr
(28)
where we have used the failure mode condition to obtain the last equality. Now, note that Eq. (28) defines a standard easeg model. Consequently, given enough capacity for the predictive model, it trivial verify that there exists θ such that the condition about failure mode is met and pe equal p. Cluster collapse is therefore an admissible solution. Thirdly, test for the inconsistency of cluster assignments. Indeed, we have that
LGEN (O,D)= =
- Ep(zy flog (x:)) i=1
33

---

Published in Transactions on Machine Learning Research (08/2025)
1
Σ y== el(lv)/T
log
S L el(l))/d de
i=1
15
Σ Zy=1 elilu)iT
E Ep(z
log
(29)
y= eti(ul/Tdx
i=1
where t(1) (zz)( Similarly, we have that
LGEN (0,D")
En- Ey-1e cl16)00/
Bp(aem)
log
from Eq. (29) i=1
y=1 c(-(1)()
E Ep(erc))
log
Ey=1 elUlN/
Ep(ar) log
12 dl(l))/ddr
7(1)
LGEN (O, D)
Hence, failure mode an admissible solution.
Let's continue by proving statement and recalling that
(oC LINv(O, = Σ
())
(30)
1=1
Firstly, test for reprosentational collapse. In this case, have that for all i E {1....,..}
plyil θ) p(ul2 θ) Softmax(U" k/T)
Based on this result, we observe that the cross-entropy terms in Eq. (30) can be made by proper choice of h. Therefore, represontational collapse is an admissible solution. Secondly, we test for cluster collapse. Here, is easy see that the cross-entropy terms in Eq. (30) are all 0. Therefore, also cluster collapse is admissible. Thirdly, test for the inconsistency of cluster assignments. On one hand, we have that the cross-entropy terms for (9,v(9, D)) in Eq. (30) can be rewritten in the following way:
Ch(plysaa Q), p(y((rn)))
etilun//
=CE
(31)
ES e(lu)//
ES etilu)/T
and the optimal solution is achieved only when E for all i (l.....n.. On the other hand, the cross-entropy terms for (Inv((, D") are given by the following equality:
CE(p(ys p),p(()));)))
Gluil/
=CE
(32)
eionr Eg elt(o)lu/TT
However, the optimal solution cannot be achieved in general as 12 & bx(()) for some i E fl, nn. Therefore, CINV is not permutation invariant to cluster assignments.
Let's conclude by proving statement and recalling that
[PRIOR(O, = n
- CE p(u)), Σ Eplu=vlann
(33)
1nndeed, note that 1/ (x(1) for all occurs only when of the first. failure modes.
34

---

Published in Transactions on Machine Learning Research (08/2025)
Firstly, test for representational collapse. One can easily observe that if enc(z) = for all E RR pyy 2; )) becomes uniform, namely p(y|æ;; O) = 11cc for all y E {1, [T,...,c} for all i E (I,...,,r) m). Now, since p(y+ = 1/c for all i . n.), the cross-entropy terms in Eq. (33) reach their maximum value - -p)( for all é E (1...., n). Therefore, representational collapse attains the global maximum of LPRIOR and an admissible solution. Secondly, we lest. for cluster collapse. By using the definition of cluster collapse, we observe that
e). Consequently, دi= P(u yilm; 2 1/c
1
{
0 = ye j
plu = 9al θ)
(34)
Therefore, the resulting distribution is non-uniform, differently from p(v)). The cross-entropy terms in Eq. (33) are not optimized and cluster collapse is not admissible. Thirdly, we test for the inconsistency of cluster assignments. We observe that
p(u Ma O)
y=1 chi(0)/T
(35)
Ly=1 (2xc))(1/TT
which is permutation invariant to cluster assignments. Consequently, also LPRIOR(O, LPRION(O, D²). This concludes the proof.
L Hyperparameters for Synthetic Data
For the backbone enc, 2 neurons and ReLU activation functions. For the projection head proj (f for GEDI and its variants), we use a MLP with one hidden layer and 4 neurons and an output layer with 2 neurons (batch normalization is used all layers for Barlow and SwAV as required by their original formulation). All methods use a batch size 400. Baseline JEM (following the original paper): Number of iterations 20K; learning rate le 3; 0.9, 82 0.999); SGLD steps 10; buffer size 10000; reinitialization frequency 0.05; 0.01", SGLD noise 0.01. For the self-supervised learning methods, please refer to Table 7.
MLP with two hidden layers and 100 neurons per layer, an output layer with
Adam optimizer(B]
SGLD step-size 2 We also provide an analysis of sensitivity to hyperparameters for GEDI (Figure 8).
Table 7: Hyperparameters used in the synthetic experiments.
Methods
Barlow
SwAV
GEDI gen GEDI
Iters
20k
Learning rate
le - 3
Optimizer
Adam 81
0.9, B8 0.999
Data augmentation noise
0.03
SGLD steps T
Buffer size [B]
10000
10000
Reinitialization frequenc
0.05
0.05
SGLD step size
0.01" 0.012
0.01"
SGLD noise
0.01
0.01
Weight for LORN
Weight for LINV
50
50
Weight for LrRIOR
10
10
M Hyperparameters for SVHN, CIFAR10, CIFAR100
For the backbone enc, we use a ResNet with 8 layers as in (Duvenaud et al., 2021), where its architecture is shown in Table 8. For the projection head proj (f for GEDI and its variants), we ML. with one hidden
35

---

Published in Transactions on Machine Learning Research (08/2025)
Label invariance
moons: 2
Label invariance
circles: 2
0.0
0.0
10.0
10.0
20.0
20.0
30.0
30.0
40.0
40.0
50.0
50.0
20
0
20
30.0
40.0
50.0
0.0
10.0
20.0
40
50.0
Class uniformity
Class uniformity
(a) Moons
(b) Circles
Figure 8: Sensitivity analysis on the discriminative performance of GEDI fr different loss weights (in the range (0, 10, 30,30,40 40, 50)). Performance are averaged over different random seeds. Yellow means perfect NMI.
Table 8: Resnet architecture. Co,, applies a 2d convolution to input with B channels and produces an output with C channels using stride (1, 1), padding (1, 1) and kernel size (A, A).
Name
Layer
Res. Layer
(3.32D(3.3.F)
AvgPool2D(2)
(ea))REL.U00.2)
Conv2D(3.F.FF AvgPool2D((2)
Block
Conv21((13.F)) no padding
Sum
LeakyRELU(0.2))
Conv2D(3.F.F)
Block 2
LeakyRELU(0.2)
C3.v2)((,F.F)
AvgPool2D(2)
LeakyRELU(0.2)
Conv2D(3,F.F)
Block
LeakyRELU(0.22)
Conv21)(3.F.F)
LcakyREEJUU0.22)
Conv2D(3.F.F)
Block
(o.kyRELU(0.22)
Conv2D(3.F.F)
(vgPool2IDall))
layer and 2 F neurons and an output layer with F neurons (batch normalization is used in all layers for Barlow and SwAV as required by their original formulation + final L2 normalization). F = 128 for SVHN, CIFAR-10 (1 million parameters) and F 256 for CIFAR-100 (4.1 million parameters). For JEM, we use the same settings of (Duvenaud al., 2021). All methods use batch size of 64. Bascline JEM (following the original paper): number of epochs 20, 200, 200 for SVHN, CIFAR-10, CIFAR-100, respectively; learning rate le 4; Adam optimizer; SGLD steps 20; buffer size 1 10000; reinitialization frequency 0.05 SGLD step-size
36

---

Published in Transactions on Machine Learning Research (08/2025)
Table 9: Hyperparameters (in terms of sampling, optimizer, objective and data augmentation) used in all experiments.
Class
Name param.
SVHN
CIFAR-10
CIFAR-100
MNIST
Addition
Color jitter prob.
0.1
0.1
0.1
0.1
0.1
Gray scale prob.
0.1
0.1
0.1
0.1
0.1
Data augment.
Random crop
Yes
Yes
Yes
Yes
Yes
Additive Gauss. noise (std)
0.03
0.03
0.03
0.2
0.2
Random horizontal flip
No
Yes
Yes
No
No
SGLD iters
20
20
20
20
Buffer size
10k
10k
10k
10k
10k
SGLD
Reinit. frequency
0.05
0.05
0.05
0.05
0.05
SGLD step-size
1
1
1
SGLD noise
0.01
0.01
0.01
0.01
0.01
Batch size
64
64
64
60
Epochs
20
200
200
Adam $1
0.9
0.9
0.9
0.9
0.9
Optimizer
Adam 82
0.999
0.999
0.999
0.999
0.999
Learning rate
le 4
le
le
le 4
1e
L2 regularization
0
1e 4
1e 4
LGEN
1
1
1
LINV
50
50
50
50
0
Weights for losses
LPRIOR
25
50
25
400
LNESY
0
25
#. For the 3 different datasel sizes (100, 1000, 10000), we trained for 500, 100 and 25 epochs respectively.
1; SGLD noise 0.01; data augmentation (Gaussian noise) 0.03. For the self-supervised learning methods, please refer to Table 9.
N Additional Experiments on SVHN, CIFAR-10, CIFAR-100
We conduct linear probe evaluation of the representations learnt by the different models Table 10. These experiments provide insights on the capabilities of learning reprosentations producing lincarly separable classes. From Table 10, observe a large difference in results between Barlow and SwAV. Our approach provides interpolating results between the two approaches.
We also provide additional qualitative analyisis on the generation performance on SVHN and CIFAR-100. Please, refer to Figure 9 and Figure 10.
Finally, we evaluate the performance terms of OOD detection, by following the same methodology used in (Grathwohl et al., 2020). We use the OOD score criterion proposed in (Grathwohl et al., 2020), namely s(x) = -1 po (x 22. From Table 5, we observe that GEDI achieves almost optimal performance. While these results are exciting, it important
dz
mention that they are not generally valid. Indcod, when training on CIFAR-10 and performing OOD evaluation on the other datasets, we observe that all approaches considered
achieve similar performance both CIFAR-100 and SVHN, suggesting that all datasets in-distribution, see Table 11. A similar observation is obtained when training CIFAR-100 and evaluating on CIFAR-10 and SVHN, see Table 12. Importantly, this is phenomenon which has been only recently observed by the scientific community on generative models. Tackling this problem is currently out of the scope of this work. For further discussion about the issue, we point the reader to the works in (Nalisnick et al., 2019).
37

---

Published in Transactions on Machine Learning Research (08/2025)
129
115
94 16
1913 19
69
D0 13
14 16
24 19
(a) Barlow
(b) SwAV
(c) GEDI gen
(d) GEDI
Figure 9: Qualitative visualization of the generative performance for the different discriminative strategies
SVHN. Results are obtained by rumning Stochastic Langevin Dynamics for 500 iterations.
(a) Barlow
(b) SWAV
(c) GEDI no gen
(d) GEDI
Figure 10: Qualitative visualization of the generative performance for the different discriminative strategies on CIFAR-100. Results are obtained by running Stochastic Langevin Dynamics for 500 iterations.
Table 10: Supervised linear evaluation in terms of accuracy on test set (SVHN, CIFAR-10, CIFAR-100), The linear classifier is trained for 100 epochs using SGD with momentum, learning rate 1e and batch size 100. Mean and standard deviations are computed over results from different initialization seeds.
Dataset
JEM
Barlow
SwAV
GEDI gen GEDI
SVHN
0.20.10.00
0.75.10.01
0.4410.04
0.59=0.02
0.54110.01
CIFAR-10
0.24.10.00
0.6511.00
0.50
0.63-0.01 0.31100.11
0.63110.01
CIFAR-100
0.03.10.00
0.2710.01
0.1410.01
0.3110.01
Table 11: OOD detection in terms of AUROC on test set (SVHN, CIFAR-100). Training is performed on CIFAR-10.
Dataset
JEM
Barlow
SwAV
GEDI no gen GEDI
SVHN
0.44
0.32
0.62
0.11
0.57
CIFAR-100
0.53
0.56
0.51
0.51
0.61
Table 12: OOD detection in terms of AUROC on test set (SVHN, CIFAR-10). Training performed on CIFAR-100.
Dataset
JEM
Barlow
SwAV
GEDI no gen
GEDI
SVHN
0.44
0.45
0.30
0.55
0.53
CIFAR-10
0.49
0.43
0.47
0.46
0.48
0
Details on the MNIST addition experiment.
The hyperparameters for the experiment without and with the NeSy constraint reported in Table 9. The data was generated by uniformly sampling pairs b such that
< 9, o
< 9 and 0 <
38

---

Published in Transactions on Machine Learning Research (08/2025)
a+bs 9. For each triplet (a, b, c), assigned to a, b, c, random MNIST images with corresponding labels, without replacement. For the experiment with the NeSy constraint, use slightly different uniformity regularization. We maximize the entropy of the mean output distribution for each batch, ef. Manhaeve et al. (2021).
0.1 Replacing LINV with LNESY
For these experiments, we replace LINV with LNESY as the NeSy loss also forces clustering of the digits, making LINV redundant. Figure 11 shows that both the NeSy Loss and the clustering loss progress similarly.
NeSy loss
Clustering
2
2500
2500
(a) LNESY
(b) LINN
Figure 11: Comparing NNSSY with LINV on the Addition experiment, where only the NeSy loss optimized.
39


Reviews:
[{"review_id": "IzrqVjuLPI", "reviewer": "Reviewer_6oNa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The presentation of SSL methods as probabilistic models provides a nice unifying view of a wide variety of methods. This interpretation seems natural and could lead to developing new methods, like GEDI.\n2. The GEDI loss formulation seems more natural than the cluster-based family presented in Section 2.3. Each loss term has a clear and natural interpretation, and the formulation avoids ad-hoc fixes like stop gradients and alternating optimizations used in prior work.\n3. The analysis of the failure modes is quite nice and very intuitive. It is clear how each loss term has a role in preventing a kind of learning collapse that cannot be avoided by the other loss terms.\n4. The experiment section shows convincing ablation of loss terms and comparison to related cluster-based SSL. On the scale of CIFAr-10 and CIFAR-100, the proposed method can outperform other clustering SSL by a significant in some cases and it is the only method with any kind of generative capabilities (however, the generative capability is very limited, and I am not sure how essential it is). The neuro-symbolic grounding experiment offers some evidence that SSL can improve existing methods in this area.\n\n**Weaknesses**\n\nThe main weaknesses are related to the experimental evaluation.\n\n1. The scale of the proposed method is limited to small-scale datasets like CIFAR-10 and CIFAR-100. ImageNet scale experiments are standard in this area. How well does the method perform at scale? Does the EBM loss term and MCMC learning limit the scalability of the proposed method?\n2. Standard SSL experiments like linear probing, few-shot learning, and transfer learning using SSL features are missing. The paper would be much stronger if these standard experiments were included.\n3. It might be good to compare NeSy + Prior SSL (like SwAV) and compare that to NeSy + GEDI in Section 6.4.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies probabilistic interpretations of self-supervised learning methods and proposes a novel self-supervised learning paradigm based on this analysis. The paper begins with discussing three families of self-supervised algorithms (Contrastive, Negative-Free, and Cluster-Based) and presents a probabilistic interpretation of each family, which incorporate a wide variety of popular SSL algorithms as special cases. Furthermore, the work observes that the entropy term that is often bypassed in SSL frameworks can be estimated by a loss term of involving an unconditional EBM derived from the probabilistic SSL interpretation. Next, the GEDI method is proposed, which reformulates the Cluster-Based learning objective into three terms: a term that encourages two views of the same image to be sorted to the same cluster, a term that encourages marginal cluster labels to follow a uniform prior, and an EBM term that gives the cross entropy between the marginal model and marginal data distributions. The first two terms are learned in a supervised way, and the last term uses standard MCMC EBM learning methods. An analysis of failure modes presents three forms: representational collapse, cluster collapse, and label inconsistency. The failure modes are prevented by the generative, prior, and cluster classification terms respectively. The experiment section first presents results on toy experiments, ablating the three loss terms and outperforming related clustering methods. The next experiment examines SSL on SVHN, CIFAR-10, and CIFAR-100, showing better clustering performance than related SSL methods and better generation results (of which the presented method is the only one that can resemble the training data). OOD detection is also examined. Finally, a symbol grounding experiment is performed on MNIST, showing that incorporating GEDI with existing neural grounding frameworks shows significant improvement over the baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My suggested changes would be to strengthen the experimental section by including large scale data like ImageNet (or at least smaller version like Tiny Imagenet) and to investigate linear probing, few-shot, and transfer learning with the SSL features.", "broader_impact_concerns": "None were presented, but this does not impact my assessment of the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f9tzq0jeu8", "reviewer": "Reviewer_eh7p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- Unification of different approaches to SSL is interesting and important, and the proposed probabilistic formulation looks reasonable for that purpose.\n- The experiment is done not only with synthetic data but also with more realistic image data. Also the method is applied to a more non-trivial task.\n- Related work is intensively surveyed.\n\n### Weaknesses\n\n- Although the paper is nicely written, as a reader who is not very familiar with SSL literature, I felt adding a little more explanation would be helpful, as detailed in the next form.\n- The first two views presented in Section 2 (contrastive and negative-free SSL) have much less attention than cluster-based SSL, in the subsequent parts of the paper. It is not problematic to focus on one of them, but it would further strenghten the paper to give some more discussion on the other SSL approaches. For example, can we think of the unification of contrastive (or negative-free) SSL with generative modeling as well? What is the potential difficulty? Any existing work on it? etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A framework combining energy-based generative modeling and clustering-based representation learning is proposed. It is based on a probabilistic formulation of cluster-based self-supervised learning. The integration with generative modeling is introduced through lower-bounding the entropy term appearing in the probabilistic formulation. The authors analyze the derived objective function and show it can avoid three failure modes that cluster-based SSL often encounters. The utility of the proposed method is demonstrated with empirical evidence.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some parts of the paper seem to overly rely on a reader's knowledge of recent SSL studies. Adding more basic explanation would be highly helpful.\n\n1) In Section 2, the ground truth distributions and the model distributions of the three SSL objectives are introduced without much explanation. Adding a brief description of each SSL objective in the main text, e.g., on the idea behind each approach, would be helpful.\n\n2) In Section 2, the definitions of $\\Theta$ and $\\theta$ are unclear. Such unclearness may be inevitable to maintain the generality, but more intuition on what it is for each case would be useful.\n\n3) The proof of Theorem 1, at least a sketch of it, should be included in the main text, as it is one of the main claims of the paper.\n\nBelow are minor things.\n\n- Please define the notation $\\{ x_i \\}_i^n$.\n- Every now and then the reference numbers are wrong. E.g., \"Eq. (21)\" in Page 7, \"Definition 12\" (and 13, 14) in Page 9, ... Maybe there are others too.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uprEyDRD3Z", "reviewer": "Reviewer_mBoX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Combining SSL and generative models is a problem of significant importance, and this paper builds connections from theoretical perspectives.\n\n2. Experiments on toy data and simple yet classic computer vision datasets demonstrate that GEDI works as expected.\n\nWeakness:\n\n1. Missing related work. The paper mainly discusses related work in SSL, but little on methods that combine SSL and generative models. A missed key reference is [1].\n\n2. Missing baselines. Similar to W.1, the authors only compare to SSL baselines, but no other methods that combine SSL and generative models. Additionally, classification accuracy (KNN, linear probing) is conceived standard in benchmarking SSL methods. Using only NMI is valid, but it will be better if the accuracies can also be presented.\n\n3. Writing conciseness. It is well-known that many versions of SSL objectives are similar but have subtle differences. So the reviewer doubts the necessity of re-introducing many definitions for different kinds of SSL models. Since the paper focuses specifically on SSL clustering and energy-based models, it might be sufficient and more focused if the discussion is kept to the key message and key contribution of the paper.\n\n\n[1] Tong, S., Dai, X., Chen, Y., Li, M., Li, Z., Yi, B., LeCun, Y. and Ma, Y., 2022. Unsupervised learning of structured representations via closed-loop transcription. arXiv preprint arXiv:2210.16782.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper attempts to bridge the gap between SSL models and generative models in a unified framework, and demonstrate an instantiation with SSL clustering and energy-based models. The proposed method, GEDI, is tested on two toy data distributions and three simple yet classic computer vision datasets. The results were evaluated with normalized mutual information (NMI) and Frechet Inception Distance (FID) and demonstrate that the model can generate and also perform the discriminative downstream task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Discussion on the missed key reference.\n\nIn the experiments, include baseline(s) that combine SSL and generative modeling if possible.\n\nAdd classification accuracy as part of the SSL evaluation.", "broader_impact_concerns": "The reviewer sees no immediate broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)