T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning
Authors: Julie Mordacq
David Loiseaux
Vicky Kalogeiton
Steve Oudot
Venue: NeurIPS 2025 spotlight ()
Keywords: self-supervised learning
minimum spanning tree
dimensional collapse
dimension estimation
topological data analysis
Primary area: general_machine_learning
TL;DR: We introduce a minimum-spanning-tree-based regularization for self-supervised learning that provably improves representation uniformity and mitigates dimensional collapse.

Abstract: Self-supervised learning (SSL) has emerged as a powerful paradigm for learning representations without labeled data, often by enforcing invariance to input transformations such as rotations or blurring.
Recent studies have highlighted two pivotal properties for effective representations: (i) avoiding dimensional collapse-where the learned features occupy only a low-dimensional subspace, and (ii) enhancing uniformity of the induced distribution.
In this work, we introduce T-REGS, a simple regularization framework for SSL based on the length of the Minimum Spanning Tree (MST) over the learned representation.
We provide theoretical analysis demonstrating that T-REGS simultaneously mitigates dimensional collapse and promotes distribution uniformity on arbitrary compact Riemannian manifolds.
Several experiments on synthetic data and on classical SSL benchmarks validate the effectiveness of our approach at enhancing representation quality.

T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning
Julie Mordacq
David Loiseaux
Vicky Kalogeiton
Steve Oudot
I Inria Saclay
LIX, CNRS, École Polytechnique, IP Paris
Abstract
Self-supervised learning (SSL) has emerged as powerful paradigm for learning representations without labeled data, often by enforcing invariance input transfor- mations such as rotations or blurring. Recent studies have highlighted two pivotal properties for effective representations: (i) avoiding dimensional collapse- -heree the learned features occupy only a low-dimensional subspace, and (ii) enhancing uniformity of the induced distribution. In this work, we introduce T-REGS, simple regularization framework for SSL based on the length of the Minimum Spanning Tree (MST) over the learned representation. We provide theoretical anal- ysis demonstrating that T-REGS simultancously mitigates dimensional collapse and promotes distribution uniformity arbitrary compact Riemannian manifolds. Several experiments on synthetic data and on classical SSL benchmarks validate the effectiveness of our approach enhancing representation quality. Code is available here
Introduction
Self-supervised learning (SSL) has emerged as a powerful paradigm for learning meaningful data representations without relying human annotations. Recent advances, particularly in visual domains [4] 3T I7 [53] 660, have demonstrated that self-supervised representations can rival or surpass those learned through supervised methods. A dominant approach in this field is foint embedding self- self-supervised learning (JE-SSL) [12 4 [56,, where two networks are trained to produce similar embeddings for different views of the same image (see Figure [ The fundamental challenge in JE-SSL is to prevent representation collapse, where networks output identical and non-informative vectors regardless of the input. To address this challenge, researchers have developed various strategies. Contrastive approaches [12, 30) encourage embeddings of different views of the same image be similar while pushing away embeddings of different images. Non-contrastive methods bypass the need of negative pairs, often employing asymmetric architectures 13
28
10 or enforcing
decorrelation among embeddings through redundancy reduction 65
63
Recent studies have identified a more subtle form of collapse known as dimensional collapse 29 41 This phenomenon occurs when the embeddings span only lower-dimensional subspace of the representation space, leading to high feature correlations and reduced representational diversity. Such collapse can significantly impair the model's 's ability capture the full complexity of the data, limiting performance on downstream tasks (26). Another crucial aspect of representation quality is uniformity, which measures how evenly the embeddings are distributed across the representation space. It ensures that the learned representations preserve the maximum amount of information from the input data and avoid clustering in specific regions of the space. This property is fundamental because it helps maintain the discriminative power of the representations, and allows for better generalization downstream tasks (33 52 24
33 33
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Transformed
Representations
Embeddings
Uniformly distributed aligned embeddings
images
II
M
S
S
Images
X
Z
10.0
(0.0)
V-T
X
Z'
MST 2
LASE ( (2 enforces view-invariance
L2(2), (Z), (2) enforce sphere constraint
(2) increase MST(Z) and 22) lengths
Figure 1: Overview of T-REGS. (Left) Two augmented views X' are encoded by fo and projected by he into emheddings Z, Z' Training jointly: (i) minimizes the Mean Squared Error, LMSE (Z, Z'), enforce view invariance (or alternatively the objective function of a given SSL. method, Lss Z'), when used as an auxiliary term); (ii) maximizes the monimum-spanning-tee length on each branch, Cx( ()) and Lx(Z'), repelling adge-connected points in MST(Z) and MST(Z"); and (iii) applies sphere constraints (Z( and (Z (Right) As result, T-REGS induces uniformly distributed embeddings without dimensional collapse.
While existing methods have made progress in mitigating dimensional collapse and enforcing uniformity, they present limitations: contrastive methods are sensitive the number of negative samples [26 4). and require large batch sizes, which can be computationally expensive; redundancy reduction methods, which enforce the covariance matrix to be close the identity matrix, only leverage the second moment of the data distribution and are blind to, e.g., concentration points of the density, which can prevent convergence the uniform density (Figure 6 and asymmetric methods lack theoretical grounding to explain how the asymmetric network helps prevent collapse [63].
Given these limitations, Fang et al. (3 suggested rethinking the notion of good SSL regularization, and proposed an Optimal Transport-based metric that satisfies a set of four principled properties (instance permutation, instance cloning, feature cloning, and feature baby constraints) that prevent dimensional collapse and promote sample uniformity. Their approach has its drawbacks: the optimal transport distances are costly to compute in general, and the proposed closed formula for accelerating the computation holds only on the sphere and requires square roots over SVD computations, which may lead to numerical instabilities.
To address these limitations, we propose T-REG, a novel regularization approach that is conceptually simple, easy to implement, and computationally efficient. T-REG naturally satisfies the four principled properties of Fang et al. [23] (Appendix G and provably prevents dimensional collapse while
promoting sample uniformity (Figure 2ap. These properties make T-REG suitable for joint-embedding self-supervised learning, where can be applied independently to each branch, yielding T-REGS
(see Figure T.
The central idea of T-REG is to maximize the length of the minimum spanning tree (MST) of the samples in the embedding space. It has strong theoretical connections to the line of work on statistical dimension estimation via entropy maximization [57] (Section 4.1). D. More explicitly, given a point cloud Z in Euclidean space, spanning tree (ST) of Z is an undirected graph G = (V, E) with vertex set V = Z and edge set E C x V such that is connected without cycle. We define the length of
G
B(G) = Σ
||
22112- 2
(2,2))E
minimum spanning tree of E. denoted by MST(Z). is an ST of Z that minimizes length E; is unique under a genericity condition Z. Since the length of MST(2) (Z) scales under rescaling of Z, maximizing it alone leads the points to diverge (see Figure 2b). To prevent trivial scaling, T-REG constrains embeddings to compact manifold, encouraging full use of the representation dimension and a uniform distribution.
Our main contributions can be summarized as follows:
i) We introduce T-REG (Equation (6). regularization technique that maximizes the length of the minimum spanning tree (MST) while constraining embeddings to lie on a hypersphere (Section[4) 4).

---

T-REG
Initial Point Cloud
Initial Point Cloud
TREG
1.0
2.0
10.0
1.0
2.0
-1.0
0.0
-2,0
0.0
0.0
0.0
200
400
600
800
1000
1.0
2.0
Epoćas
(a) Optimization w/ T-REG
(b) Optimization w/ Le
(c) Losses
3-d point cloud optimization
0.00
T-REG
30000
Initial T-REG
0.25
0..0
20000
00.75
10000
-1.00
1.25
0.00
0.25
0.50
0.75
1.00
-1.50
0
1000
2000
3000
Cosine Similarity
Epochs
(d) Optimization w/ T-REG
(e) Loss
256-d point cloud optimization
Figure 2: Illustration of T-REG with synthetic data. (a-c) 3-d point cloud analysis: (a) T-REG successfully spreads points uniformly the sphere by combining MST length maximization and sphere constraint, (b) using only MST length maximization leads excessive dilation, (c) stable convergence of T-REG whereas LE alone fails to converge. (d-e) Higher-dimensional analysis (256-d): (d) T-RFG enforces effective convergence to 255-d regular simplex (Theorem 4.1) (e) stable optimization behavior of T-REG.
ii) We show both theoretically and empirically that T-REG naturally prevents dimensional collapse while enforcing sample uniformity (Sections 4.1 and 4.2
iii) We apply T-REG to SSL either as standalone regularization, combining directly with view- invariance, or as an auxiliary loss to existing methods, yielding the T-REGS framework- whose effectiveness evaluated through experiments on standard JE-SSL benchmarks (Section 5).
2
Related Work
Our work builds upon recent advances in Joint Boind-Embedding Supervisedd Learning (JE-SSL), which can be broadly categorized into two main approaches: contrastive and non-contrastive methods [26].
(i) Contrastive methody 32
30 14, 9 are commonly based on the InfoNCE loss [46] These
methods encourage the embeddings augmented views of the same image to be similar, while ensuring that embeddings from different images remain distinct. Contrastive pairs either be sampled from a memory bank, as in MoCo 30 14), or generated within the current batch, as in SimCLR [2]. Furthermore, clustering -based methods 8 9 27] can also be seen contrastive methods between prototypes, or clusters, instead of samples. SWAV [9] for instance, learns online clusters using the Sinkhorn- Knopp transform. However, despite their effectiveness, both approaches require numerous negative comparisons to work well, which can lead to high memory consumption. This limitation has spurred the exploration of alternative methods.
(ii) Non-contrastive methods bypass the reliance on explicit negative samples. Distillation-bused methods incorporate architectural strategies inspired by knowledge distillation to avoid representation

---

collapse, such as an additional predictor [13], self distillltlaoon 10] or moving average branch as in
attempt to produce embedding variables that are decorrelated from each other, thus avoiding collapse. These methods can be broadly categorized into two groups: those that enforce soft whitening through regularization and those that perform hard whitening through explicit transformations.
BYOL (28). Meanwhile, redundancy reduction methods [65 20 68
63
56
62]
BarlowTTwnn [65] and
VICReg [4] regularize the off-diagonal terms of the covariance matrix of the embedding to have a covariance matrix that is close to the identity. W-MSE [20] transforms embeddings into the eigenspace of their covariance matrix (batch whitening) and enforces decorrelation among the resulting vectors. 68 combines both batch and channel whitening techniques. Building upon these methods, INTL 63 proposed to modulate the embedding spectrum and explore functions beyond whitening to prevent dimensional collapse.
CW-REG (62) introduces channel whitening, while Zero-CL
In this paper, we introduce a self supervised learning (SSL.) approach that uses novel regularization criterion: the maximization of the embeddings' minimum spanning tree (MST) length. Our approach aligns with the framework proposed by Fang et al. [23], satisfying the same four principled properties: instance permutation, instance cloning. cloning, and feature baby constraints.
3 MST and dimension estimation
Steele [57 studies the total length of a minimal spanning tree (MST) for random subsets of Euclidean spaces, Let X,, be an i.i.d s-sample drawn from a probability measure Px with compact support
on _. For d 2 2, Theorem of (57) controls the growth rate of the length of MST(X,,) as follows:
E' (M)T(X,)) ~ On 1)/d almost surely, as n + 0o,
(2)
where ~ denotes asymptotic convergence, and where C is a constant depending only on Px and
This asymptotic rate makes it possible to derive several estimators of the intrinsic dimension of the support of a measure of its samples [53 2 [5]. Among these estimators, the following one theoretically coincides with the usual dimension in non-singular cases, and it empirically coincides with the real-valued Hausdorff dimension (21] in classical manifold examples or even fractal examples, such the Cantor set or the Sierpiński triangle.
Definition 3.1. Given a bounded metric space M, the MST dimension of M, denoted by dimMsT (M), is the infimal exponent d E N such that E (MST(X)) is uniformly bounded for all finite subsets X C M:
dimmsT (M) == inf(d 3C such that E (MXT(X))/XX1++ S C. for every finite subset X of M).
Persistent Homology Dimension. The MST also appears in Topological Data Analysis (TDA)
where it relates to the total persistence in degree 0 of the Rips filtration [47] Moreover, Persistent
Homology (PH) has been used to define family of fractal dimensions [1. dimpu(M) for each homological degree i 2 0. particular, for = O this coincides with the MST-based dimension, i.e., dimpH (M) = dimMsr (M)) The PH dimension can be derived from entropy computations and has already been used in several dimension-estimation applications [38 19 In this work, we directly leverage the connection the entropy obtain uniformity properties on compact Riemannian manifolds (see Section 4.1.2..
MST optimization. TDA further provides mathematical framework for optimizing the length of MST(X) with respect to the point positions of X [1]40. Within this framework, E (MST(X) is differentiable almost everywhere, with derivatives given by the following simple formula:
X V&E (MS((())
Σ
2
Σ (x.=) edge of MST(X)
| 1x~2 (x 2). (3)
(2.3) edge of MST(XX)
Furthermore, under standard assumptions on the learning rate, stochastic gradient descent is guaran- teed to converge almost surely to critical points of the functional. In particular, Equation (3) shows
that each pair of points forming an edge in the MST exerts a repulsive force on the other during optimization.
Independent and identically distributed.

---

MST computation. Given finite point set X C R several classic sequential procedures exist to compute MST(X), notably Kruskal's, Prim's, and Boruvka's algorithms, which all have least quadratic running time in the size of X. However, fast GPU-based parallelized implementations exist, for instance [22]. which unifies Kruskal's s and Boruvka's algorithms and incorporates optimizations such path compression and edge-centric operations.
4 T-REG: Minimum Spanning Tree based Regularization
Our regularization T-REG has two terms: a length of the minimum spanning tree, and a soft sphere-constraing Ls that increases with the distance to fixed sphere S. These two terms combined force the embeddings to lie on S (or close to it), while spreading them out along S.
loss LE that decreases with the
Formally, given {21.... 2n} S R. the MST length maximization loss is defined as:
CE(Z) - (MST(2)),
(4)
where E' (M)T(Z)) denotes the length of the MST of Z. The soft sphere-constraint is given by:
(s(Z)= = 1 Σ
(1 -
1)a.
It penalizes points that move away from the unit sphere. Maximizing the MST length alone would cause the points to diverge infinity; the sphere constraint prevents this by keeping the embeddings within fixed region around S (Figures 2b and 2c). The overall T-REG loss combines these two terms:
LEEEG 7 LE + (z)),
where 7 and À are hyperparameters controlling the trade-off between spreading out the embeddings and maintaining them on the sphere.
The remainder of the section provides theoretical analysis (Section 4.1 and empirical evaluation (Section 4.2 of T-REG.
4.1
Theoretical analysis
4.1.1
Behavior on small samples
We begin by considering the case where n S d + 1. It particularly relevant since, in SSL, batch sizes are often smaller than or comparable to the ambient dimension. In order to account for the effect of the soft sphere constraint, we assume the points of X lie inside some fixed closed Euclidean d-ball B of radius r centered at the origin (see below for the explanation).
Theorem 4.1. Under the above conditions, the maximum of E (X)T(X)) over the point sets X C B of fixed cardinality is attained when the points of X lie the sphere $ = êB, at the vertices of regular (n 1) simplex that has as its smallest circumscribing sphere.
Recall that k-simplex is the convex hull of set of k + points that are affinely independent in Idd which is possible only for & I d. The simplex is regular if all its edges have the same length, have the following
ie., all the pairwise distances between its vertices equal. In such relation between its edge length a and the radius r of its smallest circumscribing sphere:
2(k 1)
Theorem 4.1 explains the behavior of T-REG as follows: first, minimizing the term LE in Equation (6) expands the point cloud until the sphere constraint term Ls becomes the dominating term (which happens eventually since Ls grows quadratically with the scaling factor, versus linearly for L2); at that stage, the points stop expanding and start spreading themselves out uniformly along the sphere of directions. The amount of expansion before spreading is prescribed by the strength of the sphere constraint term versus the term in the loss, which driven by the ratio between their respective mixing parameters X and 7.
5

---

The proof of Theorem 4.1 relies on the following two ingredients: a standard result in convex geometry (Proposition 4. and technical lemma- proved in Appendix A relating the length of the MST to the sum of pairwise distances (Lemma
Under the conditions of Theorem 4.1 and assuning = d + 1, the sum of pairwise distances Listesen || - 23112 2 is maximal when the points of X lie the bounding sphere S, at the vertices of a regular d-simplex. Lemma 4.3, For any points 2
Proposition 4.2 (Eq. (14.25) in Apostol and Mnatsakanian
Zn E Rdd
E' (MST ((Z), 2n (n) S
Σ Igi<jSn
1 23112
Proof of Theorem 4./ We prove the result in the case d + The case < d + 1 is the same modulo some extra technicalities and can be found in Appendix A
Let 21.
ES lie at the vertices of regular d-simplex. Then, for any points 21,...,,2n E B:
E (MST (!z1,..., )))
Lemmale
2 Eisieisa | 22i12
Propocition
2 Eisisisa
|| - -
Bq N
2 2
2(d+1]
(n 1) rv
2(++
d
E (MST (z]),..,,;)))
4.1.2 Asymptotic behavior on large samples
We now consider the case where >> d + 1, focusing specifically on the asymptotic behavior as n - 00. We analyze the constant in Equation (2.. which can be made independent of the density of the sampling X. This, in particular, allows us to show that uniform and densities are asymptotically optimal for (MMS(.)). We fix a compact Riemannian d- d-manifold, M, equipped with the d-dimensional Hausdorff measure μ.
maximizing
Theorem 4.4 (15) Corollary 5)). Let X, be an iid n-sample of a probability measure on M with density fx W.E. μ. Then, there exists constant C' independent of fx and of M such that:
d-1
(MST((X)) n-+oc C'
du almost surely.
(8)
As pointed out by Costa and Hero [15. the limit Equation g related to the intrinsic Rényi d-1 entropy:
1 d-I log
ب (1)
dp,
which is known to converge to the Shannon entropy as d-1 achieves its maximum at the uniform distribution on compact sets (48). This result can be shown by directly studying the map Φ: [+ fe dj,, which shows that an optimal density function fx maximizes the dimensionality of the sampling X,, Given compact set K C M, we consider the space Dx of positive, continuous probability densities ff on K.
+ 1 66. The Shannon entropy, in turn,
Proposition 4.5. For any < p < 1 and any compuct set C M, the map Φ admits a unique maximum at the uniform distribution UR on K. Furthermore, we have ((UA) < ((Un) (U for all sets A, C M such that p{A) < pBB).
Proof. The map is strictly concave, as the composition of the strictly concave function x? with the linear map æ + x du. Since density functions integrate to 1 over K, maximizing ø corresponds to optimization problem under constraint, which can be solved using the Lagrangian:
Cx(f) = { fo du (Jr fdu (du-1) with differential d (Cx), (h) / (pfo-1 A) h du.

---

Then, d(Cx) = O is equivalent to pf À being 0 almost everywhere, i.c. f being equal to
(a constant determined by f dμ = almost everywhere, i.e. f being the density of the uniform measure on K.
Now, recalling that p 1, we have:
(UK) - / dUk]
J m(K) du =
u((K) (u(K))p dUx - m(K)
du
dy
which proves the second part of the statement.
A direct consequence of Proposition 4.5 and the fact that DM is dense in the set of probability densities of (P(M) is the following corollary.
Corollary 4.6. Let D be the of probability densities over M, and p (0, 1). Then, the map f D f f du reaches its maximun at the density f dUs, dp
4.2 Empirical evaluation
We conduct empirical study synthetic data to validate T-REG's ability to prevent dimen- sional collapse and promote sample uniformity.
-025
-W
-Bv
Preventing dimensional collapse. Follow-
ing Fang al. 233,, assess T-REG's effective- ness against dimensional collapse by measuring how sensitive its loss LE is simulated collapse. Specifically, we generate 10, 000 data points in dimension 1024 from isotropic Gaussian dis- tribution, then zero out a fraction of their coor- dinates to control the collapse level. As shown in Figure 3. the sensitivity of the T-REG loss 17 similar that of the W7 loss from Fang et al. 233. indicating that T-REG effectively pe- nalizes dimensional collapse.
0.75
210
-1.00
10
-1.25
20
10
60
80
100
Collapse lovel, 79 (56)
Figure 3: Sensitivity to Dimensional Collapse. The metrics -Wa and LL jointly decrease the collapse level (1)) increases.
Promoting sample uniformity. We apply T-REG alone to optimize given point cloud and analyze its behavior in both -iw-dimeniional and high-dimensional scenarios (Figure 2]. For this, we use two different input point clouds: (i) degenerate set of 256 points on a 1-d curve (corresponding to the orange dots in Figures 2a and 2b): (ii) a set of 256 points in 256 initially concentrated around specific point on the unit sphere, where each point 2; is sampled I; + εú, with e; drawn uniformly from a ball radius 0.001.
As illustrated in Figure 2a optimization with T-REG successfully transforms the initial point cloud in 3-d space into a uniformly distributed point cloud on the sphere, as per Corollary [46 This is achieved through the combination of MST length maximization and sphere constraint. The sphere constraint crucial here: when optimizing only the MST length, Lz, (see Figure 2b the optimization fails to converge.
In high dimensions (Figure 2db. we analyze the distribution of cosine similarities between the embeddings. The initial distribution shows sharp peak near 1, indicating highly correlated samples on the sphere. After optimization with T-REG, the distribution becomes almost Dirac slightly below 0, indicating that the configuration of the points is close to that of the vertices of the regular simplex, per Theorem
T-REGS: T-REG for Self-supervised learning
T-REGS extends T-REG to Joint-Embedding Self-Supervised Learning. For an input image i, two transformations 1, are sampled from a distribution T produce two augmented views = ((i) and z' = t'()). These transformations typically random crops and color distortions. We compute
7

---

91.3
68.5
Zero-CL 68
+
91.3
68.4
W2
91.4
68.5
90.7
60.3
MoCo v2 14
91.0
61.2
91.4
89.3
63.7
BYOL 28
+ C,
90.1
+ 1
90.1
65.2
Ersnnss
91.2
68.2
Barlow Twins
91.4
W
91.4
68.5
Ersnes
CuSE
Cassos
91.3
67.4
Table I: Comparison with W22 regularization (23) on CIFAR-10/100 The table is inherited from Fang et al. 233. and follow the same protocol: ResNet-18 models are pre- pre-trained for 500 epochs on 11FAR-100/10, with a batch size of 256, followed by linear probing. We report Top-1 accuracy (%). Boldface indicates best performance.
embeddings h (x)( and z' (foff(x)) using backbone fo and projector has T-REGS acts
as regularization applied separately to the embedding batches Z = [2|, ..... 2n and Z' = [z], . 2 Specifically, embeddings from each view batch, Z and Z', are treated points in high-dimensional space, and Kruskal's algorithm (38) is used to construct two Minimum Spanning Tree (MST), one for each view batch. These MSTs yield two T-REG regularization terms, which are combined into the T-REGS objective as follows:
CTREGS (Z, 2') = LB(Z) + ALs(Z) + LE((Z) ALs(Z')
(10)
Cemar (2)
Caumxl (2)
where À control the contribution of each term.
In practice, T-REGS can be used as (i) standalone regularization, combined directly with an invariance term such as the Mean Squared Error: L(Z, Z') -3 LMss Z') + LE-REGS (Z, Z'), where LMSE(22 [Z, Z") existing SSL methods: L(7. 7") = 3 Lssi. [z, Z') + LTrEGS Z, Z'), where Lssi (Z, Z') denotes the objective function of given SSL method, and /3 is mixing parameter. overview of T-REGS
Σ, || 22 and /3 is mixing parameter; or (ii) auxiliary loss to
presented in [
The remainder of the section provides evaluations of T-REGS on standard SSL benchmarks (Sec- 5..n5.1 and on multi-modal application (Section 5.2): : as well as loss coefficients and computational
analyses (Section 5.3) Implementation details and further analyses are Appendices C and E
5.1 Evaluation standard SSL benchmark
We evaluate the representations obtained after training with T-REGS, either directly combined with view invariance integrated with existing methods (i.e., BYOL, and Barlow Twins) 100
CIFAR-10/100 (37, ImageNet- 100 [59], and ImageNet [18]. Our implementation is based on solo-learn [16), and use torchph 7 for the MST computations. For T-REGS standalone regularizer, we use 10, 7 = 0.2, X 8e 4.
Evaluation on CIFAR- 10/100. We first focus comparisons with Fang et al. 26 (Wa-regularized methods), following the same protocol on CIFAR- [37] with ResNet-18, As shown Table [ T-REGS demonstrates strong standalone performance, achieving results within 0.1% of the best Wz-regularized approach on CIFAR- 10. Additionally, using T-REGS as an auxiliary loss consistently improves performance over the respective baselines, and over variants that use C,, or W2 additional regularization terms.
Evaluation on ImageNeet1000/11 To assess the scalability of T-REGS, we evaluate our model on ImageNet- 100 and ImageNet- using ResNet-18 and ResNet-50, respectively, following the standard linear evaluation protocol on ImageNet and comparing with the state of the art. We report Top-1 accuracy. As shown Table T-REGS is competitive with methods that use the same number of views (c.g., INTL), and improves existing methods when used as auxiliary loss.

---

Imagenet-100
ImageNet-1k [18]
Method
Top
Batch Size
Top
SwAV 9
74.3
4096
66.5
FroSSI. 56
79.8
SSOLE
82.5
256
73.9
SimC
71.0
4799
66.3
McCo v2
79.3
67.4
SimSiam
78.7
256
68.1
W-MSE
69.
512
65.
Zero-CL
1024
VICReg
79.4
1024
68.3
RGP
INIL
63
81.7
512
69.5
80.3
1024
66.5
BYOI.
28
2048
Barlow Twins
LTrecs
80.9
2048
67.8
CaEE
Direos
80.3
512
68.8
Table 2: Linear Evaluation on ImageNNet110011k, We report Top-1 accuracy (%). The top-4 methods are boldfaced. For ImageNet-100. ResNet- 18 pre-trained for 400 epochs using batch size of 256; for ImageNet- Ik, ResNet-50 pre-trained 100 epochs. The table is mostly inherited from Weng et [63
5.2 Evaluation on Multi-modal application: image-text retrieval
Flickr30k
(49
MS-COCO
43
[
Method
R@D
R@S
R@I
R@5
R@1
R@S
R@D
R@5
Zero-Shot
71.1
90.4
68.5
88.9
31.9
56.9
28.5
53.1
Finetune
81.2
80.7
95.8
36.7
63.6
36.9
63.9
[41]
71.8
90.0
68.5
88.9
31.9
56.9
28.7
53.0
i-Mix [39
72.3
91.7
69.0
91.1
34.0
63.0
34.6
62.2
Un-Mix 54]
78.5
95.4
74.1
91.8
38.8
66.2
33.4
61.0
m MMM 45
82.3
95.9
82.7
96.0
41.0
68.3
39.9
67.9
CCLP LIREGS
83.2
96.0
80.8
96.4
41.6
68.7
41.5
68.7
Table Cross-Modal Retrieval after finetuning CLIP. Image-to-text (i++1 1) and text-to-image + i) retrieval results (top 1/5 Recall: R @ 1, R@5). The table is mostly inherited from Oh et al. [45]
Boldface indicates the best performance.
T-REGS can also be applied when branches differ in architecture and data modalities, as it regularizes each branch independently. Accordingly, we demonstrate its capabilities in a joint-embedding multi-modal setting.
Pre-trained multi-modal models, such as CLIP [51], provide broadly transferable embeddings. However, several works have shown that CLIP preserves distinct subspaces for text and image the Prior analyses 45 64 relate this gap to low embedding uniformity: notably, CLIP's embedding space often remains non-uniform even after fine-tuning, which hinder transferability. Given that T-REGS improves embedding uniformity when used as an auxiliary loss, we evaluate its impact on CLIP fine-tuning. We fine-tune CLIP using T-REGS as auxiliary regularizer; more precisely, LT.rEGS is applied independently to the image and text branches and combined with the standard LCLPP objective (5T) encourage more robust and uniformly distributed representations. We follow the protocol of m -Mix [45]. We report R@ and R@5 5 for image-to-text and sot-to-image retrieval on Flickr30k and MS-COCO in Table which shows that T-REGS improves performance over prior methods.
modality gap 42 45 36].
5.3 Analysis
Loss coefficients. We determine the final coefficients for T-REGS as a standalone regularizer on ImageNet-Ik as follows (the same approach was applied when combining T-REGS with existing methods). Initial experiments revealed that maintaining 2 7 2 À was essential to prevent representation collapse. To efficiently explore the parameter space while managing computational

---

Scaling
0
Top-1
collapse
collapse
Muthod
Complesity
ramge
Wall-Bock time
collapse
SimCLK
12
OL
D)
12048-4096
1256-1022 1024
0.0
10
0.5
5e-2
20
10
25.7
VICRog
[1024-400
10
0.2
2e-2
50
10
45.4
Cuss Liancs
D logB))
[512-1024]
[512 2043]
0.20 a 0.001
10
2.5e-3
20
10
0.2
le:3
30
200
65.3
Table 5: Complexity and computational cost. Compari-
10
0.5
2e:3
20
250
64.9
son between different methods is performed, with training on ImageNet- Ik distributed across Tesla H100 GPUs. The wall- clock time (sec/step) averaged over 500 steps. B, D ranges +Treoss top-1 accuracy (%) ImageNet-Ik are reported from Bardes et Garrido et al. 25 with online evaluation protocol over 50 epochs. Boldface indicates best performance.
10
30
Table 4: Impact of coefficients. ΣASE
costs, fixed B (the largest coefficient) and systematically varied the ratios 2 and 3 using 50- epoch online probing [5]. As shown in Table both LE and Ls contribute performance, with Ls requiring a smaller weight. This suggests that the actual radius of the sphere is not critical, thereby validating our choice of a soft sphere constraint instead of a hard one.
Computational cost. We evaluate the computational cost of T-REGS. The MST's are computed with Kruskal's ' algorithm [38]. whose worst-case time is (D-
with B the batch size and D the embedding dimension. Although Kruskal's main loop is sequential, preprocessing (computing the distance matrix and sorting its entries) dominates in practice and can be efficiently parallelized on GPUs (as in torchph [7]. used in our implementation), Empirically, T-REGS matches the per-step wall-clock of VICReg and SimCLR (averaged over 500 steps during training on ImageNet-Ik with B 512, D 1024; see Table 5)
6 Conclusion
We introduced T-REG, a regularization approach that prevents dimensional collapse and promotes sample uniformity. Our method maximizes the length of the minimum spanning tree (MST), coupled with a sphere constraint. Our analysis connects MST optimization to entropy maximization and uniformity compact manifolds, providing theoretical guarantees corroborated by empirical results. We extend T-REG to Self-supervised learning, yielding T-REGS. On CIFAR- 10/100 and ImageNet- 100/1k, T-REGS is competitive with Wa-regularized and state-of-the-ar methods, both standalone regularizer and as an auxiliary term, underscoring its effectivvness.
Acknowledgments
This work was partially supported by Inria Action Exploratoire PREMEDIT (Precision Medicine using Topology). HilParis grant and ANR/France 2030 program. We were granted access to the HPC resources of IDRIS under the allocations 2024-AD01 1014747R1 and ADD0A made by GENCI We would like to thank Robin Courant for his helpful feedback.
10

---

References
[1] H. Adams, M. Aminian, E.. Farnell, M. Kirby, J. Mirth, Neville, Peterson, and C. Shon- kwiler. A fractal dimension for measures via persistent homology. In Topological Data Analysis: The Abel Symposium 2018, pages 1-31. Springer, 2020.
[2] H. Adams, M. Aminian, Farnell, M. Kirby, J. Mirth, R. Neville, Peterson, and C. Shon- kwiler. A Fractal Dimension for Measures via Persistent Homology. In N. A. Baas, G. E. Carlsson, G. Quick, M. Szymik, and M. Thaule, editors, Topological Data Analysis. Springer International Publishing, 2020.
[3] T. M. Apostol and M. A. Mnatsakanian. New horizons in geometry, volume 47, American Mathematical Soc., 2017.
[4] A. Bardes, J. Ponce, and Y. Lecun. Vicreg: Varfanc-iivaaranee-covvaannn regularization for self-supervised learning. In ICLR, 2022.
[5] T. Birdal, A. Lou, L. J. Guibas, and U. Simsekli. Intrinsic dimension, persistent homology and generalization in neural networks. Advances in neural information processing systems, 34: 6776-6789, 2021.
[6] P. A. Bromiley, N. A. Thacker, and E. Bouhova-Thacker. Shannon entropy, Renyi entropy, and information. Statistics and Inf. Series (2004-004), 9(2004):--8, 2004.
[7] M. D. C. Hofer, R. Kwitt and M. Niethammer. Connectivtt-oppimized representation learning via persistent homology. In Proc. ICML, 2019.
[8] M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
[9] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020.
[10] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In CVPR, 2021.
[11) M. Carriere, F. Chazal, M. Glisse, Y. Ike, H. Kannan, and Y. Umeda. Optimizing persistent homology-based functions. In Proc. ICML, 2021.
[12] Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In Proc. ICML, 2020.
[13] X. Chen and K. He. Exploring simple siamese representation learning. In CVPR, 2021.
[14] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv, 2020.
[15] A. Costa and A. O. Hero. Determining Intrinsic Dimension and Entropy of High-Dimensional Shape Spaces. In H. Krim and A. Yezzi, editors, Statistics and Analysis of Shapes. Birkhäuser, 2006.
[16] V. G. T. Da Costa, E. Fini, M. Nabi, N. Sebe, and E. Ricci. solo-learn: A library of self- supervised methods for visual representation learning. J. Machine Learning Research, 2022.
[17] T. Darcet, F. Baldassarre, M. Oquab, J. Mairal, and P. Bojanowski. Cluster and predict latent patches for improved masked image modeling. arXiv, 2025.
[18] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[19] B. Dupuis, G. Deligiannidis, and U. Simsckli. Generalization bounds using data-dependent fractal dimensions. In International Conference on Machine Learning, pages 8922-8968. PMLR, 2023.
[20] A. Ermolov, Siarohin, Sangineto, and N. Sebe, Whitening for self-supervised representa- tion learning, In Proc. ICML, 2021.

---

[21] K. Falconer. Fractal Geometry: Mathematical Foundations and Applications. John Wiley & Sons, 2013.
[22] A. Fallin, A. Gonzalez, J. Seo, and M. Burtscher. A High-Performance MST Implementation for GPUs. In Proc. Int. Conf. High Perform. Comput. Netw. Storage Anal. ACM, 2023.
[23] X. Fang, J. Li, Q. Sun, and B. Wang. Rethinking the uniformity metric in self-supervised learning. In ICLR, 2024.
[24] T. Gao, X. Yao, and D. Chen, SimCSE: Simple contrastive learning of sentence emheddings. In EMNLP, 2021.
[25] Q. Garrido, R. Balestriero, L. Najman, and Y. Lecun. Rankme: Assessing the downstream performance of pretrained self--upprvised representations by their rank. In Proc. ICML, 2023.
[26] Q. Garrido, Y. Chen, A. Bardes, L. Najman, and Y. Lecun. On the duality between contrastive and non-contrastive self-supervised learning. In ICLR, 2023.
[27] C. GE, J. Wang, Z. Tong, S. Chen, Y. Song, and P. Luo. Soft neighbors are positive supporters in contrastive visual representation learning. In ICLR, 2023.
[28] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurlPS, 2020.
[29] B. He and M. Ozay. Exploring the gap between collapsed & whitened features in self-supervised learning. In Proc. ICML,, 2022.
[30] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[31] K. He, Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[32] R. D. Hjelm, A. Fedorov, S. Y. Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2018.
K. Grewal, P. Bachman, A. Trischler, and
[33] T. Hua, W. Wang. Z. Xue, S. Ren, Y. Wang, and H. Zhao. On feature decorrelation in self- supervised learning. In CVPR, 2021.
[34] L. Huang, Q. Qiu, and G. Sapiro. Ssole: Rethinking orthogonal low-rank embedding for self-supervised learning. In ICLR, 2025.
[35] L. Jing. P. Vincent, Y. LeCun, and Y. Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
[36] J. Kim and S. Hwang. Enhanced ood detection through cross-modal alignment of multi-modas representations. Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29979-29988, 2025.
[37] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. ICLR, 2009.
[38] J. B. Kruskal. On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical society, 1956.
[39] K. Lee, Y. Zhu, K. Sohn, C.-L. Li, Shin, and H. Lee. i-mix: A domain-agnostic strategy for contrastive representation learning. In ICLR, 2021.
[40] J. Leygonie, S. Oudot, and U. Tillmann. A framework for differential calculus on persistence barcodes. Foundations of Computational Mathematics, 2022.
[41] A. C. Li, A. A. Efros, and D. Pathak. Understanding collapse in non-contrastive siamese representation learning. In ECCV, 2022.
12

---

[42] V. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. NeurIPS, 2022.
[43] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
[44] I. Misra and L. v. d. Maaten. Self-supervised learning of pretext-invarian representations. In CVPR, 2020.
[45] C. Oh, J. So, H. Byun, Y. Lim, M. Shin, J.JJ. Jeon, and K. Song. Geodesic multi-modal mixup for robust fine-tuning, NeurIPS, 2023.
[46] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 187.:7880. 2018.
[47] S. Y. Oudot. Persistence theory: from quiver representations to data analysis. American Mathematical Soc., 2017.
[48] S. Y. Park and A. K. Bera. Maximum entropy autoregressive conditional heteroskedasticity model. Journal of Econometrics, 150(2):219 2300 June 2009. ISSN 0304-4076. doi: 10.1016/j. jeconom.
[49] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting eegion-to-phrase correspondences for richer mmage-to-sentenee models. In ICCV, 2015.
[50] T. G. Project. GUDIII User and Reference Manual, GUDHI Editorial Board, 3.11.0 edition, 2025. URL htttps://///dd inria. ft/ddc//...///o/ /tt////...//////////...........t
[51] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell,
P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In Proc. ICML, 2021.
[52] N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, and H. Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In Proc. ICML, 2019.
[53] B. Schweinhart. Persistent homology and the upper box dimension. Discrete & Computational Geometry, 2021.
[54] Z. Shen, Z. Liu, Z. Liu, M. Savvides, T. Darrell, and E. Xing. Un-mix: Rethinking image mixtures for unsupervised visual representation learning. In AAAI, 2022.
[55] O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. Dinov3. arXiv, 2025.
[56] O. Skean, A. Dhakal, N. Jacobs, and L. G. Sanchez Giraldo. Frossl: Frobenius norm minimiza- tion for efficient multiview self-supervised learning. In ECCV, 2024.
[57] M. Stecle. Growth rates of euclidean minimal spanning trees with power weighted edges. The Annals of Probability, 16(4):1777-1787, 1988.
[58] C. B. Tan, I. Garria-Redonde Q. Wang. M. M. Bronstein, and A. Monod. On the limitations of fractal dimension as a measure of generalization. Advances in Neural Information Processing Systems, 37:60309-60334, 2024.
[59] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. In ECCV. Springer, 2020.
[60] S. Venkataramanan, V. Pariza, M. Salchi, L. Knobel, S. Gidaris, E. Ramzi, A. Bursuc, and Y. M. Asano. Franca: Nested matryoshka clustering for scalable visual representation learning. arXiv, 2025.
[61] Wang and Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In Proc. ICML, 2020.
13

---

[62] X. Weng, L. Huang, L. Zhao, R. Anwer, S. H. Khan, and F. Shahbaz Khan. An investigation into whitening loss for self-supervised learning. NeurIPS, 2022.
[63] X. Weng, Ni, Song, Luo, R. M. Anwer, Khan, F. Khan, and L. Huang. Modulate your spectrum in self-supervised learning. In ICLR, 2024.
[64] S. Yamaguchi, D. Feng, S. Kanai, K. Adachi, and D. Chijiwa. Post-pre-training for modality alignment in vasion-language foundation models. In CVPR, 2025.
[65] Zhontar, L.. Jing, I. Misra, Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. Proc. ICML, 2021.
[66] D. Zhang, Y. Li, and Z. Zhang. Deep metric learning with spherical embedding. NeurIPS, 2020.
[67] Zhang, Zhang, R. Vasudevan, and M. Johnson-Roberson. Hyperspherical embedding for point cloud completion. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5323-5332, 2023.
[68] S. Zhang, Zhu, J. Yan, R. Zhao, and X. Yang. Zero-cl: Instance and feature decorrelation for negative-free freee symmetric contrastive learning. In ICLR, 2021.
14

---

Appendix to
T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning
Table of Contents
Missing proofs from Section 4.1
15
Algorithm
Implementation Details standard SSL Benchmark
17
C.1
Architectural and training details.
17
Augmentations]
18
Implementation details on Image-text retrieval
18
D Compute cost
18
E
Ablation Study
19
E.1
Projector Architecture,
19
Batch size.
Normalization
20
E,4
Sensitivity the sced.
20
E.S
Study of the embeddings.
21
Empirical Experiments
21
F.1
Study of redundancy-redues reduction methods
21
2
Promoting sample uniformity
23
G
Uniformity properties
23
Example a Minimum Spanning Tree
24
A Missing proofs from Section 4.1
Proof of Theorem 4. (case n < + 1). Let 21...., 2n E B, and let HC Id be an n-dimensional affine space containing 21,...,2z,
Zn- Let By be the (n d)-dimensional Euclidean ball B n H, and Sh = S n H its bounding sphere. Finally, let TH < r be the radius of Bh (and of Sæ). Inside H, the same calculation in the case
+ shows that
2n
E (MST (f21...., 2n) < E (MST
= (n 1) r)
n n-1
for any points : " lying at the vertices of a regular (n 1)-simplex inscribed Sy. The quantity on the right-hand side of the equality bounded above by (n 1)r 7 which attained when r'H i.e., when the affine subspace H contains the origin, equivalently, when S is the smallest circumscribing sphere ,?...., "
Proof of Lemma 4.3
We fix and proceed by induction on
For we have:
E (MST ((z1))) =
2
Σ IgiKsSn
||
21112-
15

---

Assume now that the result holds for all n up to some My 2 1, and let us prove it for n = Rop 1. Let 2z) be an edge of MST (f zn3) of maximum length. Then, the graph G MST ((=1.... ) {e) has two connected components C and D, each of which is tree with less than n vertices. Up to a relabeling of the points of Z, we can assume without loss of generality that j = + 1 and that the vertices of C are the points 21....; 2 while the vertices of D are the points
We can then make the following observations:
(i) C MST
2:)) and D MST (f241)...... 2,,)). Indeed, otherwise, replacing C by MST ((=1.... }) and D by MST ((z++),.... }. and connecting them with edge would yield spanning tree of (z1,...; ]n) of strictly smaller length than G U {e), which would contradict the fact that (U = MST ((=1 (#1)...,,,])
(i) For all l S i < I, we have || - =ll2 2 |||, - 2 for otherwise the graph G U ((x, 2)} would be spanning tree of {zi..... 2n) of strictly smaller length than G U {e), again contradicting the fact that U [e} = MST ((Z,,..,,7]).
Then, i and the induction hypothesis imply:
Σ k<
2
2112 2 i E(C)),
(11)
2
Σ i<kel
1|
21l12 2 2 EDD)
(12)
Meanwhile, (i)
implies:
2 - 21l12 2 i(n - 1) ||: - 2441l22
(13)
KSK
And since e = (Zi; 1141) is an edge of MST (f=,, )n of maximum length, we have:
|| - 24-1 21 i i E(C),
(14)
||2 24-1 2 n - E(D).
(15)
Hence:
Σ || za
i(n i) || Zi+1 1l2
(2 + (n- + i(m-i-l)) 122-2441l
(16)
Fas
z 1
2+11l2 11l + 2 E(C) + E(D).
It follows:
122 21l12
Σ || 21l12 + Σ || 21122 + Σ 1| =1122
ISkeiSh
ksiel
k<lsi
ickkl
Equ
and
1
12
|| - 2 |||22 + " + B(D) + + 2 E(C)+ =rd
|| - 2 11| + E(C) + B(D)
EMST ( ((21,....
16

---

B Algorithm
Algorithm T-REGS combined with view invariance using PyTorch pseudocode
# encoder network, projection network
# 1, 7, A: coefficients of the invariance, MST length and sphere constraint losses
for x in loader: do load a batch with N samples
# two randomly augmented versions of x
x_1, x_2 augment (x), augment (x)
compute the zepresentations and the embeddings
y_1, y.2 f(x_1), f(x_2)
2_1, 2_2 h(y_1), h(y_2)
inv_lose Cran (2_1,2___ # invariance loss Length_mst_lost = (z_1) La (z_2) # MST length
sphere_lose = La (z) L (z_2) # soft sphere constraint loss total loss
loss inv_loss ? length_mst_loss sphere_loss
optimization step
loss. basswaackward() optimizer.sten s) end for
C Implementation Details on standard SSL Benchmark
Our implementation is based on solo-learn [16] which is released under the MIT License. To
compute the length minimum spanning tree, we rely on torchphhhhhe under MIT Licenses.
and and Gudhi (50 both released
Our experiments are performed on
1. ImageNet dataset 18 and a subset ImageNet-100 100 which are subject to the ImageNet terms access
2. CIFAR-10, CIFAR-100
C.1 Architectural and training details.
CIFAR-10 57
CIFAR-100 37
Imagenet-100
ImageNet [8]
Backbone
backbone
Resnet-18
Resnet-18
Resnei-1
Resnet-50
Projector
proector layers
3 layers with BN and Rel.Ul
projector hidden dimension
2048
2048
projector output dimension
1024
Pre-training
batch size
256
256
256
512
optimizer leariing
LARS hase
hatch size/256
buse,
0.4
0.4
learming rate warm-up
epochs
10 epochs
learning rate schedule
cosine decay
weight docay
1c-4
Linear evaluation
batch
256 SGD
optimizer
leamingg rate schodule
cosine decay
Table 6: Training hyperparameters.
We follow the guidance of Da Costa et [16] for selecting baseline hyperparameters and use the same seed: 5. Table lists each dataset's architectural and training details.
17

---

C.2 Augmentations
We follow the image augmentation protocol first introduced in SimCLR 12 and now commonly used in Soint-Embedding Self-Supervised Learning [4] 9 66). Two random crops from the input image sampled and resized to 32 x 32 for CIFAR-10/100 and 224 x 224 for Imagenet- 100/1k, followed by color jittering, converting to grayscale, Gaussian blurring, polarization, and horizontal flipping. Each crop is normalized in each color channel using the ImageNet mean and standard deviation pixel values. The following operations performed sequentially to produce each view:
ImageNet-Ik Data Augmentation.
Random cropping with area uniformly sampled with size ratio between 0.2 to 1.0,
followed by resizing to size 224 x 224.
Color jittering of brightness, contrast, saturation and hue, with probability 0.8.
Grayscale with probability 0.2.
Gaussian blur with probability 0.5 and kernel size 23.
Solarization with probability 0.1.
Random horizontal flip with probability 0.5.
Color normalization using ImageNet mean and standard deviation pixel values (with mean (0.485, 0.456, 0,406) and standard deviation (0.229, 0.224, 0.225).)
CIFAR-10/100 Data Augmentation.
Random cropping with an area uniformly sampled with size ratio between 0.2 to 1.0,
followed by resizing size 32 32.
Color jittering of brightness, contrast, saturation and hue, with probability 0.8.
Grayscale with probability 0.2.
Solarization with probability 0.1.
Random horizontal flip with probability 0.5.
Color normalization with mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224,
0.225).
C.3 Implementation details Image-text retrieval
Following Oh et al. [45. we fine-tune CLIP ViT-B/32 on Flickr30k and MS COCO. respectively. We train for epochs with batch size 128 using the Adam optimizer (3, = 0.9, 82 0.98, E = 1e 6). We search for the best initial learning rate from {le - 6, 3e - 6, 5e - 6, 7e - 6, 1e 5] and weight decay from {le 2e 5e 2, le 2e 1]. For T-REGS combined with LcuP, the overall
objective is: C(Z, 2 = B LEIP Z, Z') LTrEOS Z2 Z'), with 8 = 1, 3e - 3, A = 2c 5.
D Compute cost
We conducted our experiments using NVIDIA H100 and V100 GPUs. Training a single T-REGS model requires:
for ImageNet- 1k: 15 hours using 4 H100 GPUs (i.e., amounting to 60 GPU-hours per model);
for ImageNet-100: 7 hours using 1 H100 GPU;
for CIFAR- 10/100: hours using V100 GPU.
The computational cost for the entire project, including all baseline computations, experiments, hyperparameter tuning, and ablation studies, amounted to approximately 10,000 H100 GPU-hours and 5,000 V100 GPU-hours.
18

---

70
Top-1 Accuracy (%)
65
60
8192-8192-d
55
4896-4096-1
2048-d
50
256512
1024
2048 Output dimension
4096
Figure 4: Impact of the projector architecture. LAS +LTREGS top-1 accuracy (%) on the linear evaluation protocol with 100 pretraining epochs.
E Ablation Study
In this section, we conduct a comprehensive ablation experiments assess the robustness and versatility of T-REGS. These experiments cover various aspects, including projector architecture, batch sizes, and seed variability.
E.1 Projector Architecture.
An essential difference between methods lies in how the projector (he, in Figure 1 is designed [26]. To assess the impact of the projector architecture T-REGS performance (when combined with view invariance), we train models for 100 epochs on ImageNet-Ik with different projector architectures. We describe projector architectures using the notation X-Y-Z, where each number represents the dimension of linear layer in sequence. Each layer (except the last) is followed by a ReLU activation and batch normalization. The final layer has no activation, batch normalization, or bias. We evaluate three projector architectures:
2048-d: the projector used in SimCLR
8192-8192-d: the projector used in VICReg
4096-4096-d: a smaller variant of the VICReg projector
with d varying from 256 to 4096. The remaining hyperparameters are the same as reported in Ap- pendix C Our results demonstrate that the choice of projector architecture significantly impacts the model's performance, with 8192-8192-C and 4096-4096-d consistently outperforming the 2048-d variant.
We also observe that the embedding dimension d (i.e., the output dimension of the projector h&) has minimal impact performance. This is particularly noteworthy other methods, such as VICReg 4 and Barlow Twins [65]. are known be sensitive to embedding dimension, typically requiring dimensions larger than 2048 for optimal performance. Our results show that T-REGS maintains high accuracy even with small embedding dimensions, demonstrating its robustness to this hyperparameter.
E.2
Batch size.
Batch Size
128
256
512
1024
Top-1
66.3
67.2
68.7
68.0
Table Impact of batch size. LMSE +LTREGs top-1 accuracy (%) using linear evaluation after 100 pre-training epochs.
19

---

167
1c6
BYOL
MSE T-REGS
BYOL T-REGS
6-
0.8
0.6
0.4
0.2
0.0
8.00
0.60 Cosine Similarity
Casine Similarity
Figure 5: Histograms of embeddings' cosine similarities on CIFAR-10. With T-REGS as a standalone regularization (orange) becomes concentrated around zero, indicating that the embeddings are highly decorrelated and approach regular simplex configuration (Theorem 4.1)
auxiliary loss (dark orange), the distribution of pairwisc cosine similarities
Many SSL methods are known to be sensitive batch sizes, for instance contrastive methods suffer from the need of many negative samples which can translate into the need of large batch sizes 12 In Table Z we study how batch size affects T-REGS performance (when combined with view invaraince). We train models for 100 epochs ImageNet- Ik with batch sizes ranging from 128 to 1024, using the same hyperparameters as in T T-REGS maintains good performance even with small batch sizes (c.g.. 128), demonstrating its robustness different batch size configurations.
E.3 Normalization
Some popular SSL frameworks, such as SimCLR and BYOL,, employ explicit normalization of features to the unit sphere, enforcing hard constraint. Others do not implement such constraints explicitly, and instead rely on soft mechanisms- as VICReg, which incorporates variance and covariance regularization the norms) of the embeddings. We chose soft sphere constraint for the following reasons: (i) hard normalization has been shown ignore the importance of the embedding norm for gradient computation, whereas a soft constraint enables better embedding optimization [66] 67], (ii) during our initial experiment, we found that relaxing the sphere constraint from a hard one soft one provides more leeway for optimization and leads to improved results, as shown in Table S
in its loss, implicitly constraining the distribution (and to some extent,
CIFAR-10
CIFAR-100
soft-constraint
91.2
66.8
hard-constraint
89.2
64.7
Table 8: Impact of normalization. LMSE +CT.EGG top-1 accuracy (%).
E.4 Sensitivity to the seed.
CIFAR-10
CIFAR-100
91.1 + 0.11
66.4 + 0.45
Table 9: Sensitivity to the seed. LMSE + + top-1 accuracy using linear evaluation after 500 pre-training epochs on CIFAR-10/100, We report results averaged across 5 seeds in the format: mean + std.
We observe strong stability across different random seeds, with standard deviations only 0. and 0.45% CIFAR-10 10 and CIFAR- respectively.
20

---

E,5 Study of the embeddings.
We analyze the learned embeddings by computing pairwise cosine similarity between embeddings on CIFAR- 10. As shown in Figure [ BYO yield embeddings with mean similarities significantly above zero 0.3). This indicates concentration within cone rather than uniformity on the hypersphere. In contrast, T-REGS yields mean similarities near zero, reflecting more uniformly distributed and decorrelated embeddings, with some values slightly negative- suggesting an arrangement close to regular simplex, as per Theorem 4.1 Additionally, applying T-REGS as an auxiliary loss effectively shifts the mean cosine similarity towards zero, as illustrated in Figure 3 thus indicating its effectiveness.
F
Empirical Experiments
F.1 Study of redundancy-reduucton methods
Redundancy-Reduce methods 4 65 63 20 attempt to produce embedding variables that are
decorrelated from each other. These methods maximize the informational content of embeddings by regularizing their empirical covariance matrix.
For instance. VICReg [4) leverages (i) term to encourage the variance (diagonal of the covariance matrix) inside the current batch to be equal to 1, preventing collapse with all the inputs mapped on the same vector; (ii) and a correlation regularization, encouraging the off-diagonal coefficients of the empirical covariance matrix to be close to O, decorrelating the different dimensions of the embeddings. More formally, let Z {z1..... 2n C Rd be set of embeddings in d-dimensional space. For each dimension j E {1, d]. we denote x as the vector containing all values at dimension j across the embeddings. The variance term is defined
Lvar
max S(2²,
(17)
where e(r,,) V Var(x)) & is addiilit standard deviation, with > 0 being small constant that prevents numerical instabilities.
The covariance term is defined as:
Leow _
(18)
($1
where C(Z) the sample covariance matrix: C(Z) = - (Z =), with 1 n E.= the mean value of the embeddings.
The overall variance- covariance regularization term is a weighted:
Lear-cov L + Lcov
(19)
where v, F are hyperparameters controlling the importance of each term in the loss.
Limitations of redundancy-reductio methods. In Figure 6 we study a limitation of redundancy- reduction methods. We sample 2000 points from a non-isotropic Gaussian distribution, and observe the resulting point cloud after optimization with Lvar.ow (using Since Gaussian distributions are fully characterized by their mean and covariance matrix, we expect that by optimizing the empirical covariance matrix, the initial point cloud will converge towards sample of the standard Gaussian distribution, thus far from a uniform distribution.
25, 1, in Bardes al. [4].
Figure 6b shows the optimization of Figure 6a using the dovariance-based approach from VICReg (Equation (9)). The optimized empirical distribution is, as expected, closer to the standard Gaus- sian distribution, but exhibits artifacts such as oow-dimesiional concentration points even holes suggesting local instabilities. These artifacts persist across different parameter choices, though their specific manifestations may vary with the learning rate.
Figure 6c shows the optimization Figure 6a using LTREG We leverage Corollary 4.6 whose guarantees are valid for any Riemannian manifold, and in particular the disk. This ensures that the limiting distribution for T-REG is the uniform distribution on the disk, which prevents dimensional collapse and naturally decorrelates the distribution while maintaining the uniform distribution.
21

---

Point Cloud
Covariance Matric
1.0
0.8
2.0
0.6
9.4
-2.0
0.2
-.00
-.0
-2.0
6.0
2.0
4.0
0.0
(a) Initial Point Cloud: sampling of a non-isotropic Gaussian measure
Point Cloud
Covariance Matrix
Lvar
4.0
2.0
0.0
-2.0
0.0
500
Epochs
(b) Optimization w/ Lvar-eos
Point Cloud
Covariance Matrix
1.0
LTREG
2.0
0.8
1.0
0.6
0.0
0.4
110
0.2
0.2
-2.0
0.0
-20
-1.0
0.0
1.0
20
0.0
500
1000
1500
2000
Epochs
(c) Optimization w/ Cy the ball
Figure 6: Limitations of redundancy-based methods for non-isotropic Gaussian measure. (a) Initial sampling from non-isotropic Gaussian distribution. (b) After Lvar-cov optimization: despite achieving near-identity covariance matrix (center), the point cloud remains concentrated around its mean, with visible artifacts: holes (left). (c) Ltreg optimization achieves both uniform distribution on the disk and near-identity correlation matrix.
22

---

T-REG
Initial Point Cloud
Initial Point Cloud
0.0
TREG
1.0
2.0
002
-0.3
0.4
1.0
2.0
-1.0
0.0
2.0
0.66
0.0
0.0
200
400
600 Epocas
800
1000
1.0
2.0
(a) Optimization w/ T-REG
(b) Optimization w/ LE
(c) Losses
Figure 7: Further Studying T-REG properties through 3-d point cloud optimization. (a) T-REG successfully spreads points uniformly on the sphere by combining MST length maximization and sphere constraint, (b) using only MST length maximization leads to excessive dilation, (c) stable convergence of T-REG whereas LE does
not converge.
F.2 Promoting sample uniformity
we conduct an additional point cloud optimization experiment with different initial configuration (Figure 7 We sample 256 points along a circle while setting the remaining dimension to zero. This setup allows for the generation of point cloud with one collapsed dimension. The experimental results demonstrate consistent behavior with the findings discussed in Section 4.2 (i) T-REG successfully transforms the initial circle into a uniformly distributed point cloud the sphere (see Figure 7a), (ii) the sphere constraint is essential here, since using only the MST length loss (as in Figure 7b)) leads to a failure of convergence of the optimization.
Building upon the empirical study presented in Section 4.2
This further validates the effectiveness of our approach in promoting uniform point distribution.
G Uniformity properties
Recall from Equation 4 that (z(Z)= - -M /Z) In practice, for datasets of fixed size, minimizing LE is equivalent to minimizing F. (MST(Z)) itself. In this section, we show that, up to renormalization. - E (M)T(Z)) satisfies the four principled properties for uniformity metrics introduced in [23 Section 3.1].
A uniformity metric Ui is scalar score function on n-samples in Rd that is large on uniform-like point clouds and small on degenerate or demost-degenerate point clouds. In the following, we fix an n-sample Z = -2,,...,,,, 2n) in Rdd
We define our uniformity metric E (MST(.)) normalized by the edge length of the regular d-simplex o4 with vertices on the unit sphere gd-1.
UT.REG (2) = E (MST(Z))
(20)
1(d+1) 1/2
In practice, for fixed ambient dimension d, minimizing E (MST(Z)) or minimizing UTE is equivalent. The motivation behind renormalization is to measure the length of the minimum spanning tree relative to reference length on the unit sphere gd-1, to make a dimensionless quantity.
We now show that our uniformity score Uy-REG satisfies the desired uniformity properties:
1. Instance permutation constraint: VT E G., U ((Zmg...., ZT.,)) (Z). By construction, MST(Z) and its length are invariant under permutations of the points' indices, therefore (MST(-)) and VT-REG are also permutation invariant.
2. Instance cloning constraint: if Z' Z, then 2 ((21...., ...,,..... 2n, ,/, 2h,) .,,.),,,,)) = =Z)..
If G = (Z, E)) is an MST of Z, then
G (Z)
Zns Zns ..... .,.., 2,,, (U :15)):si511 n))
(21)
23

---

is spanning tree of (21...., 2n, .,...,..., and it is minimal for E since G it- self is minimal on and E(G") E(G + o' EGG). Therefore.
Ur :1...... 2n)) = L.REG (.) since the ambient dimension is constant.
3. Feature cloning constraint: U(21 @ ; @ zn) < U(=).
Feature cloning corresponds to pushing the points of Z to the diagonal in g which impacts the pairwise distances by a uniform scaling by a factor of /2. In particular, the MST remains the same combinatorially and have:
E (MST(Z @ Z)) 22/2 E (MST(Z)) < - (MST(Z)).
Meanwhile, since sp: d +
(2
-1/2
is an increasing function and UTREG & 0, we
a
have:
UTREG (2 - E (MST(Z e 2)) (2d) < - (MST(Z)) = UTREG (2).
4. Feature baby constraint: Vk E N,, u(2 @ 0*) < <u(Z).
Adding constant features does impact the pairwise distances, hence does not impact the minimum spanning tree and its length We thus have E (MST 2 @ 0*)) E (MST(Z)).
-1/2
As in the previous case, since yp: +
2(d+1)
an increasing function and
UT-REG S 0, we have:
(I + 0*) = -E (MST(Z 0*)) pld+ k) - (MST(Z)) = -(T-REG (Z).
H
Example of a Minimum Spanning Tree
Figure provides an example of the MST of 2d uniformly sampled point cloud.
1.0
0.5
0.0
0.5
1.0
Figure 8: Example of the MST of 2d uniformly sampled point cloud.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's contributions. The main claims about T-REG's ability to prevent dimensional collapse and enforce uniformity are clearly stated and supported by both theoretical analysis and experimental results. The pa- per's scope is well-defined, focusing self supervised learning with novel regularization approach.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in the conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
25

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides detailed theoretical analysis in Section 4. The assumptions are clearly stated, particularly in the asymptotic analysis section, where conditions about the probability measure and manifold are specified. The proofs are properly structured and referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper provides comprehensive experimental details in Section 4, including the exact setup for synthetic experiments (number of points, dimensions, initialization methods). The main experimental settings are described in the experiments section, with additional details provided in the appendix. The code will be made available, as mentioned in the checklist's experimental settings answer.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
26

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The full code will be made available. The experiments standard datasets (ImageNet and CIFAR) which are publicly available. The synthetic experiments are fully described and can be reproduced from the paper's 's description.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//iips.cc/pull wttpccpppppppppeeeeeeeeeeeeeeeee wwwpee/ggiisss//eee htt.a///guudees/ddeuuubiissss sonoooooocss for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The main experimental settings are provided the Section 5 of the paper. The full details well as comprehensive list of the hyperparameters is provided in ??. Furthermore the full code will be made available.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
27

---

Justification:
While the paper presents comprehensive experimental results across multiple datasets and settings, does not include error bars statistical significance measures. The results are reported as single values without variance estimates as related work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute ressources are listed in the ??.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 't make into the paper).
paper (e.g., preliminary or failed experiments that
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps/s//aarrpps.ccopp////eeees
Answer: [Yes]
Justification: Our research utilizes the publicly available ImageNet and CIFAR datasets, both of which are widely recognized benchmarks in the machine learning community. These datasets used in accordance with their respective licenses. We have ensured that our use of these datasets respects privacy considerations and adheres to ethical standards. To have shared all code, hyperparameters, and
promote transparency and reproducibility, detailed instructions necessary to replicate our experiments. This commitment to openness aligns with NeurIPS guidelines on reproducibility and responsible research practices.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
28

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper presents a fundamental research contribution in self-supervised learning without direct societal applications. The work focuses on improving representation learning methods without specific societal implications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The research does not involve the release of models or datasets with high risk for misuse. It uses standard benchmark datasets and focuses on methodological improve- ments in self-supervised learning.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
29

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper standard benchmark datasets (ImageNet and CIFAR) and acknowledges their use. The appendix specifies the code and datasets used along with their respective licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces T-REG, new regularization approach, and commits releasing the implementation. The documentation is provided in the paper, including theoretical analysis and experimental validation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research does not involve any crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research does not involve human subjects or participants, so IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: The methodology of the research does not involve the of Large Language Models. The work focuses on self-supervised learning for visual representations.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https://neurips.e http///rrrpp..cccooooeeeeeeee//0 for what should or should not be described.
31


Reviews:
[{"review_id": "WL3Ch7TBOy", "reviewer": "Reviewer_Hgsb", "summary": "This paper proposes T-REGS as a regularization technique to SSL method which encourages uniformity of the representation. T-REGS maximizes the length of the minimum spanning tree, which connects to entropy maximization.", "questions": "1. Could you elaborate how exactly the length of minimum spanning tree is computed? Which algorithm is used for finding the minimum spanning tree found given a batch of features and what's the complexity of the algorithm? Could you compare the complexity with other regularization techniques?\n\n2. It's unclear to me why T-REGS only uses MSE as the representation learning loss. Why would it not be able to act as a generic regularization for all SSL methods?\n\n3. Do you have any ablations for the value of alpha?\n\n4. In many popular SSL framework, the features are normalized to unit sphere. It has also been shown such normalization is beneficial. Why does the paper propose to regularize it via the soft sphere-constraint instead of a hard constraint? Could you provide empirical evidence suggesting this way better?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\nThe theoretical foundation seems to be sound.\n\n**Weakness**\n\nThe writing of the paper can be greatly improved. E.g. proofs can be moved to appendix. In the main paper, instead of showing proofs, explain why these theoretical results are important for its application here. \n\nComparing to other existing regularization techniques, T-REGS doesn't seem to result in better performance. It's unclear what's the benefit of adopting T-REGS compared to other methods.", "was_revised": true, "final_justification": "I am satisfied with the clarification during rebuttal and modifications to the paper the authors promised.", "rebuttal": "We thank you for your feedback. Please find below our answers to the questions and weaknesses you raised:\n\n**W1: “The writing of the paper can be greatly improved. E.g., proofs can be moved to the appendix. In the main paper, instead of showing proofs, explain why these theoretical results are important for its application here.”**\n\nWe chose to include the proofs of the two pivotal results (Proposition 4.1, Proposition 4.3) in the main text because these results are central to the paper and explain the behavior of T-REG, with relatively short and elementary proofs. The final version of the paper will provide intuitive explanations of the role of the MST for preventing dimensional collapse and promoting well-distributed embeddings, based on Propositions 4.1 and 4.3—see our answer to Weakness 2 of Reviewer zFs8 for the details. This will add a few sentences to the introduction and Section 4, which will fit together with the new experimental results within the extra allocated page. \n\n**W2. “Comparing to other existing regularization techniques, T-REGS doesn't seem to result in better performance. It's unclear what's the benefit of adopting T-REGS is compared to other methods.**\n\nThe main benefits of T-REGS stem from its conceptual and implementation simplicity, strong theoretical grounding, and robustness to hyperparameter variations (i.e., batch size and embedding dimension). Indeed, T-REGS offers a novel, conceptually simple solution based on well-understood graph theory and statistical dimension estimation. It follows the principled line of thought of Fang et al. [1] while proposing a more straightforward and unexplored approach: maximizing the length of the minimum spanning tree, which we found to satisfy the same properties naturally but with significantly lower computational overhead in practice and improved numerical stability. As a result, T-REGS demonstrates better scalability on larger datasets such as ImageNet-100 and ImageNet-1k.\n\nCompared with other existing approaches, T-REGS addresses the following well-known limitations: \n\n*(i) Theoretical grounding*: Asymmetric methods lack theoretical justification to explain how the architecture helps prevent collapse [2]. Covariance-based methods only leverage the second moment of the data distribution and are thus blind to the concentration points of the density, which can prevent convergence to the uniform distribution (Figure 5 in the Supp. Mat.), \n\n*(ii) Computational limitations*: contrastive methods are sensitive to the number of negative samples and require large batch sizes [3, 4]. Covariance-based approaches often rely on both large batch sizes and large output dimensions [2].  By contrast, T-REGS proves to be robust to both batch size and embedding dimensions (Appendices D.1 and D.2).\n\nTo strengthen the empirical validation of our method, we have conducted complementary experiments based on your suggestions. We propose a revised version of Table 1 that highlights the new results (please refer to our answer to Q2).\n\n**Q1: “Could you elaborate how exactly the length of minimum spanning tree is computed? Which algorithm is used for finding the minimum spanning tree found given a batch of features and what's the complexity of the algorithm? Could you compare the complexity with other regularization techniques?”**\n\nIn practice, we use Kruskal’s algorithm, which runs in $\\mathcal{O}(B^2 (D + \\log B))$ time in the worst case, where $B$ is the number of samples in the batch and $D$ is the dimension. This choice was motivated by the simplicity of its implementation, with a resulting computational overhead that is similar to well-established methods such as VICReg or SimCLR; and faster wall-clock time per step. We report the complexities in Table A.1 below, along with wall-clock time per step, following a suggestion by Reviewer MGro. \n\n#### Table A.1: Compute comparison. The ranges are reported from Garrido et al [4] for SimCLR and Bardes et al [3] for VICReg.\n| Method | Loss Complexity | B range | D range | Wall-clock time (per step, sec) |\n| - | - | - | - | :-:|\n| SimCLR | $\\mathcal{O}(B^2⋅D)$ | [2048-4096] | [256-1024] | 0.22 +/- 0.03 |\n| VICReg | $\\mathcal{O}(B⋅D^2)$ | [512-2048] | [4096-8192] | 0.23 +/- 0.02 |\n| T-REGS | $\\mathcal{O}(B^2 (D + \\log B))$ | [512-1024] | [512-2048] | 0.20 +/- 0.001 |\n\n\n**Q2: “It's unclear to me why T-REGS only uses MSE as the representation learning loss. Why would it not be able to act as a generic regularization for all SSL methods?”**\n\nT-REG is indeed designed as a generic regularization technique and, in principle, can be combined with a wide range of self-supervised learning (SSL) objectives—not just as a standalone regularization with an MSE. In the submitted version of the paper, we decided to emphasize the behavior of T-REGS (T-REG combined with MSE invariance) as a standalone method. Given the feedback received, we have run further experiments combining T-REG with a couple of other regularizers: BYOL and Barlow Twins, on CIFAR-10/100. We provide the corresponding results below in Table A.2, which highlight the benefits of such combinations (best performances are shown in bold): \n\n#### Table A.2: CIFAR-10/100 Evaluation \n| | CIFAR-10 |CIFAR-100 |\n|-|:-:|:-:|\n| MoCo v2 | 90.7 | 60.3 |\n| MoCo v2 + $\\mathcal{L}_u$| 91.0 | 61.2 |\n| MoCo v2 + $\\mathcal{W}_2$| 91.4 | 63.7 |\n| ZeroCL | 91.4 | **68.5** |\n| ZeroCL + $\\mathcal{L}_u$ | 91.3 | 68.4 |\n| ZeroCL + $\\mathcal{W}_2$ | 91.4 | **68.5** |\n| BYOL | 89.5 | 63.7 |\n| BYOL + $\\mathcal{L}_u$ | 90.1 |62.7 |\n| BYOL + $\\mathcal{W}_2$ | 90.1 |65.2 |\n| BYOL + **T-REG** | 90.4 | 65.7 |\n| Barlow Twins | 91.2 | 68.2 |\n| Barlow Twins + $\\mathcal{L}_u$ | 91.4 | 68.4 |\n| Barlow Twins + $\\mathcal{W}_2$ | 91.4 | **68.5** |\n| Barlow Twins + **T-REG** | **91.8** | **68.5** |\n\nWe observe that adding our regularizer improves the results both for BYOL and Barlow Twins, highlighting its superiority. \n \n**Q3:  “Do you have any ablations for the value of alpha?”**\n\nWe tested  $\\alpha=1$ and $\\alpha=0.5$ and found that $\\alpha=1$ works best; however, we have not performed an exhaustive hyperparameter sweep. Table A.3 reports the results:\n\n#### Table A.3: Ablation of $\\alpha$. Results on CIFAR-10/CIFAR-100.\n| $\\alpha$ | CIFAR-10 | CIFAR-100|\n|--|:--:|:--:|\n|$\\frac{1}{2}$ | 90.6 | 66.0 |\n| 1 | 91.2 | 66.8 |\n\n\n**Q4: “In many popular SSL framework, the features are normalized to unit sphere. It has also been shown such normalization is beneficial. Why does the paper propose to regularize it via the soft sphere-constraint instead of a hard constraint? Could you provide empirical evidence suggesting this way better?”**\n\nIndeed, some popular SSL frameworks such as SimCLR and BYOL employ explicit normalization of features to the unit sphere, enforcing a hard constraint. Others do not enforce such constraints explicitly, and instead rely on soft mechanisms—such as VICReg, which incorporates variance and covariance regularization terms in its loss, implicitly constraining the distribution (and to some extent, the norms) of the embeddings.\n\nWe chose a soft sphere constraint for the following reasons:\n-  A hard normalization has been shown to ignore the importance of the embedding norm for gradient computation, whereas a soft constraint enables better embedding optimization [5,6]\n- Furthermore, during our initial experiment, we found that relaxing the sphere constraint from a hard one to a soft one provides more leeway for optimization and leads to improved results. Table A.4 shows the results with/without normalization.\n\n#### Table A.4: Results with and without normalization\n| | CIFAR-10 | CIFAR-100 |\n| - | :-: | :-: |\n| soft constraint | 91.2 | 66.8 |\n| hard constraint | 89.2 | 64.7 |\n\n\n---\n\nReferences:\n\n[1] X. Fang, J. Li, Q. Sun, and B. Wang. Rethinking the uniformity metric in self-supervised learning. In ICLR, 2024\n\n[2] X. Weng, Y. Ni, T. Song, J. Luo, R. M. Anwer, S. Khan, F. Khan, and L. Huang. Modulate your spectrum in self-supervised learning. In ICLR, 2024\n\n[3] Q. Garrido, Y. Chen, A. Bardes, L. Najman, and Y. Lecun. On the duality between contrastive and non-contrastive self-supervised learning. In ICLR, 2023\n\n[4] A. Bardes, J. Ponce, and Y. Lecun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR, 2022.\n\n[5] D. Zhang, Y. Li1, Z. Zhang. Deep Metric Learning with Spherical Embedding. NeurIPS 2020\n\n[6] J. Zhang, H. Zhang, R. Vasudevan, M. Johnson-Roberson. Hyperspherical Embedding for Point Cloud Completion. In CVPR 2023."}, {"review_id": "DXR7CVe5hB", "reviewer": "Reviewer_zFs8", "summary": "To avoid dimensional collapse and enhance uniformity in self-supervised learning, the authors proposed the Minimum Spanning Tree (MST) over the learned representation. Moreover, the authors provided theoretical analysis.", "questions": "(1) In lines 66-70, the author introduced the concept of $\\alpha$-minimum spanning tree, but the author didn’t combine it well with the context of self-supervised learning. I’m very curious about how to construct a connected acyclic graph given the embeddings of self-supervised learning.\n\n(2) I want to know how the Minimum Spanning Tree solves the problems of avoiding dimensional collapse and enhancing uniformity in self-supervised learning.", "limitations": "No", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n(1) The authors proposed the Minimum Spanning Tree (MST) to avoid dimensional collapse and enhance uniformity in self-supervised learning, which is a novel approach.\n\n(2) The authors provided a great deal of theoretical analysis for the proposed Minimum Spanning Tree (MST).\n\n\n**Weaknesses:**\n\n(1) The authors did not provide a relatively clear explanation of how the Minimum Spanning Tree (MST) is combined with Self-Supervised learning. The picture provided by the authors in Figure 1 is common knowledge. Instead of providing a common-sense picture, the authors should have provided a picture in the introduction to help readers intuitively understand how the Minimum Spanning Tree (MST) is combined with Self-Supervised learning.\n\n(2) I didn’t understand how the Minimum Spanning Tree solves the problems of avoiding dimensional collapse and enhancing uniformity in self-supervised learning.\n\nI admit that I haven’t fully understood the author’s paper. If the author can address my two main concerns and some detailed issues, I will raise the score.", "was_revised": true, "final_justification": "In the initial submission, the author did not introduce the integration of Kruskal’s algorithm into the SSL pipeline, which made me unable to understand how the author implemented the MST algorithm in SSL. After the discussion with the authors, they clarified this pipeline. Therefore, I would like to raise my rating.", "rebuttal": "We thank you for your feedback. Please find below our answers to the questions and weaknesses you raised, and let us know how we can help further clarify the method for you: \n\n**W1: “The authors did not provide a relatively clear explanation of how the Minimum Spanning Tree (MST) is combined with Self-Supervised learning. The picture provided by the authors in Figure 1 is common knowledge. Instead of providing a common-sense picture, the authors should have provided a picture in the introduction to help readers intuitively understand how the Minimum Spanning Tree (MST) is combined with Self-Supervised learning.”**\n\nWe agree that Figure 1 in the introduction is standard, and that a more targeted illustration would help clarify how the MST is integrated into the SSL framework. We will modify the figure to better demonstrate  how the minimum spanning tree is constructed from embeddings and how it fits into the pipeline. For this, we will take inspiration from the visuals of Figure 2, which will be moved from Page 7 to the introduction to provide a better initial intuition for the reader.\n\nWe will also add a few clarifying sentences to the introduction, in the same spirit as our answer to Q1 below.\n\n\n**Q1:  “In lines 66-70, the author introduced the concept of alpha-minimum spanning tree, but the author didn’t combine it well with the context of self-supervised learning. I’m very curious about how to construct a connected acyclic graph given the embeddings of self-supervised learning.”**\n\nWe kindly refer you to Equation (10) in Section 5, which explains this point. \n\nOur regularization is applied separately on the embeddings of the two transformations of the batch obtained from the two network branches. Specifically, denoting by $Z=(z_1, \\dots ,z_n)$ and $Z' = (z_1', \\dots, z_n')$ \nthese two embeddings in $d$ dimensions, two minimum spanning trees $\\mathrm{MST}(Z)$ and $\\mathrm{MST}(Z’)$ are constructed on these two sets of points. The MST of $Z$ treats each point $z_i$ as a vertex, and connects the vertices by edges to form the connected acyclic graph of minimum total length, where the length of an edge corresponds to the Euclidean distance between its vertices. Likewise for the MST of $Z'$. The regularization terms in Equation (10) tend to maximize the total lengths of these two minimum spanning trees while not expanding the point clouds too much, which provably increases their entropy and dimensionality (please refer to our answer to W2 and Q2 below for the explanation).  \n\nWe intend to add one or two sentences in the same spirit as above to the introduction in order to clarify this point. \n\n**W2 & Q2 : “I didn’t understand how the Minimum Spanning Tree solves the problems of avoiding dimensional collapse and enhancing uniformity in self-supervised learning.”/  “I want to know how the Minimum Spanning Tree solves the problems of avoiding dimensional collapse and enhancing uniformity in self-supervised learning.”**\n\nThe behavior of T-REG is explained by Proposition 4.1, which we have now extended to allow the points to lie inside the ball and not just on the sphere. First, minimizing the term $\\mathcal{L}_E$ in Equation (7) expands the point cloud until the sphere constraint $\\mathcal{L}_S$ becomes the dominating term; at that stage, the points stop expanding and start spreading themselves out uniformly along the sphere of directions, which prevents dimensional collapse and maximizes uniformity. The amount of expansion before spreading is prescribed by the strength of the sphere constraint term versus the $\\mathrm{MST}$ term in the loss, which is driven by the ratio between their respective mixing parameters $\\lambda$ and $\\gamma$.\n\nFor further visual intuition, we kindly refer you to the synthetic experiments in dimension 3 (Figures 2.a and 2.b, which will be moved to the introduction). We generate a highly concentrated point cloud of 256 points (orange points in Figures 2a and 2b). After optimization, T-REG successfully transforms this initially concentrated point cloud into a uniformly distributed point cloud on the sphere (blue points in Figure 2.a), while occupying the whole dimensional space—demonstrating no dimensional collapse. Without the sphere constraint, the optimization fails to converge as points diverge to infinity (Figure 2.b)."}, {"review_id": "XJVZAta9Q8", "reviewer": "Reviewer_MGro", "summary": "This paper introduces T-REGS, a novel regularization framework for joint-embedding self-supervised learning (JE-SSL). The core idea is to regularize the learned embeddings by maximizing the length of the Minimum Spanning Tree (MST) constructed over the embedding vectors within a batch, while simultaneously applying a soft constraint to keep the embeddings on the unit hypersphere.\n\nThe authors claim that this approach simultaneously addresses two critical issues in representation learning:\n\nDimensional Collapse: It encourages the embeddings to occupy the full dimensionality of the representation space.\nSample Uniformity: It promotes an even distribution of embeddings on the hypersphere.\nThe paper provides a solid theoretical analysis, connecting the MST length maximization to statistical dimension estimation and Rényi entropy maximization. This analysis shows that for small sample sizes, the optimal configuration of points under the T-REG loss forms a regular simplex, and asymptotically for large samples, it encourages a uniform distribution. The method's effectiveness is validated through experiments on synthetic data, which clearly illustrate its properties, and on standard SSL benchmarks (CIFAR-10/100, ImageNet-100/1k), where it achieves competitive performance compared to state-of-the-art methods.", "questions": "refer to weaknesses.", "limitations": "refer to weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths \n1.Novelty and Elegance of the Core Idea: The central contribution—using the length of the Minimum Spanning Tree as a regularizer—is highly novel and conceptually elegant. It draws a clever and principled connection between a classic algorithm from graph theory, concepts from statistical dimension estimation, and the modern challenges of self-supervised representation learning. This is a significant conceptual contribution that could inspire new research directions.\n2.Strong Theoretical Grounding: The paper is not just a heuristic proposal; it is backed by rigorous theoretical analysis (Section 4.1).\nProposition 4.1 provides a clear and intuitive result for the small-batch regime (n ≤ d + 1), showing that the loss is maximized when embeddings form a regular simplex. This directly proves that the regularizer prevents dimensional collapse in this setting.\nTheorem 4.2 and Proposition 4.3 establish an asymptotic link between maximizing the MST length and maximizing the Rényi entropy of the embedding distribution. Since maximum entropy on a compact set corresponds to the uniform distribution, this provides a strong justification for the claim that T-REGS promotes sample uniformity.\n3.Simplicity and Practicality: Compared to some competing methods that aim for similar goals, T-REGS is relatively simple. For instance, the Optimal Transport-based regularizer proposed by Fang et al. can be computationally expensive and numerically unstable. T-REGS, while requiring an MST computation, avoids complex operations like SVD and is conceptually straightforward to implement using existing fast, parallelized MST algorithms.\n\nWeaknesses \n\n1.Empirical Performance: While the results are competitive, T-REGS does not consistently set a new state-of-the-art on the reported benchmarks. For example, on CIFAR-100, it lags behind several W2-regularized methods, and on ImageNet-1k, it is slightly outperformed by methods like INTL and Zero-CL. While SOTA performance is not a strict requirement for a conceptually novel paper, a stronger showing would make the practical case for adoption more compelling. The authors acknowledge this limitation in the conclusion [305-306].\n2.Computational Overhead of MST: The paper states that fast GPU-based MST implementations exist, but it does not provide a practical analysis of the computational overhead. For a batch of size B, constructing the complete graph has a complexity of O(B²), and standard MST algorithms like Kruskal's or Prim's are at least O(B²). This could become a bottleneck for very large batch sizes, which are common in SSL. A comparison of wall-clock training time per step against baselines like VICReg (which computes a covariance matrix, O(B*D²)) or BarlowTwins would be a valuable addition to assess the practical scalability.", "was_revised": true, "final_justification": "The authors not only address all problems from my comments, but also the authors make the proposition more convincing by extending the core proposition to a more relaxed assumption with extra experiments to improve existing results.\nHence, I keep my rate \"accept\".", "rebuttal": "We thank you for your feedback and positive comments. Please find below our answers to the weaknesses you raised:\n\n**W1. “1.Empirical Performance: While the results are competitive, T-REGS does not consistently set a new state-of-the-art on the reported benchmarks. For example, on CIFAR-100, it lags behind several W2-regularized methods, and on ImageNet-1k, it is slightly outperformed by methods like INTL and Zero-CL. While SOTA performance is not a strict requirement for a conceptually novel paper, a stronger showing would make the practical case for adoption more compelling. The authors acknowledge this limitation in the conclusion [305-306].“**\n\nIn order to strengthen the empirical validation of our regularization method, we followed suggestions by Reviewers Hgsb and hUvh and conducted further experiments to integrate T-REG as an auxiliary regularizer to BYOL and Barlow Twins on CIFAR-10/100. The results are reported in Table A.2 below, which highlights the benefits of such combinations (best performances are shown in bold):\n\n#### Table A.2: CIFAR-10/100 Evaluation \n| | CIFAR-10 |CIFAR-100 |\n|-|:-:|:-:|\n| MoCo v2 | 90.7 | 60.3 |\n| MoCo v2 + $\\mathcal{L}_u$| 91.0 | 61.2 |\n| MoCo v2 + $\\mathcal{W}_2$| 91.4 | 63.7 |\n| ZeroCL | 91.4 | **68.5** |\n| ZeroCL + $\\mathcal{L}_u$ | 91.3 | 68.4 |\n| ZeroCL + $\\mathcal{W}_2$ | 91.4 | **68.5** |\n| BYOL | 89.5 | 63.7 |\n| BYOL + $\\mathcal{L}_u$ | 90.1 |62.7 |\n| BYOL + $\\mathcal{W}_2$ | 90.1 |65.2 |\n| BYOL + **T-REG** | 90.4 | 65.7 |\n| Barlow Twins | 91.2 | 68.2 |\n| Barlow Twins + $\\mathcal{L}_u$ | 91.4 | 68.4 |\n| Barlow Twins + $\\mathcal{W}_2$ | 91.4 | **68.5** |\n| Barlow Twins + **T-REG** | **91.8** | **68.5** |\n\nWe observe that adding our regularizer improves the results both for BYOL and Barlow Twins, highlighting its superiority. \n\n**W2  “2.Computational Overhead of MST: The paper states that fast GPU-based MST implementations exist, but it does not provide a practical analysis of the computational overhead. For a batch of size B, constructing the complete graph has a complexity of O(B²), and standard MST algorithms like Kruskal's or Prim's are at least O(B²). This could become a bottleneck for very large batch sizes, which are common in SSL. A comparison of wall-clock training time per step against baselines like VICReg (which computes a covariance matrix, O(BxD²)) or BarlowTwins would be a valuable addition to assess the practical scalability.”**\n\nIndeed, the worst-case asymptotic complexity is significant, but not larger than for other methods. Meanwhile, the hidden constants in the complexity bounds are small, which has an impact in practice since the batch sizes are relatively small (only a few thousand points). Furthermore, while Kruskal’s algorithm per se is sequential, its pre-processing (namely, computing the distance matrix and sorting its entries) is the most costly step. In practice, one can use a parallelized GPU implementation for it. This is what the torchPH library (used in our implementation) does.\n\nTo confirm this insight, we evaluated the computational overhead of T-REGS, VICReg (a covariance-based method), and SimCLR (a contrastive method). We report the results in the table below, where $B$ represents the batch size and $D$ represents the output dimension. The wall-clock comparisons are averaged over 500 steps for each method.\n\n#### Table A.1: Compute comparison. The ranges are reported from Garrido et al [4] for SimCLR and Bardes et al [3] for VICReg. \n| Method | Loss Complexity | B range | D range | Wall-clock time (per step, sec) |\n| - | - | - | - | :-:|\n| SimCLR | $\\mathcal{O}(B^2⋅D)$ | [2048-4096] | [256-1024] | 0.22 +/- 0.03 |\n| VICReg | $\\mathcal{O}(B⋅D^2)$ | [512-2048] | [4096-8192] | 0.23 +/- 0.02 |\n| T-REGS | $\\mathcal{O}(B^2 (D + \\log B))$ | [512-1024] | [512-2048] | 0.20 +/- 0.001 |\n\nWe observe a resulting computational overhead that is similar to well-established methods such as VICReg or SimCLR; and faster wall-clock time per step.\n  \n---\n\nReferences:\n\n[1] A. Bardes, J. Ponce, and Y. Lecun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR, 2022.\n\n[2] Q. Garrido, Y. Chen, A. Bardes, L. Najman, and Y. Lecun. On the duality between contrastive and non-contrastive self-supervised learning. In ICLR, 2023"}, {"review_id": "ZVT1jW6yPP", "reviewer": "Reviewer_hUvh", "summary": "The paper introduces T-REGS, a framework based on Minimum Spanning Tree (MST), to regularize SSL methods to prevent dimensional collapse and promote uniformity. Theoretically, under the assumption of strict sphere constraint, the authors observe that when $n \\leq d+1$ ($n$ is the point number and $d$ is the dimension), the optimal MST length is achieved when the embeddings form a simplex. For large samples, the authors first relate the MST length to the  R\\'{e}nyi entropy and then propose that uniform distribution maximizes the entropy. In the experiments, T-REG demonstrates the ability the spread the initial point cloud uniformly on the sphere. Additionally, T-REGS yields competitive results across different datasets, though it fails to outperform the baseline.", "questions": "1. Are there any results of combining T-REGS with existing SSL methods, both contrastive and non-contrastive? This would further demonstrate the effectiveness of T-REGS.\n2. Can T-REGS be generalized to supervised learning?\n3. What is the extra cost of MST computation?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper provides a detailed theoretical analysis to gain insights of MST and guide the design of T-REG.\n2. The authors conduct experiments to demonstrate the effects of T-REG (Figure 2).\n3. The idea of this paper is novel and the writing is generally easy to follow.\n\nWeaknesses:\n1. The theoretical analysis is based on strict sphere constraints, which cannot be completely ensured in reality. Since regularization is only able to gradually enforce the sphere constraints, a relaxed assumption is more suitable for the setting.\n2. There is still a gap between the performance of T-REGS and the baselines, making the effectiveness of the method not convincing enough. More trials regarding the proper choices of the parameters are needed.\n3. It is observed in Table 2 that higher values of $\\frac{\\beta}{\\gamma}$ and $\\frac{\\gamma}{\\lambda}$ lead to better performance. Intuitively, this would mean that $\\mathcal{L}_E$ and $\\mathcal{L}_S$ are not that necessary. To justify the regularization terms, you should first show that no regularization term has poorer performance than your method, and that extremely large $\\frac{\\beta}{\\gamma}$ and $\\frac{\\gamma}{\\lambda}$ also harms the classification accuracy.", "was_revised": true, "final_justification": "During the rebuttal phase, the authors extended the core proposition to a more relaxed assumption (i.e., the points can lie inside the sphere), which makes the proposition more convincing. They also did extra experiments to improve existing results. Besides, they addressed my questions on the results of the experiment on the coefficients of loss functions. Based on these improvements, I have decided to raise my rating to 4.", "rebuttal": "We thank you for your feedback. Please find below our answers:\n\n**W1. “The theoretical analysis is based on strict sphere constraints, which cannot be completely ensured in reality. Since regularization is only able to gradually enforce the sphere constraints, a relaxed assumption is more suitable for the setting.”**\n\nThank you for the nice and valuable suggestion! We have extended Proposition 4.1 to the case where the points lie inside the unit ball, not only on the sphere, with the same conclusion: the maximum length of the $\\mathrm{MST}$ is achieved by points sitting on the sphere, at the vertices of a regular simplex. This new statement will replace Proposition 4.1 in the final version of the paper. Its proof is actually simpler than the previous one, relying on a standard result in convex geometry—see e.g. Chapter 14, Theorem 14.6, in [1]---to which we add a technical lemma relating the total sum of pairwise distances to the total length of the MST.\n\nThe new Proposition 4.1 explains the behavior of T-REG when used with a soft sphere constraint: first, minimizing the term\n$\\mathcal L_E$ in Equation (7) expands the point cloud until the sphere constraint $\\mathcal L_S$ becomes the dominating term; at that stage, the points stop expanding and start spreading themselves out uniformly along the sphere of directions. The amount of expansion before spreading is prescribed by the strength of the sphere constraint term versus the $\\mathrm{MST}$ term in the loss, which is driven by the ratio between their respective mixing parameters $\\lambda$ and $\\gamma$. Note that $\\mathcal L_E$  grows linearly (or sublinearly for $\\alpha<1$) with the scale, whereas $\\mathcal L_S$ grows quadratically, which explains why $\\mathcal L_S$ eventually becomes the dominating term.\n\n\n**W2 & Q1 “There is still a gap between the performance of T-REGS and baselines, making its effectiveness not convincing enough. More trials regarding the proper choices of the parameters are needed.” /  “Are there any results of combining T-REGS with existing SSL, both contrastive and non-contrastive? This would further demonstrate its effectiveness”**\n\nGiven the feedback received, we conducted further experiments to combine T-REG with a couple of other regularizers: BYOL and Barlow Twins, on CIFAR-10/100. We provide the results in Table A.2, which highlight the benefits of such combinations (best performances in bold):\n\n#### Table A.2: CIFAR-10/100 Evaluation\n| | CIFAR-10 |CIFAR-100 |\n|-|:-:|:-:|\n| MoCo v2 | 90.7 | 60.3 |\n| MoCo v2 + $\\mathcal{L}_u$| 91.0 | 61.2 |\n| MoCo v2 + $\\mathcal{W}_2$| 91.4 | 63.7 |\n| ZeroCL | 91.4 | **68.5** |\n| ZeroCL + $\\mathcal{L}_u$ | 91.3 | 68.4 |\n| ZeroCL + $\\mathcal{W}_2$ | 91.4 | **68.5** |\n| BYOL | 89.5 | 63.7 |\n| BYOL + $\\mathcal{L}_u$ | 90.1 |62.7 |\n| BYOL + $\\mathcal{W}_2$ | 90.1 |65.2 |\n| BYOL + **T-REG** | 90.4 | 65.7 |\n| Barlow Twins | 91.2 | 68.2 |\n| Barlow Twins + $\\mathcal{L}_u$ | 91.4 | 68.4 |\n| Barlow Twins + $\\mathcal{W}_2$ | 91.4 | **68.5** |\n| Barlow Twins + **T-REG** | **91.8** | **68.5** |\n\nAdding our regularizer improves the results both for BYOL and Barlow Twins, highlighting its superiority.\n\n**W3. “Table 2: higher values of beta/gamma and gamma/lambda lead to better performance. Intuitively, this would mean that L_E and  L_S are not that necessary. To justify the regularization terms, you should first show that no term has poorer performance than your method, and that extremely large and also harms the accuracy.”** \n\nFollowing your recommendation, we propose to complete Table 2 as shown below:\n\n#### Table B.1: Impact of coefficients. We report Top-1 accuracy on ImageNet-1k.\n| β  | γ | λ | β/γ | γ/λ | ACC-1 |\n| - |:-:| - | - | - | - |\n| 1 | 0 | 0 | 0 | 0 | collapse |\n| 1 | 1 | 0 | 0 | 0 | collapse |\n| 10 | 1 | 1 | 10 | 1 | collapse |\n| 10 | 0.5 | 5e-2 | 20 | 10 | 25.7 |\n| 10 | 0.2 | 2e-2 | 50 | 10 | 45.4 |\n| 10 | 0.2 | 2.5e-3 | 20 | 200 | 65.0 |\n| 10 | 0.2 | 1e-3 | 50 | 200 | 65.2 | \n| 10 | 0.4 | 2e-3 | 20 | 250 | 64.9 |\n| 10 | 0.2 | 8e-4 | 50 | 250 | 66.1 |\n| 10 | 0.02 | 4e-5 | 500 | 500 | 51.8 | \n\nThe new version of the table (Table B.1) more clearly highlights that no regularization at all leads to collapse  (line 1), that large values of regularization also harm classification accuracy (lines 9-10: 51.8% vs best performance of 66.1%), and thus that both $\\mathcal L_E$ and $\\mathcal L_S$ are essential for optimal performance.\n\n\n**Q2. “Can T-REGS be generalized to supervised learning?”**\n\nThe regularization, T-REG, can indeed be applied to other settings beyond the one considered in the paper, when uniformity properties are helpful and/or where dimensional collapse can be an issue. While we do not have an example in supervised learning per se, we have other examples of applications in unsupervised or self-supervised settings.\n\nFor instance, Diffusion Models have been shown to suffer from dimensional collapse. Recently, Wang et al. [2] proposed to regularize a generative model's internal representations by encouraging their dispersion in the hidden space.\n\nIn multi-modal applications, several papers [3,4] have highlighted that uniformity properties can be highly beneficial to bridge the *modality gap* [5] in multi-modal models. The *modality gap* is a phenomenon observed in pre-trained multimodal models, such as CLIP [6], which provide transferable embeddings mainly by aligning images and text in the same embedding space. Better uniformity allows for the embeddings from the different modalities to more evenly distribute on the hypersphere in the embedding space. To validate the generalizability, we assess our method on image-text retrieval, a representative vision-language task, following the same experimental setting as Oh et al. (Section 5.2) [3] on Flickr30k and MS-COCO. More precisely, we apply T-REG as a regularizer to finetune CLIP: ($\\mathcal L = \\mathcal L_{\\text{CLIP}} + \\mathcal L_{\\text{T-REG}}$), where $\\mathcal L_{\\text{CLIP}}$ is the standard CLIP loss [6] and $\\mathcal L_{\\text{T-REG}}$ refers to Eq. (7). We report the top 1/5 Recall (R@1, R@5) in Table B.2 below with Image-to-text (i → t) and text-to-image (t → i) retrieval results. Except for the last line, the table is inherited from Oh et al. [3] (bold indicates the best performance). Complete results and implementation details will be included in the supplementary material of the paper. \n\n#### Table B.2: Image-to-text (i → t) and text-to-image (t → i) retrieval results\n| Method | Flickr30k  | | |  | | | | MS-COCO  | | | | |  \n|--------|:---------------:|:---------:|---|:---:|----|---------|---|:-----:|:-----:|-----|:----:|:-----:|\n| | i→t |  |  |  t→i | | ||  i→t | ||  t→i | \n| | R@1 | R@5 | | R@1 | R@5 | | | R@1 | R@5 | | R@1 | R@5 |\n| Zero-Shot | 71.1 | 90.4 | | 68.5 | 88.9 | | | 31.9 | 56.9 | | 28.5 | 53.1 |\n| Finetune | 81.2 | 95.5 | | 80.7 | 95.8 | | | 36.7 | 63.6 | |  36.9 | 63.9 |\n| ES [7]| 71.8 | 90.0 | | 68.5 | 88.9 | | | 31.9 | 56.9 | |  28.7 | 53.0 |\n| i-Mix [8] | 72.3 | 91.7 | | 69.0 | 91.1 | | | 34.0 | 63.0 | |  34.6 | 62.2 |\n| Un-Mix [9] | 78.5 | 95.4 | | 74.1 | 91.8 | | | 38.8 | 66.2 | |  33.4 | 61.0 |\n| m³-Mix [5] | 82.3 | 95.9 | | **82.7** | 96.0 | | | 41.0 | 68.3 | |  39.9 | 67.9 |\n| **T-REG** | **83.2** | **96.0** | | 80.8 | **96.4** | | | **41.6** |**68.7**| | **41.5** | **68.7** |\n\n**Q3. “What is the extra cost of MST computation?”**\n\nWe use Kruskal’s algorithm, which runs in $\\mathcal{O}(B^2 (D + \\log B))$ time in the worst case, where $B$ is the number of samples in the batch and $D$ is the dimension, with a resulting computational overhead that is similar to well-established methods such as VICReg or SimCLR; and faster wall-clock time per step.  We report the complexities in the table below, along with wall-clock time per step, following a suggestion by Reviewer MGro. \n\n#### Table A.1: Compute comparison. The ranges are reported from Garrido et al [4] for SimCLR and Bardes et al [3] for VICReg.\n| Method | Loss Complexity | B range | D range | Wall-clock time (per step, sec) |\n| - | - | - | - | :-:|\n| SimCLR | $\\mathcal{O}(B^2⋅D)$ | [2048-4096] | [256-1024] | 0.22 +/- 0.03 |\n| VICReg | $\\mathcal{O}(B⋅D^2)$ | [512-2048] | [4096-8192] | 0.23 +/- 0.02 |\n| T-REGS | $\\mathcal{O}(B^2 (D + \\log B))$ | [512-1024] | [512-2048] | 0.20 +/- 0.001 |\n\n\n---\n\nReferences:\n\n[1] T. M. Apostol and M. A. Mnatsakanian: “New Horizons in Geometry”, Mathematical Association of America, Dolciani Mathematical Expositions #47, 2012, ISBN 978-0-88385-354-2. \n\n[2] R. Wang, K. He. Diffuse and Disperse: Image Generation with Representation Regularization. Arxiv 2025\n\n[3] C. Oh, J. So, H. Byun, Y. Lim, M. Shin, J-.J. Jeon, K. Song. Geodesic Multi-Modal Mixup for Robust Fine-Tuning. NeurIPS 2023\n\n[4] J. Kim, S. Hwang.   Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations. In CVPR, 2025\n\n[5]  V. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. NeurIPS, 2022\n\n[6] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell,\nP. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision.\nIn Proc. ICML, 2021\n\n[7] A. C. Li, A. A. Efros, and D. Pathak. Understanding collapse in non-contrastive siamese\nrepresentation learning. In ECCV, 2022\n\n[8] K. Lee, Y. Zhu, K. Sohn, C.-L. Li, J. Shin, and H. Lee. i-mix: A domain-agnostic strategy for contrastive representation learning. In ICLR, 2021.\n\n[9] Z. Shen, Z. Liu, Z. Liu, M. Savvides, T. Darrell, and E. Xing. Un-mix: Rethinking image\nmixtures for unsupervised visual representation learning.  In AAAI, 2022\n\n[10] Q. Garrido, Y. Chen, A. Bardes, L. Najman, and Y. Lecun. On the duality between contrastive and non-contrastive self-supervised learning. In ICLR, 2023\n\n[11] A. Bardes, J. Ponce, and Y. Lecun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR, 2022."}]

Author rebuttal:

Meta-review: The manuscript provides a simple regularization framework for self-supervised learning based on the length of the Minimum Spanning Tree (MST) over the learned representation. The idea is simple and regularization-based, which means that its adaptability to existing methods is high. The idea is supported by a clear theoretical background. Although the original manuscript did not successfully convey the importances of the main theorems, they have been clarified in the author-reviewer discussions. It would not be difficult to include those explanations with the reviewers about the significance of those theorems. Other weaknesses, e.g., about empirical results and computational complexity, have been addressed in the discussion.
Decision: Accept (spotlight)