SSOLE: Rethinking Orthogonal Low-rank Embedding for Self-Supervised Learning
Authors: Lun Huang
Qiang Qiu
Guillermo Sapiro
Venue: ICLR 2025 Poster (2025)
Keywords: self-supervised learning
orthogonal low-rank embedding
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We address key challenges of applying orthogonal low-rank embedding to self-supervised learning.

Abstract: Self-supervised learning (SSL) aims to learn meaningful representations from unlabeled data. Orthogonal Low-rank Embedding (OLE) shows promise for SSL by enhancing intra-class similarity in a low-rank subspace and promoting inter-class dissimilarity in a high-rank subspace, making it particularly suitable for multi-view learning tasks. However, directly applying OLE to SSL poses significant challenges: (1) the virtually infinite number of "classes" in SSL makes achieving the OLE objective impractical, leading to representational collapse; and (2) low-rank constraints may fail to distinguish between positively and negatively correlated features, further undermining learning. To address these issues, we propose SSOLE (Self-Supervised Orthogonal Low-rank Embedding), a novel framework that integrates OLE principles into SSL by (1) decoupling the low-rank and high-rank enforcement to align with SSL objectives; and (2) applying low-rank constraints to feature deviations from their mean, ensuring better alignment of positive pairs by accounting for the signs of cosine similarities. Our theoretical analysis and empirical results demonstrate that these adaptations are crucial to SSOLE’s effectiveness. Moreover, SSOLE achieves competitive performance across SSL benchmarks without relying on large batch sizes, memory banks, or dual-encoder architectures, making it an efficient and scalable solution for self-supervised tasks. Code is available at https://github.com/husthuaan/ssole.

Published as a conference paper at ICLR 2025
SSOLE: RETHINKING ORTHOGONAL LOW-RANK EMBEDDING FOR SELF-SUPERVISED LEARNING
Lun Huang!
Qiang Qiu
Guillermo Sapiro
Duke University
Princeton University
AAppee qqiu@purdue. edu gulllerms@@princetco..eeu
PPurdue University
lun, www.huangdduk..e.. edu
ABSTRACT
Self-supervised learning (SSL) aims to learn meaningful representations from unla- beled data. Orthogonal Low-rank Embedding (OLE) shows promise for SSL by enhancing intra-class similarity in low-rank subspace and promoting inter-class dissimilarity in a high-rank subspace, making it particularly suitable for multi- view learning tasks. However, directly applying OLE to SSL poses significant challenges: (1) the virtually infinite number of "classes" in SSL makes achiev- ing the OLE objective impractical, leading to representational collapse; and (2) low-rank constraints may fail to distinguish between positively and negatively correlated features, further undermining learning. To address these issues, we propose SSOLE (Self-Sung Super-suee Orthogonal Low-rank Embedding), a novel framework that integrates OLE principles into SSL by (1) decoupling the low-rank and high-rank enforcement to align with SSL objectives; and (2) applying low-rank constraints to feature deviations from their mean, ensuring better alignment of posi- tive pairs by accounting for the signs of cosine similarities. Our theoretical analysis and empirical results demonstrate that these adaptations crucial SSOLE's effectiveness. Moreover, SSOLE achieves competitive performance across SSL benchmarks without relying on large batch sizes, memory banks, or dual-encoder architectures, making it an efficient and scalable solution for self-supervised tasks. Code is available at httttt:t///et tttpbbb..gtuuuuuggg
thub. com cwm/husthuaan.nnn sssole
INTRODUCTION
Self- supervised learning (SSL) (Chen et al. 2020a He et al., 2020: Bardes et al. 2022) learns meaningful representations from unlabeled data by exploiting the intrinsic structure within the data, reducing the dependence on costly labeled datasets. SSL has become crucial in fields like computer vision, natural language processing, and speech recognition, enabling models to harness vast amounts of unannotated data.
Orthogonal Low-rank Embedding (OLE) (Qiu & Sapiro 2015 Lezama et al,, 2018), originally devel- oped for supervised image classification, constructs feature space where same-class samples reside low-rank subspaces, enhancing intra-class similarity, while different-class samples are orthogonal, promoting inter-class dissimilarity. OLE operates the matrix level, optimizing the nuclear norm as a surrogate for rank, making it well suited for leveraging multiple views or augmentations of data in SSL. Integrating OLE into SSL offers several benefits: (1) Its matrix-level operation is well suited for leveraging multiple views or augmentations inherent in SSL; (2) Enforcing low-rank constraints on positive pairs not only brings their representations closer but also minimizes the number factors controlling the image representation, leading to more compact features; (3) Enforcing high-rank constraints on negative pairs not only pushes their representations apart, but also prevents dimensional collapse, maximizing the representational capacity of the embedding space.
However, OLE has not been successfully applied as a standalone metric in SSL, Previous works have only partially utilized OLE's components. For example, LORAC (Wang et al., (2022) incorporates low-rank embedding as regularization term within the MoCo framework (He et al. 2020) but relies heavily contrastive loss. Methods like VICReg (Bardes et al.) 2022) and Total Coding Rate (TCR) (Yu et al.) 2020 Tong et al.] (2023) enforce orthogonality among negative pairs via soft regularization, while W-MSH (Ermolov al. 2021) employs feature whitening. They align with

---

Published as a conference paper at ICLR 2025
OLE's high-rank constraints, but neglect the low-rank embedding of positive pairs. This gap prompts the question: Why has OLE not been fully integrated into SSL?
Our investigation reveals two key challenges. First, in SSL, each instance acts its class, creating an infinite number "classes." " The OLE objective requires enforcing orthogonality between classes and minimizing intra-class ranks, but orthogonality is mathematically impossible with infinite classes a limited feature space, leading to representational collapse. Second, enforcing low-rank constraints on feature vectors can fail to distinguish between positively and negatively correlated features. The nuclear norm ignores the signs of cosine similarities, which is critical to aligning positive pairs SSL, where supervision is absent.
To address these issues, we propose SSOLE -Sperriue Orthogonal Low-rank Embedding). framework that effectively integrates OLE into SSL. Our approach involves two key strategies: (1) Decoupling low-rank and high-rank enforcement: We separately manage positive pair attraction and negative pair repulsion, eliminating the dependence of low-rank enforcement on inter-class orthogonality. We optimize both the lower and upper bounds of the contrastive loss using nuclear norms to adapt OLE for SSL. (2) Applying low-rank constraints to feature deviations: Instead of directly applying low-rank constraints to the feature vectors, we apply them to the deviations from their mean, ensuring that the nuclear norm captures the signs of cosine similarities and aligns positive pairs correctly.
Through theoretical analysis and empirical evaluations, we demonstrate that these adaptations are essential for SSOLE's success in SSL.. Our method addresses the identified challenges, achieving state- of-the-art performance on benchmark datasets. SSOLE leverages OLE's strengths while avoiding its limitations in SSL, offering an efficient solution for representation learning.
In summary, our contributions are: (1) identifying key challenges in applying OLE to SSL and offering insights into its limitations; (2) proposing SSOLE, which decouples low-rank and high-rank enforcement and applies low-rank constraints to feature deviations; (3) validating SSOLE's superior performance through theoretical and empirical analysis; and (4) opening new avenues for matrix-level operations and rank constraints in SSL, offering potential for future advancements.
2 BACKGROUND
2.1 ORTIIOGONAL LOW-RANK EMBEDDING (OLE)
Orthogonal Low-rank Embedding (OLE) (Lezama et al. 2018) was introduced as a geometric loss function to improve deep network representations by simultaneously minimizing intra-class variance and maximizing inter-class separability. Unlike traditional softmax-basee classification losses, OLE enforces that samples from the same class lie in a low-rank subspace while ensuring that different class subspaces are orthogonal. This approach encourages compact and well-separated representations in the embedding space, leading to improved generalization and robustness.
The OLE loss is defined as:
K
COLE = Σ (Xx) [X
1=1
where X, denotes the set of samples from class X is the entire data set from K classes, (XX) represents the transformation applied to the entire dataset, and || ||. is the nuclear norm, which approximates the rank of matrix. The first term minimizes the intra-class rank, while the second term maximizes the inter-class separability by encouraging orthogonality between different classes.
OLE achieves non-negative value and reaches zero when the representations of all classes are orthogonal, implying maximal inter-class separation. At the same time, intra-class samples are compressed into low-dimensional subspace, preserving intra-class similarity. The nuclear norm serves as a convex relaxation of the rank function, facilitating efficient optimization through gradient- based methods.
2

---

Published as a conference paper at ICLR 2025
3 LIMITATIONS OF OLE FOR SSL
Although OLE has been successful in supervised settings, its application to SSL poses significant challenges. These challenges stem directly from two inherent limitations of OLE, which manage- able in the presence of labels but become problematic SSL due to the lack of supervision. Firstly, in OLE, low-rank (intra-class similarity) and high-rank (inter-class dissimilarity) constraints are deeply intertwined, making it difficult to optimize them independently. Secondly, the nuclear norm used in OLE cannot differentiate between positively and negatively correlated vectors. In the following subsections, we elaborate on the resulted challenges from them.
3.1 VIRTUALLY INFINITE NUMBER OF "CLASSES" IN SSL
key challenge of applying OLE self-supervised learning (SSL) arises from the fact that, in SSL, each instance is treated as its own class. This results in virtually infinite number of "classes," making it impossible to achieve orthogonality among all class representations. The entanglement of low-rank and high-rank enforcement exacerbates this issue, the original OLE objective depends on achieving inter-class orthogonality before minimizing intra-class ranks.
In supervised settings with finite classes, OLE works by first ensuring orthogonality between different class subspaces and then minimizing the intra-class rank, leading to compact and discriminative features. However, in SSL, where orthogonality is not feasible, OLE "cheats" by reducing vector length and the angle 9 between vectors to minimize the nuclear norm, resulting in representational collapse.
To explain this, consider the model generating random representations from a d-dimensional Gaussian distribution. For samples drawn from this distribution, the expected nuclear norm of the matrix V of these samples can be approximated by the following lemma:
Lemma 3.1. Let V be an m d matrix, with >> whose rows are sampled from d-dimensional Gaussian distribution N
Iid sPo=Ia) The expected nuclear norm of V satisfies:
s (a O (m) + +0 (Vm)), + +0(m))
if mm < d, in > d.
8 (..0.00 6 md
Lemma 3.1 follows the random matrix theory, and the proof provided Appendix A. it to OLE Toss over B images, each with N views, we can approximate the OLE loss.
Extending
Theorem 3.2. If the model's s output representation conforms to N sha
=00711) then for B images,
Vd'
each with N views, where N < < BN, the OLE loss satisfies:
Lole = SVBN (v.O(VBN)+ +0 (VB).
The proof is provided in Appendix A.2
Theorem 3.2 shows that LOLE is negatively correlated with the scale and a. The analysis in Appendix A.3 also shows how and a relate the average vector length I and angle 0 between vectors. So the OLE loss decreases as or 0 becomes smaller. In SSL, where orthogonality is unachievable, OLE reduces I and 0 to minimize the loss, leading to collapsed, trivial representations. Once this collapse occurs, further training becomes difficult, as the nuclear norm is dominated by the reduction in / and 0, preventing the model from learning separable features.
Even normalization techniques in SSL, which prevent / from shrinking, do not solve the problem. The OLE loss can still minimize 6, leading to collapse. Additionally, imposing bound on the intra-class nuclear norm, as done in the original OLE, does not resolve this. If the bound is reached, the model continues to shrink both / and 0. Once the bound is reached, the model oscillates between increasing and shrinking and 0, causing instability.
This challenge highlights a fundamental limitation for OLE. While OLE performs well in supervised settings by enforcing orthogonality, struggles in SSL, where orthogonality cannot be achieved. Despite efforts like normalization and bounded nuclear norms, there is currently no effective so- lution to this issue. To address this, Section 4.1 proposes to decouple low-rank and high-rank enforcements, allowing independent optimization of them to avoid representational collapse. It also explores the connection between nuclear norm optimization and contrastive objectives to refine these enforcements.
3

---

Published as a conference paper at ICLR 2025
3.2 LOW-RANK REPRESENTATION MAY NOT BE GOOD REPRESENTATIONS
Another significant challenge arises from the nuclear norm's 's inability to distinguish between positively and negatively correlated features. In SSL, this limitation leads to misalignment between positive pairs because the nuclear norm treats aligned and anti-aligned vectors similarly, degrading the quality of learned representations. Without distinguishing between aligned and anti-aligned vectors, the nuclear norm can treat opposite directions similarly, causing misaligned positive pairs and undermining the learning process. This issue is particularly problematic SSL, where the lack of labels prevents the model from correcting these misalignment.
Consider the nuclear norm for a 2-row matrix where the rows are two unit vectors separated by an angle 0. Its nuclear norm is given by || VVIl- = /I cos 0 + vI cos This shows that the nuclear norm only depends on the magnitude of cos 0, regardless of whether the cosine is positive or negative. Whether the vectors are aligned (cos > 0) or anti-aligned (cos < 0), the nuclear norm remains unchanged. This inability to distinguish between the directions of the vectors is problematic for SSL, where positive pairs should be aligned and negative pairs should remain distinct.
More generally, this limitation can be formalized. Let V be an m-row matrix where each row is d-dimensional unit vector, and let P be an m x m diagonal matrix whose diagonal elements are either 1 or 11, which invert preserve the signs of the rows of V. The nuclear norm of PV remains unchanged, demonstrating the unitarily invariant property of the nuclear norm. Further discussion is provided in Appendix A.4
Currently, there are no effective methods to resolve this limitation in SSL. As a result, the inability of the nuclear norm to differentiate vector directions remains significant obstacle for OLE-base methods in SSL. To address this challenge, Section 4.2 proposes using deviation matrices for low-rank enforcement. This ensures alignment of positive pairs by focusing on deviations from the mean and circumventing the nuclear norm's 's insensitivity to cosine similarity signs.
4
METHOD
We present how address the key challenges of applying OLE to SSL. To handle the issue of infinite classes in SSL, we decouple low-rank and high-rank enforcement, allowing independent control of positive pair alignment and negative pair separation. We also modify the nuclear norm enforcement to account for the signs of cosine similarities, preventing representational collapse and ensuring proper alignment of feature spaces. Our framework, SSOLE (Sper-Super Orthogonal Low-rank Embedding), integrates these solutions to effectively adapt OLE for SSL tasks.
4.1 ADDRESSING CHALLENGE 1: DECOUPLING LOW-RANK AND HIGH- RANK ENFORCEMENT
To address the challenge of infinite number of "classes" " in SSL, we propose to decouple low-rank and high-rank enforcement to ensure that each term operates independently, allowing the model to enforce both alignment within positive pairs and uniformity across negative pairs more effectively.
Normalizing feature vectors: First, we alleviate some of the collapse problems by normalizing the feature vectors. By ensuring that all feature vectors have a fixed unit length, we prevent the model from shrinking vector lengths to minimize the loss artificially.
Decoupling low-rank and high-rank enforcement: To further address the challenge, we modify the original OLE objective to separate the low-rank enforcement for positive pairs from the high-rank enforcement for negative pairs. We propose the following loss function:
C
1 Σ hy (L.ZZ B
Σ hos (Z B).
where Z E [xBxNx is the representation tensor for B images in batch, each with N augmented views. The matrix Z, represents the views of the 5thh image, and Z represents the th view across all images. h2 (-) and he(-) are strictly increasing and decreasing functions respectively and used to control the nuclear norm-based based enforcement for low-rank and high-rank. The term balances the two objectives.
4

---

Published as a conference paper at ICLR 2025
This formulation decouples the two terms and allows each objective to be enforced more appropriately, reducing the entanglement between intra-class and inter-class representations that causes collapse.
However, directly applying the nuclear norm the function h1(+) h2(-) still carries the risk of collapse. To address this, we explore relationships between nuclear norms and cosine similarities, drawing inspiration from prior work (Wang & Isola 2020), which decomposes SSL objectives into alignment (for positive pairs) and uniformity (for negative pairs).
Studying the nuclear norms. To guide the choice of (-) and h2(+), develop the following theorem, which relates nuclear norms to the cosine similarities between vectors.
Theorem 4.1. For set of N d-dimensional unit vectors {u,}. where d >> N, the nuclear norm of the matrix V, containing the vectors as columns, bounded by:
V cos + VN(N-NNN -1 V1 11- cos(O),
IV 1 VN.
(0)
where cos(O) - E N (0s Ex= E of Uy is the average cosine similarity between every vectors, and cos(0) is the average absolute cosine similarity.
The lower bound is attained when the vectors can be equally grouped into cost91 groups such that within each group, the vectors are either perfectly aligned or anti-aligned, while the vectors from different groups are orthogonal. The upper bound is satisfied when the angles are identical for all vector pairs (v,, U5) where + j.
The proof is provided in Appendix A.5 Theorem shows that the nuclear norm is both lower and upper bounded by function of the average (absolute) cosine similarity between vectors. When cos(() = co) = N. both the lower and the upper bounds reach the maximum of N. In contrast, when cos ()) = cos(0)
1, both the lower and upper bounds attain the minimum of VN. These bounds allow us to design functions hi (-) and (-)...
Inverting the bounds Theorem 4.1 we can derive lower and upper bounds for the average absolute cosine similarity [0] in terms of the nuclear norm:
N
|VI, N
arcsin( 1
S cos(());
cos(O) S cos²
arcsiin
VN
|
This insight enables us to control the cosine similarities between vectors through nuclear norm minimization maximization.
Caution in Low-rank Enforcement: However, care must be taken when using these bounds for low-rank enforcement. The lower bound on (f) ignores the signs of the cosine similarities, and the upper bound has local minimum at cos(0) = O where all off-diagonal cosine similarities are N-T which is negative. This is not an issue for high-rank enforcement, as the goal is to achieve orthogonality. However, for low-rank enforcement, there may be cases where the absolute cosine similarity is maximized, but the actual cosine similarity averages to zero, resulting in misaligned vectors. Thus, we need to be cautious in designing (-) to ensure it aligns positive pairs effectively.
4.2 ADDRESSING CHALLENGE 2: LOW-RANK ENFORCEMENT VIA DEVIATION MATRICES
To address the challenge of misalignment between positive pairs, we propose enforcing low-rank constraints on a deviation matrix. The core issue lies in the nuclear norm's insensitivity to cosine similarity signs. Applying the nuclear norm directly to positive pairs' representations risks misaligning them. By subtracting the mean vector from each individual vector, we create a deviation matrix that captures how each vector deviates from the average. The rank of the deviation matrix approximates the rank of the original matrix (with at most a difference of 1), so low-rank enforcement on the deviation matrix remains valid. Additionally, deviations in opposite directions acceptable in SSL, meaning this approach circumvents the cosine similarity sign issue.
We formalize this approach with the following theorem:
5

---

Published as a conference paper at ICLR 2025
Theorem 4.2. For set of N d-dimensional unit vectors {v;}, where d > N, the nuclear norm of V V o1 where Σ V; and 1 E RN vector of ones, bounded by:
V 1 S IVIl- VN(N- VI cos(0).
cos
The lower bound reached when the vectors {w:} can be grouped into two perfectly aligned sets, and the upper bound holds when the angles are identical for all vector pairs (v,, v2) where + j.
The proof is provided in Appendix A.6 The bounds provided by Theorem 4.2 depend on cos(O), which incorporates the signs of cosine similarities. This ensures that low-rank enforeement will not confuse with anti anti-aligned vectors, unlike in the original matrix where such signs may be ignored.
Further, when cOs( reaches its maximum value of I (perfect alignment), both the lower and upper bounds become O, indicating perfectly low-rank representation. Conversely, when cos (0) reaches its minimum value of 0 (orthogonal or anti-aligned vectors), the lower bound becomes v VN, corresponding to the lowest nuclear norm of the original matrix, while the upper bound reaches VN - 1), the highest possible nuclear norm for the deviation matrix.
By inverting the bounds in Theorem 4.2 derive lower and upper bounds for 1 cos in terms of the nuclear norm of the deviation matrix:
1 ((N 1) $ -os-00(0 (o) IV
N
These bounds offer insights into the cosine similarities based on the nuclear norm of the deviation matrix. Enforcing low-rank representations applying the nuclear norm directly to the feature matrix, ensuring that low-rank enforcement does not collapse into suboptimal solutions. This approach allows SSL to maintain the benefits of low-rank enforcement while preserving meaningful feature representations.
the deviation matrix resolves the issues caused by
4.3 TRAINING OBJECTIVE
Given the improvements to low-rank enforcement, we need modify Equation to derive the training objective for SSOLE as follows:
1 N
LSSOLE Σ hi Zo. Ik,
Σ he Z.. B).
(5)
where Zo.; denotes the deviation matrix of Zo The loss LSSOLE leverages both lower and upper bounds for intra-class and inter-class cosine similarity derived from Theorem 4.1 and Theorem 4.2 These bounds provide framework for optimizing contrastive-like objective based on nuclear norms, which aligns with the goals of maximizing intra-class similarity (alignment) and inter-class dissimilarity (uniformity).
For low-rank enforcement, we derive (.).) from Equation (4). basing optimizing the lower bound of the nuclear norm. Since the lower and upper bounds are equivalent up to a factor w-..
optimizing the lower bound is sufficient to optimize both bounds. Then h (-) is given by|
7Z. hi (I, (N 1)2
(6)
For high-rank enforcement, we derive h2(-) from Equation 3}. optimizing the average of the lower
and upper bounds of average (absolute) cosine similarity from the nuclear norm. hz(-) given by:
B
B
[Z.nlle]
voE1) B-1 (7)
1
he (..ll B)
cos² carcsin)
aresin(
2(B 1)
B
The SSOLE training objective combines the advantages of OLE for SSL while optimizing both lower and upper bound for contrastive-like loss using nuclear norms. By utilizing these bounds, the model can enforce alignment and uniformity in a more efficient manner, avoiding the representational collapse seen vanilla OLE approaches.
Note that in by (-) and (n).), we need to multiply a factor of N-1 N or B-1 obtain the estimated average (absolute) cosine similarities for all vector pairs (1,; n2)) where of

---

Published as a conference paper at ICLR 2025
5
RELATED WORK
5.1
SELF-SUPERVISED LEARNING
Self-supervised learning has advanced significantly in the realm of image recognition by leveraging
various innovative techniques. Methods such as BYOL (Grill et al. 2020) and MoCo (He et al.
2020) use bootstrapping and dynamic dictionaries enhance representation learning. SimSiam (Chen & He] 2021) explores learning representations without negative pairs, while CPCv2 (Henaff) 2020) emphasizes data-efficient image recognition. The principles of alignment and uniformity on a hypersphere have been analyzed by Wang & Isola (2020). Meanwhile, the impact of view (augmentation) selection contrastive learning is investigated by Tian et al. (200)). DINO (Caron ett al., (02T)) stands out as a robust self- self-supervised method. having evolved its training protocol to achieve competitive results. MoCo v3 (Chen et al.,
2021) builds upon momentum contrast for training Vision Transformers. VicReg (Bardes et al.. 2022) introduces an approach based on variance, invariance, and covariance, while Wang et al. Wang et al. (202T) address inefficiencies in representation learning. iBOT (Zhou et al. 2022) focuses on Image BERT pre-training and RELIC v2 (Tomasev et al. p ambitiously aims to outperform supervised learning on ImageNet without labels. Furthermore, the work He et al. (202T) introduces Masked Autoencoders (MAE), scalable vision learner that benefits from the reconstruction of masked image patches. This method implicitly utilizes multiple views by treating visible and masked patches differently during the learning process. Wu et al. (2018) propose a non-parametric approach to instance discrimination.
Multi-Vies Seff-Supervised Learning (MV-SSL) has recently emerged as potent paradigm to harness the information from various augmentations or views of the same data. This approach has led significant advancements in SSL by promoting more generalized feature representations. SWAV (Caron et al. 2020) introduces a unique "swapped prediction' " task to SSL, utilizing cluster assignments as pseudo-label to encourage consistency across different augmentations or views. employs multiple views of an image to compute these assignments, promoting invariance across views and improving the learned representations. I.ORAC (Wang et al. 2022) extends the principles of MoCo by incorporating low-rank embedding as prior, which is particularly beneficial for SSL.,
leverages multiple views to enforce consistency. EMP-SSL (Tong et al., 2023) takes a different approach by generating extremely large number of patches or views from the input images, significantly reducing the training eopchs to converge.
This work is related to MV-SSL and LORAC While LORAC incorporates low-rank constraints regularization term within contrastive learning frameworks, it does not fully exploit the potential of OLE central metric. In contrast, our proposed SSOLE redefines the role of OLE in SSL by using it as an intrinsic metric for both positive alignment and negative separation. By strategically adapting OLE to address SSL-specific challenges, SSOLE fully leverages the unique properties of multi-view data. This represents a significant departure from prior approaches and unlocks the full potential of OLE in SSL.
5.2 ORTHOGONAL LOW-RANK EMBEDDING AND RELATED TECHNIQUES
The concept of learning low- lowddimensional, structured representations through low-rank constraints has been extensively studied across multiple domains, including Principal Component Analysis (PCA). Linear Discriminant Analysis (LDA) (Hastic et al. 2009), face recognition (Yang et al. 2016 Xuc
ct al. 2017
Lezama et al. 2017 Xue et al., 2019), and image classification caaatioonoannggg et al., 2013)
et al.
2014 Zhang et al. (0166) High-rank regularization has also been explored for applications such
as learning orthogonal projections deep networks (Vorontsov et al. 2017), improving recurrent network performance (Bansal et al. 2018), and capsule subspace projection (Zhang et al. 2018).
In deep learning. Orthogonal Low-Rank Embedding (QLE) (Lezama et al. 2018 extended these ideas by introducing low-rank and orthogonal constraints into supervised classification tasks. By enforcing intra-class low-rank and inter-class orthogonality, OLE achieves compact and discriminative representations. Inspired by these principles, subsequent works like LORAC (ang et al., 2022) incorporated low-rank priors into SSL, albeit as regularizations rather than direct metrics.
Further extensions of OLE include data-dependent regularizations (Zhu al.) enhance pattern discovery and prevent overfitting, and Meta-OLE (Wang et al. 2023), which adapts OLE for meta-lea
2019). which aim to
7

---

Published as a conference paper at ICLR 2025
While these works establish the utility of low-rank and high-rank constraints, they primarily focus on supervised settings. Their direct application to SSL faces unique challenges. Our proposed method, SSOLE, builds upon OLE by addressing these challenges through two key innovations. These adaptations enable SSOLE to effectively extend OLE principles to SSL..
6 EXPERIMENTS
6.1 ABLATION STUDIES
We focus on exploring effective adaptations of OLE for SSL. The ResNet-18 (He et al. (016) architecture is employed on the ImageNet100 (Deng et al. 2009) dataset. Detailed information on data augmentation and training procedures is provided in Appendix C
6.1.1 ADAPTING OLE TO SSL
We introduce a baseline method of InfoNCE-M, an extension of the InfoNCE loss adapted for MV- SSL. InfoNCE-M uses the mean of all views as the anchor for each image, computing the InfoNCE loss for each view and averaging these values. The formula for InfoNCE-M is
1 BN
e 22. m://
LIMONCEE-M
log
(8)
Σ k= esiml (..
where m, = Eソ= Zr, is the mean embedding (anchor) of all views for the i-th image, and is the temperature scaling parameter. The function sim(-.. computes the cosine similarity.
We then explore adaptations of OLE for SSL. The results of these adapta- tions are summarized in T The standard LIniONCE-X M with hyper- parameter (r = 0.2) sets a baseline with respectable Top-1 accuracy of 76.4% and Top-5 accuracy of 93.0%. Then we observe that the direct ap- plication of LOLE: faced convergence issues. When we normalize the rep- resentation vectors, the training col- lapses. This indicates inherent chal- lenges in applying OLE to SSL without suitable modifications. Decoupling the low-rank and high-rank enforcement, but without using the deviation matrix for intra-class alignment, helps to stabilize the training but has bad performance, and it showed high sensitivity to the hyperparameter X. Specifically, training tended to collapse to constants for S 2.10 and random values for 2 2.20. A temporary stabilization occurred at À 2.15, but the training loss eventually diverged after about 30 epochs. leading to SVD errors. This instability indicates the difficulties directly applying low-rank constraints to the original embedding matrix.
Table 1: Studying adaptations of OLE for SSL using various strategies. We use 5 views per image. all models trained for 100 epochs.
Objective
h.param.
Top-1
Top-5
LINONCE-M
0.2
76.4
93.0
LOLE + normalization
failed to converge
collapse
loss decoupling
2.15
43.2
71.2
enhanced low-rank (CsSOLE)
À 0.7
78.5
94.4
Table 2: Impact of À on linear probing top-1 Accuracy (%).
In contrast, the LSSOLE loss with further enhanced low-rank enforcement demonstrated remarkable insensitivity to the value of X, which was varied between 0.5 and 1.9, shown Table 2 For smaller values of À E [0.1, 0.4), the model prioritizes intra-class compactness, which can lead to under- penalized inter-class overlap, slightly degrading performance. For larger values of [2.0, 3.0], the model emphasizes inter-class separability, sometimes at the expense of intra-class consistency, resulting in over-dispersed features. Optimal performance was achieved around 0.7, with top-1 accuracy of 78.5% and top-5 accuracy of 94.4%. This relative robustness of LSSOLE highlights its suitability for SSL tasks and verifies the effectiveness of our approach adapting OLE to the SSL framework.
8

---

Published as a conference paper at ICLR 2025
6.1.2 STUDIES ON THE NUMBER OF VIEWS
Understanding the impact of the number of views on model performance is crucial in SSL. As shown in Figure 1 both models exhibit an increase in Top-1 accuracy with more views, but SSOLE consistently outperforms InfoNCE-M. SSOLE's performance notably improves up to views, after which plateaus, suggesting that the intrinsic rank of views from the same instance likely less than This plateau indicates diminishing returns beyond that point. SSOLE's consistently superior performance highlights its effectiveness in utilizing additional views, while InfoNCE-M shows more modest gains, underscoring its relative inefficiency in leveraging extra views.
Narber
Figure Comparison between InfoNCE-M and SSOLE with var- ious numbers of views.
6.2 COMPARISON ON IMAGENETI00 00
Table 3 provides a detailed comparison various methods on the ImageNet100 dataset, using ResNet- 18 the backbone. Our proposed method, SSOLE, achieves the highest Top-1 accuracy of 82.5%, outperforming established methods such as MoCo-M, SwAV, VICReg, BYOL, LORAC, and EMP-SSL. This result highlights SSOLE's superior performance, especially considering its modest computational demands.
In contrast methods like BYOL and VICReg, which require large batch sizes, SSOLE achieves higher performance with batch size of only 128. Additionally, SSOLE employs a dingle-branch encoder, while methods like MoCo, BYOL, and LO- RAC use an EMA teacher model. Lastly, unlike MoCo and LORAC, which rely on large memory banks, SSOLE avoids addi- tional memory requirements. We utilize 8 crops per image, consisting of 4 large and 4 small crops. Details our multi-crop strategy are provided in Appendix [C.1] For a fair compari- son, we train SSOLE for just 100 epochs, ensuring that the number of forward passes per iteration (#forwards) the number of epochs (#epochs) is equal or less than those of other methods.
Table 3: Comparative performance on ImageNet 00. The table includes batch size (bs), number of epochs, number crops, number of forward passes, and Top-1 accuracy (%).
Method
Aspociis
Accops
Top-1
McCo-M
Want et
100
77.0
SWAV
Cava 4t
2040
256
400)
74.0
ViCRee
2048
400)
2.
79.2
BYOL
Certl
2000
4096
400
2
80.2
LORAC
Wange ei 2022
128
100
10
78.7
EMP-SSI
100
10
300
200
78.9
INTL
81.7
SSOLE
6.3 EVALUATION ON FULL IMAGENET-IK
In this subsection, assess the performance of our proposed SSOLE method when evaluated the full ImageNet- 1k dataset. We follow similar evaluation protocol as used for ImageNet 100, with the addition of semi-supervised learning settings where only fraction of the labels are available.
6.3.1 SELF-SUPERVISED AND SEMI-SUPERVISED PERFORMANCE
Table showcases the performance of various methods on the full ImageNet dataset, with a special emphasis on the SSOLE method here proposed. The table reports both linear probing and semi- supervised learning accuracies, highlighting the efficacy of SSOLE in different learning regimes.
SSOLE demonstrates superior overall performance, particularly when compared to LORAC, which also leverages multiple views and employs low-rank embedding as a regularization prior. While LORAC achieves a respectable Top-1 accuracy of 73.2% and Top-5 accuracy of 91.6%, SSOLE surpasses it with a Top-1 accuracy of 73.9% and a Top-5 accuracy of 91.7%. Although SSOLE slightly underperforms INTL, which uses an EMA teacher, it is important to note that SSOLE does not rely on dual-branch encoders or EMA. Moreover, SSOLE significantly outperforms INTL when EMA is not applied. In semi-supervised learning, the advantage of SSOLE becomes even more pronounced. SSOLE outperforms LORAC by over 5 percentage points in the 1% labeled data setting and approximately percentage points in the 10% labeled data setting. These improvements underscore the effectiveness of integrating OLE directly as metric, rather than using it merely as a regularization term, as is the with LORAC.

---

Published as a conference paper at ICLR 2025
Table 4: Performance on full ImageNet of different methods. The table reports Top-1 and Top-5
accuracies (%) for linear probing and semi-supervised settings with I % and 10% labeled data.
Method
BS
Fpochs
#Cropp
AFoewards
Linear Probing
Top-1
Top-5
Top-1
Semi-suervised (1S)
Top-5
Semi-supervieed (10%)
Top-1
Top-5
Supervised
76.5
25.4
56.4
48.4
80.4
McCo x2
Chen
256
SimCI.R
Litur
4056
1000
2
69.3
89.0
48.3
75.5
65.6
87.8
SimSua
Fen Hi
2322
256
800
2
71.3
Bastow Twins
Piveraa 2
2021
2048
1000
2
73.2
91.0
55.0
79.2
69.7
89.3
VICRog
2048
1000
2
73.2
91.1
54,8
79.4
69.5
89.5
BYOI
4296
4
73.2
SwAY
256
10
72.7
91.5
76.1
88.7
LORAC
Waigg 2r
256
200
10
73.2
91.6
50.0
76.3
68.0
88.9
MEC
L10 ot cdeed
256
100
4
73.5
Matrist-S $5
256
400
4
73.6
INTL. (Wigg
512
73.1
55.0
80.8
69.4
89.8
(EMDY (Wong onal
256
74.3
SSOLE (Ouns)
256
200
8
73.9
91.7
55.4
79.6
70.3
90.3
6.3.2
TRANSFERRING TO OTHER DATASETS
In this section, we evaluate the adaptability and robustness of the SSOLE framework through transfer learning. We apply the fea- ture extractor trained on ImageNet and fine- tune on MS-COCO (Lin et al. 2015) for object detection and instance segmentation tasks. We also evaluate transfer learning to linear classification tasks on datasets includ-
Table 5: Transfer Learning on object detection and instance segmentation on MS-COCO.
Object Detection
Instance Segmentation
Model
AP50
AP
APTs
Apms
Apmi
AP
SimCLR MoCo v2
57.7
37.9
40.9
54.6
33.3
35.3
58.9
39.3
42.5
55.8
34.4
36.5
BYOL
57.8
37,9
40.9
54.3
33.2
35.0
SWAV
58.6
38.4
41.3
55,2
33.8
35.9
SimSiam
59.3
39.2
42.1
56.0
34.4
36.7
ing CIFARIO (Krizhevsky & Hinton 2009).
Barlow Twins
59.0
39.2
42.5
56.0
34.3
36.5
CIFARI00 (Krizhevsky & Hinton 2009).
VICReg
40.0
36.7
Aircraft (Maji et al.) 2013), DTD Cimpoi
MEC
59.8
39.8
43.2
56.3
34.7
36.8
et al. 2014). and Flowers (Nilsback & Zis-
Matrix-SSL
60.8
41.0
44.2
57.5
35.6
38.0
INTI.
61.0
41.0
44.5
57.7
35.6
37.8
serman 2008), each offering unique image
SSOLE (Ours)
61.5
41.3
44.8
58.0
35.9
38.4
content and complexity challenges. This
diverse set enables a comprehensive assessment of how SSOLE's learned representations generalize
across visual domains.
Table 5 shows that SSOLE outperforms all other methods on COCO object detection and instance segmentation, highlighting its robustness and adapt- ability: Table 6 illustrates SSOLE's superior per- formance in transfer settings compared to state-of- the-art methods like MoCov2, SwAV, and LORAC SSOLE achieves the highest accuracy on CIFAR 10, Aircraft, DTD, and Flowers, indicating its capabil-
Table 6: Transfer Learning on linear classifica- tion various datasets.
Methal
CIFARIO
CIFAR
Airerutt
DTTD
Flowers
Supervisod
90.0
73.4
47.6
68.8
89.7
MaCoo2
89.8
71.0
SWAV
90.8
73.4
72.2
LORAC
75.3
72.7
SSOLE (Ouns)
92.2
74.4
73.3
ity to capture generalizable and robust features. SSOLE's performance on CIFAR 10/100 highlights its ability handle complex small-image classifications. Its success in the fine-grained classification on the Aircraft dataset and in diverse recognition tasks on DTD and Flowers further showcases its adaptability. Overall, SSOLE's consistent effectiveness across these varied datasets attests to its versatility and efficiency as a feature extractor.
7 CONCLUSION
In this paper, we presented SSOLE, which integrates OLE into the SSL paradigm. Our method addresses two critical challenges in applying OLE SSL: the difficulty of enforcing orthogonality in the presence of an infinite number of classes, and the nuclear norm's inability to distinguish between positive and negative correlations. By decoupling low-rank and high-rank enforcement and applying low-rank constraints to feature deviations, SSOLE effectively adapts OLE for self- supervised tasks. Through extensive experiments, we demonstrated that SSOLE achieves state-of- the-art performance across linear probing, semi-supervised, and transfer learning tasks, all while maintaining computational efficiency. Notably, SSOLE achieves these results without relying on large batch sizes, memory banks, or complex architectures. SSOLE sets benchmark for integrating orthogonal low low-rank representations into SSL, opening up promising directions for future research in SSL.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Work partially supported by ONR, NSF, Simons Foundation, and gifts/awards from Google, Amazon, and Apple.
REFERENCES
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can gain more from orthogonality regu- larizations in training deep networks? Advances in Neural Information Processing Systems, 31, 2018.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variaace-ivvrrann--vvaainne regularization for self-supervised learning. In International Conference on Learning Representations, 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances Neural Information Processing Systems, 33:9912-9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. arXiv preprint 2rXY:20003.0429,, 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on compuler vision and puttern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In International conference on machine learning, pp. 3015- 3024. PMLR, 2021. ISBN 2640-3498.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, and Mohammad Gheshlaghi Azar. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 1-21221-22128 2020.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu. Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 9729-9738. 2020.
11

---

Published as a conference paper at ICLR 2025
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr DolIY ar. and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, November 2021.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In Proceedings of the 37th International Conference on Machine Learning, pp. 4182-4192 PMLR, November 2020.
Ziheng Jiang, Ping Guo, and Lihong Peng. cation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014.
low-rank coding for image classifi-
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
José Lezama, Qiang Qiu, and Guillermo Sapiro. Not afraid of the dark: Nir-vis face recognition via cross-spectral hallucination and low-rank embedding. In Proceedings of the IEEE conference 2017.
computer vision and pattern recognition, pp.
José Lezama, Qiang Qiu, Pablo Musé, and Guillermo Sapiro. Ole: Orthogonal low-rank embedding- plug and play geometric loss for deep learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8109-8115 -118, 2018.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pictro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, February 2015.
Xin Liu, Zhongdao Wang, Ya-Li Li, and Shengjin Wang. Self-supervised learning via maximum entropy coding. Advances in Neural Information Processing Systems, 35:34091-34105, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained' visual classification of aircraft. arXiv preprint 10655:13 2013.
Kanti V. Mardia and Peter E. Jupp. Directional statistics. John Wiley & Sons, 2009.
George Marsaglia. Choosing a point from the surface of a sphere. The Annals of Mathematical Statistics, 43(2):645-646, 1972.
Mervin E. Muller. A note method for generating points uniformly -dimensional spheres. Communications of the ACM, 1940) April 1959. ISSN 0001-0782, 1557-7317. doi: 10.1145/ 377939.377946.
Maria-Elene Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008
Qiang Qiu and Guillermo Sapiro. Learning transformations for clustering and classification. J. Mach. Learn Res., 16(1):187-225, 2015.
Mark Rudelson and Roman Vershynin. Smallest singular value of a random rectangular matrix. Communications on Pure and Applied Mathematics, 67712)11707-1739, December 2009. ISSN 0010-3640, 1097-0312. doi: 00.1002/cp.20294.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.05111, 2022.
Shengbang Tong. Yubei Chen, Yi Ma, and Yann Lecun. Emp-ssl: Towards sulf-supe learning in one training epoch. arXiv preprint 230i::2304.0977) 2023.
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. In International Conference on Machine Learning. pp. 3570-3578. PMLR, 2017.
12

---

Published as a conference paper at ICLR 2025
Guangrun Wang. Keze Wang, Guangcong Wang, Philip HS Torr, and Liang Lin. Solving ineffi- ciency of self-supervised representation learning. In Proceedings of the IEEE/CVF Interaational Conference Computer Vision, pp. 9505-9515.20 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929-9939. PMLR, 2020. ISBN 2640-3498.
Yu Wang. Jingyang Lin, Qi Cai, Yingwei Pan, Ting Yao. Hongyang Chao, and Tao Mei. A low rank promoting prior for unsupervised contrastive learning. IEFE Transactions on Pattern Analysis and Machine Intelligence, 45(3):2667-26811 2022. ISSN 0162-8828.
Ze Wang. Yue Lu, and Qiang Qiu. Meta-ole: Meta-learned orthogonal low-rank embedding. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5305- 5314, 2023.
Xi Weng, Yunhao Ni, Tengwei Song, Jie Luo, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan, and Lei Huang. Modulate your spectrum in self-supervised learning. In International Conference on Learning Representations, 2024.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Niannan Xue, Yannis Panagakis, and Stefanos Zafeiriou. Side information in robust principal component analysis: Algorithms and applications. Proceedings of the IEEE international conference on computer vision, pp. 4317-4325, 2017.
Niannan Xue, Jiankang Deng, Shiyang Cheng, Yannis Panagakis, and Stefanos Zafeirion. Side information for face completion: robust pea approach. IEEE transactions on pattern analysis and machine intelligence, 41(10):2349-233 2364. 2019.
Jian Yang, Lei Luo, Jianjun Qian, Ying Tai, Fanlong Zhang, and Yong Xu. Nuclear norm based matrix regression with applications to face recognition with occlusion and illumination changes. IEEE transactions on pattern analysis and machine intelligence, (((1):166-711 2016.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. In Advances in Neural Information Processing Systems, volume 33, pp. 9422-9434 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021. ISBN 2640-3498.
Liheng Zhang, Marzich Edraki, and Guo-Jun Qi. Cappronet: Deep feature learning via orthogonal projections onto capsule subspaces. Advances in neural information processing systems, 31, 2018
Yangmuzi Zhang, Zhuolin Jiang, and Larry Davis. Learning structured low-rank representations for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 676-683, 2013.
Yifan Zhang, Zhiquan Tan, Jingqin Yang, Weiran Huang, and Yang Yuan. Matrix information theory for sulf-is learning. In International Conference on Machine Learning, 2024.
Zhao Zhang, Fanzhang Li, Mingbo Zhao, Li Zhang, and Shuicheng Yan. Joint low-rank and sparse principal feature coding for enhanced robust representation and visual classification. IEEE Transactions on Image Processing, 25(6):2429-22 2443, 2016.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In International Conference on Learning Representations, 2022.
Wei Zhu, Qiang Qiu, Bao Wang, Jianfeng Lu, Guillermo Sapiro, and Ingrid Daubechies. Stop memorizing: A data-dependent regularization framework for intrinsic pattern learning. SIAM Journal on Mathematics of Data Science, 1(3):476-499 496, January 2019. ISSN 2577-0187. doi: 10.1 1171/9991 1236886.
13

---

Published as a conference paper at ICLR 2025
A PROOFS AND ANALYSIS
A. PROOF FOR LEMMA
Proof. We begin by centering the matrix V to create a zero-mean matrix V follows:
s1 vd
V
This gives rows of V sampled from N (o. T then correct for the non-centered matrix.
We first compute the nuclear norm for and
Cosine similarity of vectors.
Let 6; and di be two distinct rows of V, sampled from N(0, L these vectors is given by:
The cosine similarity between
(ūt v.,)
cos cOs().))
05
Then both
and follow uniform probability distribution the unit sphere S
(Muller
|
1959:
Marsagha
1972).
shows that 1+cos(0...) follows a Beta distribution, Beta( d-1 d+1), and 2
Mardia & Jupp 2009)
E|cos (0 O,
1
Var[cos( (enfcos(O..)) (0t.)) d'
Using Chebyshev's inequality, we bound the tail probability of cos(Or.)) for given threshold :
P ((ccos(0 Vd
C
s
C
where a constant and satisfies < C < Vd.
Expected nuclear norm for < d.
The nuclear norm of V depends on the norm of each row and cosine similarity between them.
When the rows of V are perfectly orthogonal, V achieves the maximum nuclear norm for given row norms, then V is an m x diagonal matrix where the 2th diagonal element is ह; | 2 Then for i = 1, 1,..., VVTT has eigenvalue of || V has corresponding singular value of || This determines the upper bound for the expected nuclear norm:
E |+] < m E]]
Since follows N (o, ='o²1s)
i follows the chi distribution with degrees of freedom, i.e.
so
X₂. Therefore.
Vd [||
أ|| = V2 d+)
50
r($)
Vd
Varll
أ|| = d - =
sa
Using the recurrence relation for the Gamma function and Stirling's approximation for large we can bound [[[]] v
b|]] as:
Vd vā
) = Va vd+
1 T
El|
o
S0
4d
(V
14

---

Published as a conference paper at ICLR 2025
Further,
vd d
1
(
+ (1)
Varll ū1]
d
-
O
80
2
Thus,
ا|||
+
0
(10)
d
5²02
Var[|
+
(11)
2d
d²
Therefore,
B|| o...
(12)
Now, we derive the lower bound. || VII. I.
negatively relates to the absolute cosine similarity between its rows. Applying Equation ( we derive the below probability:
m(m--) 1)
C
P
max cos coos( <= Va 2 (J,f))
=1-0
(13)
C
On the other hand,
|| positively relates to the norm of its rows.
Using Chebyshev's inequality with Equation
10 and Equation [1]. we have:
8. 8
1
1 d2
P || o
+
(14)
dd
20
Further,
>= . (1 (1-
1
[1 d2
m
1 C2
P min ||0|||
2 1
+
-
(15)
20
Combining Equation 13 and Equation 15 have: with probability at least (1 - 0 (
(1 01-0 ())
0 (z). every two distinct rows has absolute cosine similarity at most C
v7
(1 (1- va
and each row has norm at least
When every two distinct rows of V has cosine similarity of C. and each row has norm of .े
) then its nuclear norm is:
1+ (m - 1) C
+ (m - 1 Vd
sa
Vd
Vd
1 1
(m - 1)2 03)
+ (m -0 11(1-
c
C
=8
2 (m- 1)
+
+
0
Vd
d
2
d
2
=00
V7
m + 0
d
=8 -
Vd
Therefore.
[|-]] 2 1 1-0 1
(
8 a m o = 8.0 a.a.mmm----m - - (22 - 0
(16)
Combining Equation 122) and Equation [6 we obtain:
E
V [[] O (s ass..mo mmm.
(17)
15

---

Published as a conference paper at ICLR 2025
Expected nuclear norm for d.
When m >> d, V is full rank, with d singular values approximately uniformly distributed.
The singular values of V. denoted as X, for i = l,..,.., d, are the square roots of the eigenvalues of VTV.
Since the trace of V V the sum of all its eigenvalues, we have:
TT(V'V)
Σ
|
(18)
Therefore,
d d
Σ
X?
773
A S
d
0 11.
(19)
d 1--1 |, 80
Since
follows chi distribution with md degrees of freedom, i.e. Xmss
|لار a Vmd - O ( (v/L
E
d
(20)
Vmd
==1
Thus,
B||l V [1.] = E
A S 0 Vmd- 0
(21)
Vmd
1=1
On the other hand, Rudelson & Vershynin (0099) show that:
vdl, so
min
I vm vd -
() sc t>
(22)
Let /2log C, where 1 C < em Then, with probability at least - J.
|V d. Vd (Vm Vd /2log a..a (Vnd- - 0(d))
(23)
Therefore,
2 1 ( s (Vmd- o(d)) = s Vmd -ma
(24)
Combining Equation 21 and Equation 24 we obtain:
[|] o (s (s.a Vmd)
(25)
Correction for non-centered V.
Finally, for the non-centered matrix V, we have ||]| < [[ + Vmll
$17 and |
Vi ||:
>
[ ||+, |V|||= 2
When large, || is dominated by || |<: whereas when o is Il..
small, [ is dominated by Mmll
21
va
We add correction term of O (s mm) to V |l,, yielding:
8 (a O (m) + O (vm)),
if m < d. if m > d.
B[]] =
0 md + 0 (vm)
16

---

Published as a conference paper at ICLR 2025
A.2 PROOF FOR THEOREM 3.2
Proof. The OLE loss is based on the difference between intra-class compactness and inter-class separation. For each class with N augmentations, the nuclear norm of the intra-class representations follows the behavior for m & d:
E|| Vinlll-] =s (O
+0
For B classes,
the nuclear norms:
Linma 8 B (OO(N) (VN)).
For inter-class separation across all BN samples, the nuclear norm follows the behavior for m >> d:
(oo (VBNa) + + +O(BB))
B| Vanll=] |=] =
Thus, the total OLE loss is:
LOLE B (OO(N) + (OO(N)+ (VN)
(o0 (VBNd) +0 +0(VBN) (VBN) SVBB ( (0 (0VVB)) -0
-0 (vd) + +0 (0 (VB) -1)
BV (0.0 (VBN)=O
A.3 ANALYSIS OF RANDOM GAUSSIAN MATRIX
To analyze the structure of the matrix V in Lemma 0;- Each row of V is sampled from M +14 s Ia
we start by examining the length of the rows first compute the expected norm of e,.
Va
Expected length of each vector:
The squared norm of is:
||
8 la + 0
E 1|| =
Vd
Expanding this, we get:
0 |2] = 2 + 0²02
where B|]|@,1131 =² due the variance of the Gaussian distribution. Thus, the expected norm is:
B||l, = =
Cosine similarity between two vectors:
The cosine similarity between v; and vy is:
cos(Br,)) = (n,, Uz}
| 1,,l
Since vs and vy are independent, have:
Bi(v,, U,)] s²,
and the expected cosine similarity is:
x2
E(cos((r,)) s² 0²) 1
1
For large a, the cosine similarity approaches zero and approaches 2, meaning the vectors are approximately orthogonal; for small the cosine similarity approaches one and 0 approaches 0. meaning the vectors are nearly identical. Both o and control the vector length, and are positively correlated.
17

---

Published as a conference paper at ICLR 2025
A.4 DISCUSSION OF NUCLEAR NORM'S S UNITARY INVARIANCE
The nuclear norm of a matrix A, denoted as |||| is defined as the sum of its singular values. The singular values of matrix A are the square roots of the eigenvalues of A A. Therefore, for any matrix V E Rmxd, the nuclear norm is given by:
I = Σ oV()),
where o, (V) are the singular values of V, and is the rank of V.
Now, consider the matrix PV, where P is a diagonal matrix with diagonal elements =1, which flips the signs of the rows of V. To compute the nuclear norm [PV]| we need to determine its singular values. The singular values are the square roots the eigenvalues (PV) (PV):
(PV) (PV) VVP.P.
Since P is diagonal matrix with 1 entries, we have P = Im where In is the identity matrix of size m. Thus, the expression simplifies to:
(PV) = V V.
This shows that the matrix PV has the same singular values as V, because the eigenvalues of V are unchanged by the multiplication with P.
Therefore, the nuclear norm of PV is equal to the nuclear norm of V:
PV = |
A.5 PROOF FOR THEOREM
Proof. The nuclear norm of V is defined as the sum of the singular values of V. denoted T1, 21,02..... ON, so that:
|V
Σ
di
We will use the properties of the singular values establish bounds on |
First, we know that the sum of the squared singular values equal to the Frobenius norm of V, which is:
Σ - =
This is because each column of V is a unit vector, so the total squared length of the matrix is N.
We know that the square of singular values of V are eigenvalues of V V. then we have
N
Σ
ot = VV = N2 (O).
Conbining these, we have
| N
N cos (0)
Since f1(x) V is concave function, and fe(x) 2 is convex function; V IIE is negatively related to N2 cos = - f2(a?) with fixed Σ of.
To find the lower and upper bounds of V We need to get the max and min value of cos²
Minimization of (0): Using the Cauchy-Schwar inequality, have the following chain of inequalities for cos(0.,) where i *
cos2 (os)(0 2 cos((1,))* cos(O1.))
18

---

Published as a conference paper at ICLR 2025
Since cos = 1 for all i, we focus the off-diagonal elements cos(Ot.) for i + j. The value of cos)(0) minimized for fixed cos(o) when all the off-diagonal cosine similarities cos(Os,)) are identical, i.e., when the vectors form an equally spaced configuration in high-dimensional space.
For such configuration, the matrix V has one eigenvalue corresponding to the eigenvector 1 1= (1,1,,...., with eigenvalue Ncos(0). The remaining N 1 eigenvalues have the same value Therefore.
VN (N and correspond to eigenvectors orthogonal to 1.
N-1
V S VN. cos(O) V N/NN 1) V 1
Maximization of (0): We note that cos,(),,)) I cos((.,)) Equality holds when cos(( is either I or O, meaning that the vectors are either perfectly aligned, completely anti-aligned, or orthogonal to each other. This scenario occurs when the vectors can be grouped into distinct groups, such that within each group, the vectors are either aligned or anti-aligned, while vectors from different groups are orthogonal.
Using this setup, the matrix VTV will have G non-zero singular values, each corresponding to group of vectors. To maximize the nuclear norm, we apply the Souchy-Sehwan inequality to obtain:
| 2 VGN,
where G represents the number of groups.
Minimizing G: We need find the minimum possible value of G. The minimum value of G occurs when the vectors are partitioned into the fewest number of groups, while ensuring that the vectors within each group are either aligned or anti-aligned, and vectors between different groups are orthogonal. The smallest G is given by:
N 2 (0)
N
G
cos ())
Substituting this into the earlier inequality for the nuclear norm, we obtain:
||- 2 V cos(()
Combining the results from the maximum and minimum cases, we obtain the desired bounds for the nuclear norm:
V S VN. V cos(()) + VN(N- - 1) V cos(
(0)
A.6 PROOF FOR THEOREM
Proof. Let V B be matrix where the i-th column is the unit vector v; IRd. and define
= Di=1 d; as the mean vector. The deviation matrix is defined as V V- T where
E
T vector of ones.
The Frobenius norm of the deviation matrix is:
VII
Σ 10 - 0
For each vector vú, we can expand || - -6|12 as:
|/
2v07 5 + ||
19

---

Published as a conference paper at ICLR 2025
We calculate ||0|| as:
N
cos(O..)):
N
N
i=
where Oi is the angle between vectors vs and Vj. Thus, the Frobenius norm of the deviation matrix is:
| N (1 (1-c0pp cos(())
where cos(O) is the average cosine similarity.
The nuclear norm sum of the singular values of the matrix. The sum of the squared singular values equals the Frobenius norm, [V = N(1 - cos(f)).
The nuclear norm is bounded between two extremes. The lower bound occurs when the vectors can be grouped into two perfectly aligned sets. In this case, the deviation vectors are either identical or the opposite. The deviation from the mean will be minimized, and the nuclear norm satisfies:
I 1- 2 N
cos
The upper bound is attained when the angles between all vectors are identical. In this configuration, the deviation vectors are orthogonal to each other, yielding:
VI VN(N- -1) / cos(O.
cos
Therefore, the nuclear norm of the deviation matrix bounded by:
VN. V c01-cos((4) (0) - VI VN(NN 1) V 1ccs(.)
B LIMITATIONS
While this study introduces SSOLE as a robust and efficient training objective for Multi-view Self- Supervised Learning (MV-SSL), integrating Orthogonal Low-rank Embedding (OLE) concepts into SSL, does have certain limitations. Our experiments validate SSOLE's performance using ResNet- 18 and ResNet-50 ImageNet100 and the full ImageNer(1 K) dataset. However, the potential of SSOLE with more complex architectures, such as wide ResNet-50, Vision Transformers (ViT), and Swin Transformers, remains unexplored. These architectures have shown promise in SSL, achieving state-of-the-art results. Future work could explore the applicability of SSOLE with these advanced architectures. Additionally, investigating SSOLE complementary regularization technique alongside other SSL methods presents another promising avenue for future research.
C IMPLEMENTATION DETAILS
This section outlines the configurations for our experiments ablation studies, focusing on InfoNCE- M and SSOLE methods.
C.I INCORPORATION MULTIPLE CROPS
The multiple crops technique, introduced in SwAV Caron et al. (2020), draws inspiration from discussions in Wang et al. (2022) on handling outlier views and balancing low-rank and high-rank constraints. In SSL, the diversity of views during training has a significant impact on the effectiveness of models. As highlighted in Wang et al. (2022). incorporating high-variance views such as small crops requires careful consideration because they deviate from the principal components of larger views. These deviations challenge the low-rank assumptions central to SSL, necessitating a balanced approach in nulti-view learning.
20

---

Published as a conference paper at ICLR 2025
To address this, propose two-step approach to low-rank and high-rank enforcement for large and small crops. Our strategy ensures that gradients from small crops do not interfere with those from large crops, which represent the primary views used for low-rank enforcement. The steps detailed as follows:
Let Zz IR Bxxxx*x represent the feature matrix of the large crops, where N is the number of large crops, and is the feature dimension. First, we compute the deviation matrix Zr,a. by subtracting the mcan of the large crops' feature vectors:
N Z L.b,n)
ZL,b, ZL,b sg( NL
where sg(+) denotes the stop-gradient operation. The low-rank loss for the large crops is then applied to this deviation matrix as:
C h1 - (1Z,,,...
Let Zall E Bx xd represent the feature matrix for all crops, where N includes both large (L) and small (S) crops. To account for the variance introduced by small crops, calculate the deviation matrix for all crops, but the mean is replaced by the mean of the large crops:
NL
ZuI,b.: Zallb, sg(
Σ ZL,.n).
NL
To further prevent the gradients of small crops from affecting the large crops, we stop the gradient flow the large crops in this step. The low-rank loss for all crops is then computed as:
((A hn (1Zan,,; Noli
Then the low-rank loss for small crops is then computed
((S) intra
c(AIt)
sg( (t(
For high-rank enforcement, we perform the same operation on all crops, including both large and small views. Let Z.,, represent the feature matrix of the rith view. The high-rank loss is computed as:
Clnter l22 (|) LL, B),
Sntee h2 (| Is B).
To adjust the influence of the small crops, we introduce weighting factor 3. The low-rank and high-rank losses for the small crops are scaled by B, ensuring that the small crops' contribution is balanced against that of the large crops:
Cintra
(L)
intra + B ((S)
Lintral
Linter
(L) inter + B inter
(9)
This approach ensures that diverse crops are incorporated effectively while preserving the benefits of low-rank and high-rank enforcement. The factor 8 allows for fine-tuning of the influence of small crops, optimizing the balance between view diversity and stability SSL.
C.2 PRETRAINING
ImageNet100 Experiments: We employ ResNet-18 the backbone (fo) with three-layer MLP (4096-d hidden layer with ReLU, followed by normalization) as a projector, yielding a final embedding dimension of 4096. Training utilizes batch size of B 128 across 4 GPUs, SGD optimizer with a base learning rate of Ir = 2.0, and a cosine decay to 0.002. = 0.7. The experiment uses N₁. 4 full views and Nş 4 small views with B 0.6.
Full ImageNet (1K) Experiments: For the full ImageNet dataset, ResNet-50 is used as the backbone with enhanced three-layer MI.P (8192-d with ReLU and normalization) in the projector, leading
21

---

Published as a conference paper at ICLR 2025
to embedding size of 8192. The batch size is set at B 256, evenly distributed over 8 GPUs.
The SGD optimizer is used with a base learning rate of = 1.0, decaying to 0.001 following a cosine rule. À = 0.7. The experiment uses Nr full views and Ns small views with 8 = 0.6.
We adopt the "multi-crop" data augmentation strategy from SwAV, an exemplary multi-view training algorithm. This method enriches the diversity of input data, crucial for effective multi-vier self- supervised learning. During each iteration of training, we generate a combination views:
Full Views: We create Nr. full views for each image, each of size 224 x 224 pixels. The scale factor for these views varies within the range of [0.14, 1.0], ensuring a wide representation of the original images.
Small Views: Alongside full views, Ng small views of size 96 96 pixels generated, with a scale factor ranging from [0.05, 0.33]. These smaller views focus on different segments of the images, introducing further variance.
Each generated view undergoes a series of augmentation techniques to enhance model robustness:
Random Horizontal Flip: Applied with probability of 0.5 to introduce horizontal variability,
Color Distortion: This includes color jittering (brightness, contrast, saturation, and huc adjustments with respective strengths of 0.8, 0.8, 0.8, and 0.2) applied with probability of 0.8, and color dropping (conversion to grayscale) with probability of 0.2.
Gaussian Blur: Each view is subjected to Gaussian blur, having standard deviation in the range of [.0], to simulate focus variability.
Random Solarization: Applied with probability of 0.2 to further diversify the visual input.
C.3 LINEAR PROBING
Linear probing evaluates the representational quality of our SSOLE model. We outline distinet training protocols for ImageNet101 and the full ImageNet dataset.
Training Protocols:
ImageNet1000:
Batch Size: 256.
Optimizer: SGD with a momentum of 0.9, no weight decay.
Learning Rate: Base rate Ir = 0.1, cosine schedule over 100 epochs.
Full ImageNet:
Batch Size: = 2048.
Optimizer: SGD, momentum 0.9, no weight decay,
Learning Rate: Starting at Ir 0.6, reduced by 0.3 every 20 epochs, across 100 epochs.
Data Augmentation:
Training Images: Random cropping and resizing 224 224, plus random horizontal flips (probability 0.5).
Test Images: Resize to 256 256, then center crop 224 224.
Implementation Note: For the full ImageNet, we implemented SwAV based on the available code
and compared our results with those reported Wang et al. (0022 (referenced in Table
C.4 SEMI-SUPERVISED LEARNING ON IMAGENET
We conduct fine-tuning experiments with limited labeled data, specifically 1% and 10% subsets of the ImageNet dataset. These subsets are the same those used in Chen al. 2020a)..
22

---

Published as a conference paper at ICLR 2025
For the fine-tuning process, employ SGD optimizer with batch size of B 256 and momentum 0.9, without any weight decay. The fine-tuning is carried out for 20 epochs for both the 1 1% and 10% labeled datasets.
Following the learning rate scaling strategy from SwAV, we set different learning rates for the linear layers and the backbone network weights. Specifically, the linear layers' learning rates scaled up by 250 times and 20 times for the 1% and 10% tasks, respectively. We determined the optimal base learning rates for the linear layers to be 5.0 for the 1% task and 0.2 for the 10% task after conducting a search in the range of 0.01 to 10. These learning rates are then reduced by factor of 0.2 at the 12th and 16th epochs during the training period.
C.5 TRANSFER LEARNING FROM IMAGENET TO OTHER DATASETS
For transfer learning tasks, we maintain consistent image pre-processing protocols during linear classifier training and testing, as used in the linear evaluation on ImageNet. However, for CIFARIO and CIFAR 100, adjust the image size to 224 x 224 and apply random horizontal flipping to training images with probability of 0.5.
In these experiments, focus is on training linear classifiers top of the frozen feature represen- tations extracted from the pre-trained network. We employ an SGD optimizer with a batch size of B 256, momentum of 0.9, and weight decay. To determine the optimal base learning rate for each algorithm, we conduct search across seven logarithmically-span
values ranging from 0.1 to 100. Once the optimal learning rate is identified for each algorithm, we apply cosine decay rule for its annealing.
For the CIFARIO and CIFARI00 datasets, the linear classifiers are trained for a total of 30,000 iterations, In contrast, for the Aircraft, DTD, and Flowers datasets, we limit the training to 5,000 iterations.
We report t accuracies (%) achieved by each method on CIFARIO, CIFAR 100, and DTD datasets; and mean per class on Aircraft and Flowers, following practices in Wang et (2022): Grill al.) (2020)
23


Reviews:
[{"review_id": "DwRFybu4SM", "reviewer": "Reviewer_4arZ", "summary": "The paper presents orthogonal low-rank embedding for self-supervised learning (SSOLE) by decoupling low/high-rank enforcement on positive/negative pairs and low-rank enforcement via deviation matrices.", "questions": "1.What is the relationship between the three limitations, the two challenges, and the proposed method?\n\n2.How does Eq. (2) decouple the low-rank enforcement and high-rank enforcement?\n\n3.The title of the paper does not mention “multi-view learning”, but why do the authors discuss “multi-view learning” throughout the paper (especially in experiments)? Besides, since there are several multi-view self-supervised learning methods, why not compare the proposed with these works as well?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The paper is with a good clarity by providing a deep analysis on the problem when applied OLE in SSL. The two challenges to be solved are well discussed.\n2.The authors provided a detailed theoretical analysis to illustrate the research problem as well as the developed method.\n3.Sufficient experiments are performed, and the results demonstrate the work’s effectiveness.", "weaknesses": "1.The main idea of the paper of employing low/high-rank enforcement to adjust the distance between contrastive sample pairs has been proposed and discussed in several previous works of supervised learning (like LDA). As a result, the paper makes incremental contribution by extending this idea to self-supervised learning, which makes its novelty somehow limited. Besides, some related works are not discussed in this paper.\n2.The relationship between the three limitations, the two challenges, and the proposed method could be further discussed. \n3.The authors consider decoupling the low-rank enforcement and high-rank enforcement with Eq. (2), which needs more explanation to analyze how Eq. (2) achieves this aim.\n4.It is mentioned that achieves competitive performance across SSL benchmarks without relying on large batch sizes, memory banks, or dual-encoder architectures, which lacks detailed verification or discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VVhqjdd6ZV", "reviewer": "Reviewer_D3sb", "summary": "This paper primarily focuses on applying OLE to SSL and propose a novel method that integrates Orthogonal Low-rank Embedding into the Self-Supervised Learning paradigm. The authors mainly addresses two key challenges in applying OLE to SSL: the enforcement of orthogonality with an infinite number of classes and the limitations of the nuclear norm in distinguishing between positive and negative correlations. By decoupling low-rank and high-rank enforcement and applying constraints on feature deviations, SSOLE adapts OLE for self-supervised tasks. The paper demonstrates how SSOLE adapts OLE for self-supervised tasks and showcases its superior performance in various learning scenarios while maintaining computational efficiency.", "questions": "1.In the case of weakly supervised datasets, such as when the dataset contains noisy labels, does this method have adaptability?\n2.Regarding the description in the appendix A.4 that the product of diagonal matrix P and its transpose is the identity matrix, why are all the diagonal elements of the resulting matrix either 1 or -1? Could you provide a more detailed explanation?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper offers comprehensive experimentation, strengthening the validity of the presented approach.\n2.The method is described in detail, enhancing its reproducibility and understanding.\n3.In terms of experiments,  this paper evaluate the adaptability and robustness of the SSOLE framework through transfer\nlearning to various linear classification tasks and demonstrates its superior performances.", "weaknesses": "1.The writing of this paper needs to be improved.\n2.Some experiments are not sufficiently thorough, such as when evaluating the performance of the method in semi-supervised learning, the datasets used are somewhat limited, and the experimental results are not particularly striking. If additional experiments on other datasets could be conducted to demonstrate the method's effectiveness, it would be more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VKFhJxGRTl", "reviewer": "Reviewer_3eqn", "summary": "The paper points out that traditional Orthogonal Low-Rank Embedding (OLE) methods face significant challenges in self-supervised learning (SSL), mainly due to representational collapse caused by an excessively large number of classes, and the difficulty of distinguishing between positively and negatively correlated features under low-rank constraints. To address these issues, SSOLE decouples the low-rank and high-rank constraints and applies low-rank constraints to the deviation matrix of features. This approach effectively prevents representational collapse and enhances the ability to differentiate between positive and negative sample pairs. Experimental results demonstrate that SSOLE achieves excellent performance across various SSL benchmarks, showing good scalability and efficiency, especially without requiring large batch sizes and complex architectures.", "questions": "See \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Methodological Innovation: The paper introduces a low-rank bias matrix and a decoupled constraint mechanism based on the integration of OLE and SSL, addressing the issue of representation collapse that traditional methods struggle to resolve in unsupervised scenarios.\n\n2. Theoretical and Experimental Support: The effectiveness of the SSOLE framework is supported by both theoretical analysis and experimental validation, demonstrating strong performance across various datasets, particularly in settings with limited computational resources.\n\n3. Broad Applicability: In image classification tasks, SSOLE shows good generalization ability, adapting to different datasets and further enhancing the robustness of feature representation.", "weaknesses": "1. Some proofs are omitted and not specific enough, such as\n\n i) Why is the probability of $|cos(\\theta_{ij})|>1/d$ is larger than $1/d$? By what, Chebyshev inequality?\n\n ii) Some statements are as follows: \"Since the rows of  $\\tilde{V}$ are nearly orthogonal, the nuclear norm is dominated by the sum of the row norms.\" Why can it hold? The readers need more detailed explanations.\n\n2. Some proofs are unnecessary. THEOREM 3.3 states that the nuclear norm is unitarily invariant, a property that is very common in linear algebra textbooks.\n\n3. What does $\\approx$ mean? Does it mean that the equation holds with high probability, or that the values are close? If so, how close are the values? The paper needs to give a clear definition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DMQHI2k9VY", "reviewer": "Reviewer_dJ94", "summary": "This paper proposes a self-supervised orthogonal low-rank embedding (SSOLE), which integrates OLE into the SSL paradigm. It addresses two challenges in applying OLE to SSL: the difficulty of enforcing orthogonality in the presence of an infinite number of classes, and the nuclear norm’s inability to distinguish between positive and negative correlations. By decoupling low-rank and high-rank enforcement and applying low-rank constraints to feature deviations, SSOL adapts OLE for self-supervised and other tasks.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is well-written with clear motivations.\n2.\tIt is technically sound with comprehensive theoretical analysis.\n3.\tExperimental results demonstrate the effectiveness of the method.", "weaknesses": "1.\tThe parameter $\\lambda$ controls the balance between intra-class compactness and inter-class separability enforcement. It will be better to analyze its influence to the final performance.\n\n2.\tThe authors enforce intra-class low-rank property via deviation matrix instead of original feature matrix, it is also suggested to investigate its effectiveness by ablation study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)