SimO Loss: Anchor-Free Contrastive Loss for Fine-Grained Supervised Contrastive Learning

## Abstract

Abstract We introduce a novel anchor-free contrastive learning (AFCL) method leveraging our proposed Similarity-Orthogonality (SimO) loss. Our approach minimizes a semi-metric discriminative loss function that simultaneously optimizes two key objectives: reducing the distance and orthogonality between embeddings of similar inputs while maximizing these metrics for dissimilar inputs, facilitating more fine-grained contrastive learning. The AFCL method, powered by SimO loss, creates a fiber bundle topological structure in the embedding space, forming class-specific, internally cohesive yet orthogonal neighborhoods. We validate the efficacy of our method on the CIFAR-10 dataset, providing visualizations that demonstrate the impact of SimO loss on the embedding space. Our results illustrate the formation of distinct, orthogonal class neighborhoods, showcasing the method’s ability to create well-structured embeddings that balance class separation with intra-class variability. This work opens new avenues for understanding and leveraging the geometric properties of learned representations in various machine learning tasks.

## 1 Introduction

The pursuit of effective representation learning (Gidaris et al. (); Wu et al. (); Oord et al. ()) has been a cornerstone of modern machine learning, with contrastive methods emerging as particularly powerful tools in recent years. Despite significant advancements, the field of supervised contrastive learning (Khosla et al. (); Balestriero et al. ()) continues to grapple with fundamental challenges that impede the development of truly robust and interpretable models.

Our research unveils persistent challenges in embedding methods (Wen et al. (); Grill et al. (); Hjelm et al. ()), notably the lack of interpretability and inefficient utilization of embedding spaces due to dimensionality collapse (Zbontar et al. (); Jing et al. ()). Certain techniques, such as max operations in loss functions (e.g., max(0, loss)) (Gutmann & Hyvärinen ()) and triplet-based methods (Sohn (); Tian et al. ()), introduce complications like non-smoothness, which disrupt gradient flow. These approaches also suffer from biases in hand-crafted triplet selection and necessitate extensive hyperparameter tuning, thereby limiting their generalizability (Rusak et al. ()). Contrastive loss functions, including InfoNCE (Chen et al. ()), often rely on large batch sizes and negative sampling, leading to increased computational costs and instability. While recent metric learning approaches (Movshovitz-Attias et al. (); Qian et al. ()) have made strides in improving efficiency and scalability, they frequently sacrifice interpretability. This trade-off results in black-box models that offer minimal insight into the learned embedding structure.

We present SimO loss, a novel AFCL framework that addresses these long-standing issues. SimO introduces a paradigm shift in how we conceptualize and optimize embedding spaces. At its core lies a carefully crafted loss function that simultaneously optimizes Euclidean distances and orthogonality between embeddings – a departure from conventional approaches that typically focus on one or the other (Schroff et al. ()).

The key innovation of SimO is its ability to project each class into a distinct neighborhood that maintains orthogonality with respect to other class neighborhoods. This property not only enhances the explainability of the resulting embeddings but also naturally mitigates dimensionality collapse by encouraging full utilization of the embedding space. Crucially, SimO operates in a semi-metric space, a choice that allows for more flexible representations while preserving essential distance properties.

From a theoretical standpoint, SimO induces a rich topological structure in the embedding space, seamlessly blending aspects of metric spaces, manifolds, and stratified spaces. This unique structure facilitates efficient class separation while preserving nuanced intra-class relationships – a balance that has proven elusive in previous work. The semi-metric nature of our approach, allowing for controlled violations of the triangle inequality, enables more faithful representations of complex data distributions that often defy strict metric assumptions.

Our key contributions are:

- •
We propose an anchor-free pertaining method (AFCL) for supervised and semi-supervised contrastive learning
- •
We introduce SimO, an anchor-free contrastive learning loss that significantly advances the state-of-the-art in terms of embedding explainability and robustness.
- •
We provide a comprehensive theoretical analysis of the induced semi-metric embedding space, offering new insights into the topological properties of learned representations.

As we present SimO to the community, we do so with the conviction that it represents not just an incremental advance, but a fundamental reimagining of contrastive learning—one that addresses the core challenges that have long hindered progress in the field.

## 2 Related Work

### 2.1 Anchor-Based Contrastive Learning

In contrastive learning, anchor-based losses have evolved from simple pairwise comparisons to more sophisticated multi-sample approaches (Khosla et al. ()). The triplet loss (Coria et al. ()), which compares an anchor with one positive and one negative sample, has found success in applications like face recognition (Chopra et al. ()), despite its tendency towards slow convergence. Building on this foundation, the (N+1)-tuplet loss extends the concept to multiple negatives, approximating the ideal case of comparing against all classes. Further refinement led to the development of the multi-class N-pair loss, which significantly improves computational efficiency through strategic batch construction, requiring only 2N examples for N distinct (N+1)-tuplets (Sohn ()). Recent theoretical work has illuminated the connections between these various loss functions. Notably, the triplet loss can be understood as a special case of the more general contrastive loss. Moreover, the supervised contrastive loss (Khosla et al. ()), when utilizing multiple negatives, bears a close resemblance to the N-pairs loss. Nevertheless, anchor-based methods are sensitive to negative sample quality, which can lead to inefficiencies in small datasets and struggle with false negatives. It also relies heavily on effective data augmentations and large batch size, with a risk of overlooking global relationships.

### 2.2 Dimensionality Collapse in Contrastive Learning Methods

Dimensionality collapse, a significant challenge in contrastive learning, occurs when learned representations converge to a lower-dimensional subspace, thereby diminishing their discriminative power and compromising the model’s ability to capture data structure effectively (Jing & Tian ()). To address this issue, researchers have proposed several innovative strategies. The NT-Xent loss function (Chen et al. ()) implements temperature scaling to emphasize hard negatives, promoting more discriminative representations . Another approach involves the use of a nonlinear projection head, which enhances representation quality through improved hypersphere mapping (Grill et al. ()). The Barlow Twins method (Zbontar et al. ()) takes a different tack, focusing on redundancy reduction by minimizing correlations between embedding vector components through optimization of the cross-correlation matrix. Architectural innovations have also played a crucial role in combating dimensionality collapse. Methods like BYOL and SimSiam employ asymmetric architectures to prevent the model from converging to trivial solutions (Chen & He ()). The use of stop gradient in these methods ensures that the models do not converge to produce the same outputs over time. Additionally, the use of batch normalization (Ioffe ()) has been empirically shown to stabilize training and prevent such trivial convergence, although the precise mechanisms underlying its effectiveness remain an area of active research (Peng et al. ()).

### 2.3 Explainability of Contrastive Learning

The underlying mechanisms driving the effectiveness of contrastive learning remain an active area of investigation. To shed light on the learned representations, Zhu et al. () introduced attribution techniques for visualizing salient features. Cosentino et al. () explored the geometric properties of self-supervised contrastive methods. They discovered a non-trivial relationship between the encoder and the projector, and the strength of data augmentation with increasing complexity. They provided a theoretical framework for understanding how these methods learn invariant representations based on the geometry of the data manifold. Furthermore, Steck et al. () examined the implications of cosine similarity in embeddings, challenging the notion that it purely reflects similarity and suggesting that its geometric properties may influence representation learning outcomes. Wang & Liu () investigate the behavior of unsupervised contrastive loss, highlighting its hardness-aware nature and how temperature influences the treatment of hard negatives. They show that while uniformity in feature space aids separability, excessive uniformity can harm semantic structure by pushing semantically similar instances apart. Wang & Isola () identified alignment and uniformity as key properties of contrastive learning. Alignment encourages closeness between positive pairs, while uniformity ensures the even spread of representations on the hypersphere. Their work demonstrates that optimizing these properties leads to improved performance in downstream tasks and provides a theoretical framework for understanding contrastive learning’s effectiveness in representation learning. Together, these works lay the groundwork for a deeper theoretical understanding of contrastive learning, highlighting the necessity for additional investigation.

## 3 Preliminaries

### 3.1 Metric Space

A metric space is a set $X$ together with a distance function $d:X\times X\to\mathbb{R}$ (called a metric) that satisfies the following properties for all $x,y,z\in X$:

- 1.
Non-negativity: $d(x,y)\geq 0$
- 2.
Identity of indiscernibles: $d(x,y)=0$ if and only if $x=y$
- 3.
Symmetry: $d(x,y)=d(y,x)$
- 4.
Triangle inequality: $d(x,z)\leq d(x,y)+d(y,z)$

### 3.2 Semi-Metric Space

A semi-metric space is a generalization of a metric space where the triangle inequality is not required to hold. It is defined as a set $X$ with a distance function $d:X\times X\to\mathbb{R}$ that satisfies:

- 1.
Non-negativity: $d(x,y)\geq 0$
- 2.
Identity of indiscernibles: $d(x,y)=0$ if and only if $x=y$
- 3.
Symmetry: $d(x,y)=d(y,x)$

## 4 Method

### 4.1 Similarity-Orthogonality (SimO) Loss Function

We propose a novel loss function that leverages Euclidean distance and orthogonality (through the squared dot product) for learning the embedding space. This function, which we term the Similarity-Orthogonality (SimO) loss, is defined as:

$$ $\mathcal{L_{\text{SimO}}}=y\left[\frac{\sum_{i,j}d_{ij}}{\epsilon+\sum_{i,j}o_ {ij}}\right]+(1-y)\left[\frac{\sum_{i,j}o_{ij}}{\epsilon+\sum_{i,j}d_{ij}}\right]$ (1) $$

- $\forall i,j,\ i\ \neq\ j\ and\ i\leq j$ are indices of the embedding pairs within a batch

- $y$ is a binary label for the entire batch, where $y=1$ for similarity and $y=0$ for dissimilarity

- $d_{ij}=||e_{i}-e_{j}||^{2}_{2}\cdot e_{j}$ is the squared Euclidean distance between embeddings $e_{i}$ and $e_{j}$

- $o_{ij}=(e_{i}\cdot e_{j})^{2}$ is the squared dot product of embeddings $e_{i}$ and $e_{j}$

- $\epsilon$ is a small constant to prevent division by zero

SimO loss function presents a novel framework for learning embedding spaces, addressing several critical challenges in representation learning. Below, we highlight its key properties and advantages:

- •
Semi-Metric Space function: The SimO loss function operates within a semi-metric space, as formalized in the SimO Semi-Metric Space Theorem. This allows for a flexible representation of distances between embeddings, particularly useful for high-dimensional data where traditional metrics may fail to capture complex relationships (Theorem A.1).
- •
Preventing Dimensionality Collapse: The orthogonality component of the SimO loss plays a pivotal role in preventing dimensionality collapse, a phenomenon where dissimilar classes become indistinguishable in the embedding space. By encouraging orthogonal embeddings for distinct classes, SimO ensures that the learned representations remain well-separated and span diverse regions of the embedding space, preserving class distinctiveness (Theorem A.2).
- •
Mitigating the Curse of Orthogonality: Our embedding techniques are constrained by the Curse of Orthogonality, which limits the number of mutually orthogonal vectors to the dimensionality of the embedding space. SimO overcomes this limitation by leveraging orthogonality-based regularization (orthogonality leaning factor) informed by the Johnson-Lindenstrauss lemma (Theorem A.4), thus enabling more effective utilization of the available space without falling prey to orthogonality saturation (Theorem A.3).

Figure: Algorithm 1 SIMO Loss Function

### 4.2 Anchor-Free Contrastive Learning

Figure: Algorithm 2 Anchor-Free Contrastive Learning with SimO loss pseudo-implementation

We introduce a novel contrastive learning pretraining strategy (Algorithm 2) that uses the SimO loss function. For each iteration, we create a batch of $k$ images sampled from n randomly selected classes where $num_classes=batch_size//k$. We generate the embeddings using our model.

The loss computation strategy is the sum of three different operations:

- To calculate the loss over embeddings from the same class, we reshape the embeddings to $(num_classes,k,embeddings_dim)$:

$\mathcal{L}_{\text{same}}=\text{SimO_loss}(embeddings,1.0)$ SimO is applied class-wise (axis 0) to calculate the loss over similar embeddings then sum it up.

- Using the same $embeddings$ predicted, we continue to do the following:

- •
Compute the mean embedding for each of the $n_{classes}$ represented in the batch: $\mu_{i}=\frac{1}{k}\sum_{j=1}^{k}f_{\theta}(I_{j})$, where $I_{j}\in mb_{i}$
- •
Calculate the loss using these mean embeddings:
$\mathcal{L}_{mean_dissimilar}=\text{simo}({[\mu_{1},\mu_{2},...,\mu_{m}],0+
olean})$, with $olean$ is the orthogonality leaning factor

- For $\mathcal{L}_{dissimilar}$ We reshape the embeddings to $(k,num_classes,embeddings_dim)$
and then we calculate $\mathcal{L}_{dissimilar}=\text{SimO_loss}(embeddings,0.0+olean)$ where $olean$ is the orthogonality leaning factor.

This dual-batch approach allows the model to learn both intra-class compactness (through $\mathcal{L}_{\text{same}}$) and inter-class separability (through $\mathcal{L}_{mean_dissimilar}$ and $\mathcal{L}_{dissimilar}$). By operating on class means, we encourage the model to learn more robust discriminative features that generalize well across class instances with reduce the impact of negative sampling.
The overall training objective alternates between these two batch types, optimizing:
$\mathcal{L}=\mathbb{E}[\mathcal{L}_{same}]+\mathbb{E}[\mathcal{L}_{mean_
different}]+\mathbb{E}[\mathcal{L}_{dissimilar}]$
where the expectation is taken over the random sampling of batches during training.

### 4.3 Experimental Setup

Our experiments were conducted using GPU-enabled cloud computing platforms, with experiment tracking and visualization handled by Weights & Biases Biewald (). We implemented our models using JAX/Flax and TensorFlow frameworks. For our experiments, we utilized the CIFAR-10 dataset Krizhevsky & Hinton (). CIFAR-10 consists of 60,000 32x32 RGB images across 10 classes, with 50,000 for training and 10,000 for testing.

In the pretraining phase for CIFAR-10, we used an embedding dimension of 16 for the linear projection head following the ResNet encoder with layer normalization instead of batch normalization, with a batch size of 96 and 32 randomly selected images per class from 3 classes. During the linear probing phase, we fed the projection of our frozen pretrained model to a classifier head consisting of one MLP layer with 128 neurons, followed by an output layer matching the number of classes in CIFAR-10 dataset.

## 5 Results

Figure: (a)
Refer to caption: extracted/5908041/assets/Train_t_SNE_Plot_1060001.png

Figure: Figure 3: Pairwise Manifold Visualization using TSNE (Lower-Triangular Plots
Refer to caption: extracted/5908041/assets/pairwise_TSNE_PCA_Plot.png

Figure: (a) Normalized Similarity Matrix
Refer to caption: extracted/5908041/assets/Normalized_Similarity_Matrix.png

Our extensive experiments on the CIFAR-10 dataset demonstrate the effectiveness of SimO in learning discriminative and interpretable embeddings. We present a multifaceted analysis of our results, encompassing unsupervised clustering, supervised fine-tuning, and qualitative visualization of the embedding space.

To assess the transferability and discriminative capacity of our learned representations, we conducted a supervised fine-tuning experiment. We froze the SimO-trained encoder and attached a simple classifier head, fine-tuning only this newly added layer for a single epoch. This minimal fine-tuning yielded impressive results:

**Table 1: Model Performance Metrics over 1 epoch fine-tuning of a classifier head and the frozen pretrained model**
| Dataset | Model | Projection Head dim. | Train Accuracy (%) | Test Accuracy (%) |
| --- | --- | --- | --- | --- |
| Cifar10 | ResNet18 | 16 | 94 | 85 |

The rapid convergence to high accuracy with minimal fine-tuning underscores the quality and transferability of our SimO-learned representations. It’s worth noting that this performance was achieved with only 1 epoch of fine-tuning, demonstrating the efficiency of our approach.

Examination of the confusion matrix revealed that the model primarily struggles with distinguishing between the ’cat’ and ’dog’ classes. This observation aligns with our qualitative analysis of the embedding space visualizations (Figure 2, Figure 3, Figure 4). The challenge in separating these classes is not unexpected, given the visual similarities between cats and dogs, and has been observed in previous works Khosla et al. (); Zhang et al. ().

We conducted a longitudinal analysis of the embedding space evolution using t-SNE projections at various training iterations (Figure 5). This analysis revealed intriguing dynamics in the learning process:

Continual Learning Behavior: We observed a tendency towards continual learning (Figure 5), where the model appeared to focus on one class at a time. This behavior suggests that SimO naturally induces a curriculum-like learning process, potentially contributing to its effectiveness.

Persistent Challenges: The ’cat’ and ’dog’ classes remained challenging for the model from 100,000 iterations up to 1 million iterations. This persistent difficulty aligns with our quantitative error analysis and highlights an area for potential future improvements.

Progressive Separation: For training, we observed a clear trend of increasing inter-class separation and intra-class cohesion, with the exception of the aforementioned challenging classes.

These results collectively demonstrate the efficacy of SimO in learning rich, discriminative, and interpretable embeddings. The observed continual learning behavior and the challenges with visually similar classes provide insights into the learning dynamics of our approach and point to exciting directions for future research.

Figure: 5,000th

## 6 Ablation Study

Our ablation studies provide crucial insights into the effectiveness of SimO’s key components.

Orthogonality Leaning Factor When removing the orthogonality constraint from our loss function, we observed a significant degradation in performance. The model failed to learn representations for all classes, instead converging to a state where only 4 out of 10 classes from CIFAR-10 were distinguishable, with the remaining 6 classes grouped together. This result underscores the critical role of the orthogonality leaning factor in SimO, acting as a regularizer that encourages the model to utilize the full dimensionality of the embedding space and prevent the clustering of multiple classes into a single region which validate.
In our initial experiments with batch composition, we encountered an interesting phenomenon where certain classes dominated the loss function and, consequently, the embedding space. This dominance prevented the model from learning adequate representations for the other classes, resulting in unstable learning. To address this issue, we implemented a class sampling strategy inspired by techniques used in meta-learning, randomly selecting less than 50% of the total number of classes for each batch. This approach led to more balanced learning across all classes increasing stability in the learning process.

Lower Bound of Embedding Dimension To explore the lower bounds of the embedding dimension and understand the compressive capabilities of SimO, we conducted an experiment where we pretrained a ResNet18 model with a projection head outputting only 2 dimensions, maintaining the orthogonality learning component. Remarkably, we achieved a clustering accuracy of 60% on CIFAR-10 with this extreme dimensionality reduction. This result is particularly impressive given that standard contrastive learning methods typically struggle with such low dimensions, often failing to separate classes meaningfully. This demonstrates the power of SimO’s orthogonality constraint in creating discriminative embeddings even in very low-dimensional spaces, pointing to its potential in scenarios where compact representations are required.

## 7 Discussion and Limitations

In our proposed framework, the embedding space generated by the SimO loss exhibits notable geometric properties (Figure 2, Figure 3) that can be interpreted through the lenses of stratified spaces, quotient topology, and fiber bundles. Specifically, we can view the overall embedding space as a stratified space, where each stratum corresponds to a distinct class neighborhood. This structure is facilitated by the orthogonality encouraged by our loss function, promoting clear separations between classes while maintaining cohesive intra-class relationships. Furthermore, we propose considering a quotient topology in which points within the same class neighborhood are identified, simplifying the representation of the embedding space to a point for each class. This transformation not only highlights the distinctness of classes but also emphasizes their orthogonality in the learned space. Additionally, our method generates a structure reminiscent of a fiber bundle, where each fiber corresponds to a specific class and is orthogonal to other fibers. This fiber bundle-like organization allows for a rich representation of class relationships and facilitates a more interpretable understanding of the learned embeddings. Collectively, these geometric interpretations underscore the robustness and effectiveness of our SimO loss with our AFCL framework in structuring embeddings that balance class separation with interpretability requiring small batch sizes unlike other loss functions.

While SimO demonstrates significant advancements in contrastive learning, our extensive experimentation has revealed several important limitations and areas for future research.

Redefinition of Similarity Metrics: A key finding of our work is that embeddings learned through SimO no longer adhere to traditional similarity measures such as cosine similarity. This departure from conventional metrics necessitates a paradigm shift in how we evaluate similarity in the embedding space. Our SimO loss itself emerges as the most appropriate measure of similarity or dissimilarity between embeddings. This also presents challenges for integration with existing systems and methods that rely on cosine similarity. Future work should focus on developing efficient computational methods for this new similarity metric and exploring its theoretical properties.

Sensitivity to Data Biases: Our method’s ability to learn fine-grained representations comes with increased sensitivity to biases present in the training data. This is particularly evident in the case of background biases in object recognition tasks. For instance, our model struggled to separate the neighborhoods of Dog and Cat classes even though it learned from the 120,000th iteration to the 1 millionth iteration, despite having learned most other classes effectively. This sensitivity necessitates robust data augmentation techniques to mitigate the impact of such biases. While this requirement for strong augmentation can be seen as a limitation, it also highlights SimO’s potential for detecting and quantifying dataset biases, which could be valuable for improving dataset quality and fairness in machine learning models.

The Orthogonality Learning Factor: The performance of SimO is notably influenced by the orthogonality learning factor, a hyperparameter that balances the trade-off between similarity and orthogonality objectives. Finding the optimal value for this factor presents a challenge we term ”the curse of orthogonality.” Too low a factor leads to insufficient separation between class neighborhoods, while too high a factor can result in overly rigid embeddings that fail to capture intra-class variations. Our experiments show that this factor often needs to be tuned specifically for each dataset and task, which can be computationally expensive. Developing adaptive methods for automatically adjusting this factor during training represents an important direction for future research.

Computational Complexity: While not unique to SimO, the computational requirements for optimizing orthogonality in high-dimensional spaces are substantial. This can limit the applicability of our method to very large datasets or in resource-constrained environments. Future work should explore approximation techniques or more efficient optimization strategies to address this limitation.

Despite these limitations, we believe that SimO represents a significant step forward in contrastive learning. The challenges identified here open up exciting new avenues for research in representation learning, similarity metrics, and bias mitigation in machine learning models. Addressing these limitations will not only improve SimO but also deepen our understanding of the fundamental principles underlying effective representation learning.

## 8 Conclusion

Our AFCL method introduces the SimO loss function as a novel approach to contrastive learning, effectively addressing several critical challenges related to embedding space utilization and interoperability. By optimizing both the similarity and orthogonality of embeddings, SimO prevents dimensionality collapse and ensures that class representations remain distinct, even in lower dimensions, requiring smaller batch sizes and embedding dimensions.

Our experimental results on the CIFAR-10 dataset demonstrate the efficacy of SimO in generating structured and discriminative embeddings with minimal computational overhead. Notably, our method achieves impressive test accuracy as early as the first epoch. Although there are limitations, such as sensitivity to data biases and dependence on specific hyperparameters, SimO paves the way for future advancements in enhancing contrastive learning techniques and managing embedding spaces more effectively.

## Acknowledgment

Google AI/ML Developer Programs team supported this work by providing Google Cloud Credit.

## Reproducibility Statement

We provide detailed proof for all the lemmas and theorems in the Appendices. Code will be shared publicly after publication.

## Appendix A Appendix

###### Theorem A.1 (SimO semi-metric space) .

Let $([0,1]^{n},d^{\prime})$ and $([0,1]^{n},d^{\prime\prime})$ be two spaces where $[0,1]^{n}$ is the n-dimensional unit hypercube and $d^{\prime}$ and $d^{\prime\prime}$ are defined as:

$$ $d^{\prime}(e_{i},e_{j})=\frac{d_{ij}}{\epsilon+o_{ij}}=\frac{|e_{i}-e_{j}|^{2} }{\epsilon+(e_{i}\cdot e_{j})^{2}}$ $$

$$ $d^{\prime\prime}(e_{i},e_{j})=\frac{o_{ij}}{\epsilon+d_{ij}}=\frac{(e_{i}\cdot e _{j})^{2}}{\epsilon+|e_{i}-e_{j}|^{2}}$ $$

where $e_{i},e_{j}\in[0,1]^{n}$, $d_{ij}=|e_{i}-e_{j}|^{2}$ is the squared Euclidean distance, and $o_{ij}=(e_{i}\cdot e_{j})^{2}$ is the squared dot product.

Then $([0,1]^{n},d^{\prime})$ and $([0,1]^{n},d^{\prime\prime})$ are a semimetric space.

###### Proof.

We will prove that $d^{\prime}$ satisfies the following properties (1, 2, and 3 but not 4):

1. Non-negativity: $d^{\prime}(e_{i},e_{j})\geq 0$

2. Identity of indiscernibles: $d^{\prime}(e_{i},e_{j})=0\iff e_{i}=e_{j}$

3. Symmetry: $d^{\prime}(e_{i},e_{j})=d^{\prime}(e_{j},e_{i})$

4. The triangle inequality property: $d^{\prime}(e_{i},e_{k})\leq d^{\prime}(e_{i},e_{j})+d^{\prime}(e_{j},e_{k})$ for all $e_{i},e_{j},e_{k}\in[0,1]^{n}$

1. Non-negativity:
$d_{ij}=|e_{i}-e_{j}|^{2}\geq 0$ (as it’s a squared term)
$o_{ij}=(e_{i}\cdot e_{j})^{2}\geq 0$ (as $e_{i},e_{j}\in[0,1]^{n}$, their dot product is always positive)
Therefore, $d^{\prime}(e_{i},e_{j})=\frac{d_{ij}}{\epsilon+o_{ij}}\geq 0$ and $d^{\prime\prime}(e_{i},e_{j})=\frac{o_{ij}}{\epsilon+d_{ij}}\geq 0$

2. Identity of indiscernibles:
($\Rightarrow$) If $e_{i}=e_{j}$, then $d_{ij}=0$ and $o_{ij}=|e_{i}|^{4}>0$, so $d^{\prime}(e_{i},e_{j})=0$
($\Leftarrow$) If $d^{\prime}(e_{i},e_{j})=0$, then $d_{ij}=0$ (as $o_{ij}\geq 0$), which implies $e_{i}=e_{j}$. (same for $d^{\prime\prime}$)

3. Symmetry:

$d^{\prime}(e_{i},e_{j})=\frac{|e_{i}-e_{j}|^{2}}{\epsilon+(e_{i}\cdot e_{j})^{
2}}=\frac{|e_{j}-e_{i}|^{2}}{\epsilon+(e_{j}\cdot e_{i})^{2}}=d^{\prime}(e_{j}
,e_{i})$

$d^{\prime\prime}(e_{i},e_{j})=\frac{(e_{j}\cdot e_{i})^{2}}{\epsilon+|e_{i}-e_
{j}|^{2}}=\frac{(e_{j}\cdot e_{i})^{2}}{\epsilon+|e_{j}-e_{i}|^{2}}=d^{\prime
\prime}(e_{j},e_{i})$

4. For the triangle inequality to hold, we must have:

$$ $d^{\prime}(e_{i},e_{j})\leq d^{\prime}(e_{i},e_{k})+d^{\prime}(e_{j},e_{k})$ $$

$$ $d^{\prime\prime}(e_{i},e_{j})\leq d^{\prime\prime}(e_{i},e_{k})+d^{\prime \prime}(e_{j},e_{k})$ $$

Let $e_{i}$ = (1,0), $e_{j}$ = (0,1), $e_{k}$ = (1,1), and $\epsilon$ very small.
Then $d^{\prime}(e_{i},e_{k})=1/\epsilon$, while $d^{\prime}(e_{i},e_{j})+d^{\prime}(e_{j},e_{k})=2$.

Hence equality doesn’t hold for d’.

Therefore, we conclude that the function $d^{\prime}$ does not satisfy the triangle inequality for all vectors in $[0,1]^{n}$, and thus $([0,1]^{n},d^{\prime})$ is not a metric space. It remains a semimetric space, satisfying non-negativity, the identity of indiscernibles, and symmetry, but not the triangle inequality.

∎

###### Theorem A.2 (SimO Dimentionality Collapse Prevention) .

The loss function $\mathcal{L_{\text{SimO}}}$ prevents dimensionality collapse for dissimilar (negative) classes through its orthogonality term.

###### Proof.

Let $\mathcal{E}=\{e_{1},\ldots,e_{n}\}$ be a set of embeddings in $\mathbb{R}^{d}$, where $n$ is the batch size and $d$ is the embedding dimension.

The loss function $\mathcal{L_{\text{SimO}}}$ is defined as:

$$ $\mathcal{L_{\text{SimO}}}=y\left[\frac{\sum_{i,j}d_{ij}}{\epsilon+\sum_{i,j}o_ {ij}}\right]+(1-y)\left[\frac{\sum_{i,j}o_{ij}}{\epsilon+\sum_{i,j}d_{ij}}\right]$ (2) $$

where:

- •
$i,j\in\{1,\ldots,n\}$, $i\neq j$, $i<j$
- •
$y\in\{0,1\}$ is a binary label for the entire batch (1 for similarity, 0 for dissimilarity)
- •
$d_{ij}=\|e_{i}-e_{j}\|^{2}=\|e_{i}\|^{2}+\|e_{j}\|^{2}-2e_{i}\cdot e_{j}$ is the squared Euclidean distance
- •
$o_{ij}=(e_{i}\cdot e_{j})^{2}$ is the squared dot product
- •
$\epsilon>0$ is a small constant to prevent division by zero

We proceed by analyzing the behavior of $\mathcal{L_{\text{SimO}}}$ for dissimilar pairs and showing how it encourages properties that prevent dimensionality collapse.

- 1.
For dissimilar pairs ($y=0$), $\mathcal{L_{\text{SimO}}}$ reduces to:
$\mathcal{L_{\text{SimO}}}=\frac{\sum_{i,j}o_{ij}}{\epsilon+\sum_{i,j}d_{ij}}$
(3)
- 2.
To minimize this loss, we must minimize $\sum_{i,j}o_{ij}$ and maximize $\sum_{i,j}d_{ij}$.
- 3.
We first prove two key lemmas:
Lemma A.2.1.
Minimizing $\sum_{i,j}o_{ij}$ encourages orthogonality between dissimilar embeddings.
Proof.
•
$\forall i,j:o_{ij}=(e_{i}\cdot e_{j})^{2}\geq 0$
•
Minimizing $\sum_{i,j}o_{ij}$ implies minimizing each $o_{ij}$
•
Minimizing $(e_{i}\cdot e_{j})^{2}$ pushes $e_{i}\cdot e_{j}\to 0$
•
$e_{i}\cdot e_{j}=0\iff e_{i}\perp e_{j}$
Therefore, minimizing $\sum_{i,j}o_{ij}$ encourages orthogonality between all pairs of dissimilar embeddings.
∎
Lemma A.2.2.
Maximizing $\sum_{i,j}d_{ij}$ encourages dissimilar embeddings to be far apart in the embedding space.
Proof.
•
$\forall i,j:d_{ij}=\|e_{i}-e_{j}\|^{2}\geq 0$
•
Maximizing $\sum_{i,j}d_{ij}$ implies maximizing each $d_{ij}$
•
Maximizing $\|e_{i}-e_{j}\|^{2}$ increases the Euclidean distance between $e_{i}$ and $e_{j}$
Therefore, maximizing $\sum_{i,j}d_{ij}$ pushes dissimilar embeddings farther apart in the embedding space.
∎
- 4.
Now, we show how these lemmas prevent dimensionality collapse:
(a)
By Lemma 1, $\mathcal{L_{\text{SimO}}}$ encourages orthogonality between dissimilar embeddings.
•
Orthogonal vectors span different dimensions in the embedding space.
•
This prevents dissimilar embeddings from aligning along the same dimensions.
(b)
By Lemma 2, $\mathcal{L_{\text{SimO}}}$ simultaneously pushes dissimilar embeddings farther apart.
•
This reinforces the distinctiveness of dissimilar embeddings.
•
It prevents dissimilar embeddings from collapsing to nearby points in the embedding space.
(c)
The combination of (a) and (b) ensures that:
•
Dissimilar embeddings maintain their distinctiveness.
•
They occupy different regions and directions in the embedding space.
•
The effective dimensionality of the embedding space is preserved for dissimilar classes.
- 5.
Formally, let $\{e_{i}\}_{i=1}^{k}$ be a set of dissimilar embeddings. The loss function ensures:
•
$\forall i\neq j:e_{i}\cdot e_{j}\approx 0$ (orthogonality)
•
$\forall i\neq j:\|e_{i}-e_{j}\|^{2}$ is maximized (separation)
These conditions directly contradict the definition of dimensionality collapse, where dissimilar embeddings would become very similar or identical.

∎

###### Theorem A.3 (Curse of Orthogonality) .

In an $n$-dimensional embedding space, the number of classes that can be represented with mutually orthogonal embeddings is at most $n$.

###### Proof.

Let $\mathbb{R}^{n}$ be an $n$-dimensional embedding space. Consider a set of $k$ vectors $\{\mathbf{a}_{1},\mathbf{a}_{2},\ldots,\mathbf{a}_{k}\}$ in this space, where each vector represents the mean embedding of a distinct class.

Assume that these vectors are mutually orthogonal, i.e.,

$$ $\mathbf{a}_{i}\cdot\mathbf{a}_{j}=0\quad\text{for all}\quad 1\leq i,j\leq k \quad\text{and}\quad i\neq j.$ (4) $$

In $\mathbb{R}^{n}$, the maximum number of mutually orthogonal vectors is equal to the dimension of the space, which is $n$. This is because any set of mutually orthogonal vectors must also be linearly independent, and the maximum number of linearly independent vectors in $\mathbb{R}^{n}$ is $n$.

Therefore, the maximum number of mutually orthogonal embeddings, and thus the maximum number of classes that can be represented with such embeddings, is $n$.

Hence, $k\leq n.$

∎

###### Theorem A.1 (SimO semi-metric space) .

###### Proof.

###### Theorem A.2 (SimO Dimentionality Collapse Prevention) .

###### Proof.

###### Lemma A.2.1 .

###### Proof.

###### Lemma A.2.2 .

###### Proof.

###### Theorem A.3 (Curse of Orthogonality) .

###### Proof.

### A.1 Johnson–Lindenstrauss Lemma

###### Lemma A.3.1 (Johnson–Lindenstrauss Lemma) .

$0<\epsilon<1$, and let $X$ be a set of $n$ points in $\mathbb{R}^{d}$. There exists a mapping $f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{k}$ with $k=O\left(\frac{\log n}{\epsilon^{2}}\right)$ such that for all $x,y\in X$:

$$ $(1-\epsilon)\|x-y\|^{2}\leq\|f(x)-f(y)\|^{2}\leq(1+\epsilon)\|x-y\|^{2}$ $$

###### Theorem A.4 (Johnson-Lindenstrauss Lemma Addressing the Curse of Orthogonality) .

Given $k>n$ vectors in $\mathbb{R}^{n}$, there exists a projection into a higher-dimensional space $\mathbb{R}^{m}$ where $m=O(\frac{\log k}{\epsilon^{2}})$, such that the projected vectors are ”nearly orthogonal”, effectively overcoming the limitation imposed by the Curse of Orthogonality.

###### Proof.

Let $\{v_{1},v_{2},\ldots,v_{k}\}$ be $k$ vectors in $\mathbb{R}^{n}$, where $k>n$.

1) By the Johnson-Lindenstrauss lemma, there exists a mapping $f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$, where $m=O(\frac{\log k}{\epsilon^{2}})$, such that for all $i,j\in\{1,\ldots,k\}$:

$$ $(1-\epsilon)\|v_{i}-v_{j}\|^{2}\leq\|f(v_{i})-f(v_{j})\|^{2}\leq(1+\epsilon)\| v_{i}-v_{j}\|^{2}$ $$

2) Consider the dot product of two projected vectors $f(v_{i})$ and $f(v_{j})$ for $i\neq j$:

$$ $f(v_{i})\cdot f(v_{j})=\frac{1}{2}(\|f(v_{i})\|^{2}+\|f(v_{j})\|^{2}-\|f(v_{i} )-f(v_{j})\|^{2})$ $$

3) Using the upper bound from the JL lemma:

$$ $f(v_{i})\cdot f(v_{j})\leq\frac{1}{2}(\|f(v_{i})\|^{2}+\|f(v_{j})\|^{2}-(1- \epsilon)\|v_{i}-v_{j}\|^{2})$ $$

4) If $v_{i}$ and $v_{j}$ were originally orthogonal, then $\|v_{i}-v_{j}\|^{2}=\|v_{i}\|^{2}+\|v_{j}\|^{2}$. Substituting this:

$$ $f(v_{i})\cdot f(v_{j})\leq\frac{1}{2}(\|f(v_{i})\|^{2}+\|f(v_{j})\|^{2}-(1- \epsilon)(\|v_{i}\|^{2}+\|v_{j}\|^{2}))$ $$

5) The JL lemma also ensures that for each vector:

$$ $(1-\epsilon)\|v_{i}\|^{2}\leq\|f(v_{i})\|^{2}\leq(1+\epsilon)\|v_{i}\|^{2}$ $$

6) Using the upper bound from (5):

$$ $f(v_{i})\cdot f(v_{j})\leq\frac{1}{2}((1+\epsilon)(\|v_{i}\|^{2}+\|v_{j}\|^{2} )-(1-\epsilon)(\|v_{i}\|^{2}+\|v_{j}\|^{2}))$ $$

7) Simplifying:

$$ $f(v_{i})\cdot f(v_{j})\leq\epsilon(\|v_{i}\|^{2}+\|v_{j}\|^{2})$ $$

8) This shows that the dot product of any two projected vectors is bounded by a small value proportional to $\epsilon$, which can be made arbitrarily small by increasing $m$.

Therefore, while we cannot have more than $n$ strictly orthogonal vectors in $\mathbb{R}^{n}$, we can project $k>n$ vectors into $\mathbb{R}^{m}$ where they are ”nearly orthogonal”. The dot product of any two projected vectors is bounded by $\epsilon(\|v_{i}\|^{2}+\|v_{j}\|^{2})$, which approaches zero as $\epsilon\rightarrow 0$.
∎

###### Corollary A.4.1 .

The projection provided by the Johnson-Lindenstrauss lemma allows for the representation of more classes than the original embedding dimension while maintaining near-orthogonality, effectively addressing the Curse of Orthogonality.

###### Lemma A.3.1 (Johnson–Lindenstrauss Lemma) .

###### Theorem A.4 (Johnson-Lindenstrauss Lemma Addressing the Curse of Orthogonality) .

###### Proof.

###### Corollary A.4.1 .

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: I do not suggest to add significantly more experiments in the rebuttal. I'd be happy to discuss how the proposed method can be extend to semi-supervised setting and the orthogonal questions in 5 and 6.
soundness: 2
strengths: The proposed loss is simple and easy to implement. The paper also provides proofs to show SimO loss is a semi-metric and can help mitigate dimensionality collapse. Paper shows a number of t-SNE figures to showcase the learned embeddings.
summary: The paper proposes 1 new metric learning loss SimO for metric learning, which not only considering the Euclidean distance, but also orthogonality (squared dot product). The paper provides proof that SimO loss is a semi-metric and can help mitigate dimensionality collapse.
weaknesses: The major concern I have on the paper is for the evaluation. 

1. The paper only shows the result of the proposed SimO loss on toy CIFAR10 dataset under the supervised setting. It’s not very clear if the conclusion can be extended to other large-scale datasets or not. For example, adding examples on ImageNet will be helpful. 

2. There are no other baseline losses (e.g. cross-entropy) are compared in the evaluation. It’s not very clear how the proposed loss compared to SOTA. It would be helpful to compare to conventional losses such as cross-entropy and triplet

3. The paper claimed that the loss can be used for semi-supervised learning. However, no evaluation has been conducted on semi-supervised learning. For conducting semi-supervised learning experiment, it's helpful to say masking the label of x% of the samples in a supervised dataset say ImageNet,

Some other issues include:

4. At line 193, the formulation of Euclidean distance is wrong. It shouldn't have $\cdot e_j$ to the end.  

5. At line 194, it’s not clear why squared dot product can be used to measure the orthogonality, it’s not only determined by the angle between 2 embeddings but also the norm. It would be helpful if the author can provide a clarification why using squared dot product instead of normalized cross product (normalize both vectors to unit-norm before calculating the cross product.) 

6. Overall, it’s also not very clear why different categories need to be orthogonal or near orthogonal. For example, if there are only 2 classes, the optimal solution should be opposite, instead of orthogonal. It would be helpful if the author can provide some justification when the number of categories is significantly higher than the dimension of the feature (where orthogonal is not possible).

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: 1. There is a number “1060000” in the titles of Figure 2(a) and Figure 2(b). What does that refer to?

See the Weaknesses section above
soundness: 1
strengths: The paper aims to tackle the issue of dimensionality collapse, which is an important challenge in existing representation learning approaches.
summary: The paper introduces a novel anchor-free contrastive learning (AFCL) method utilizing a new loss function called Similarity-Orthogonality (SimO) loss. This approach optimizes both the distance and orthogonality of embeddings for better class separation and intra-class cohesion within a semi-metric space. Key contributions include comprehensive theoretical analysis of the embedding space, and visual validations on the CIFAR-10 dataset demonstrating structured and separable class neighborhoods.
weaknesses: 1. The writing of the work is extremely poor and unengaging. For instance

- The abstract fails to establish a clear motivation or provide adequate background, making it difficult for readers to understand the context and importance of the research.

- Section 3 reads as a collection of definitions and terminology without meaningful connections or explanations.

- In Section 4.1, the definition of the loss function lacks intuition or insights, making it challenging for readers to grasp the underlying rationale. 

- Mathematical notations are unclean for instance towards the end of Section 4.2

- Figure titles and extremely small to understand. 

- Experiments section is very hard to understand

The paper, as a whole, gives the impression of disjointed content assembled without a coherent progression.
 
2. Experimental results aren’t rigorous enough

- First, the experiments conducted are only on a small “toy” dataset in the regime of self-supervised learning

- There are no baselines in the paper

3. Most citations of the popular representation learning approaches are missing from the introduction. For example, SimSiam, BYOL, Barlow Twins, etc. Most citations seem to be quite old and outdated. 

4. The proposal method is unclear. On a hypersphere, which is a common setting in self-supervised learning approaches, minimizing the Euclidian distance is equivalent to maximizing the cosine similarity between embeddings. Further, it is unclear what $y$ is. The authors mentioned that “y is a binary label for the entire batch”. What does this mean? There are no insights into the approach.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: What is the purpose of evaluating 1-epoch fine-tuning accuracy? What is the highest accuracy that the proposed method can achieve?
soundness: 2
strengths: - The properties and advantages of the proposed loss function is supported by theoretical analysis.
- The empirical results demonstrate the potential of the proposed method to balance class separation with intra-class variability.
summary: This paper propose a new contrastive learning method, which reduces distance and orthogonality between embeddings of similar inputs and maximizing distance and orthogonality for dissimilar inputs. It conducted experiments on CIFAR-10 to demonstrate the performance of the proposed method. Visualization shows that the method has the ability to balance class separation with intra-class variability.
weaknesses: My primary concern with this paper is the lack of comparisons with baseline methods in contrastive and representation learning, such as those mentioned in the related work section. Additionally, the paper only reports a 1-epoch fine-tuning accuracy on CIFAR-10, which is not a standard metric in contrastive and representation learning evaluations. These factors make it difficult to compare the proposed method with prior works and to assess whether it offers any improvement over existing methods.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 3
questions: Would this outperform any other method for supervised contrastive learning, anchor-free or otherwise? What is the proof?

How are the embeddings learned by SimO different from those learned by other methods, in practice? What evidence can show this?
soundness: 1
strengths: [S1] The writing of the paper is clear and easy to follow.

[S2] The theorems and lemmas and well-organized and easy to understand.
summary: The paper proposes SimO, a novel loss for supervised contrastive learning. The loss is anchor-free in the sense that all samples in a batch are compared. Distances and dot products are computed between all samples in a batch, such that dissimilar samples are encouraged to have distant, orthogonal representations, while the opposite is encouraged for similar samples. The supervision (class label) is used to determine whether a pair is similar. Results, in the form of both accuracy and dimensionality analysis, are presented for a ResNet-18 trained on CIFAR-10.
weaknesses: [W1] The empirical results are not thorough. No baselines are provided. Only 1 dataset, and 1 backbone are considered. No other methods are compared against. The only quantitative results in the paper are a single line in a single table.

[W2] The empirical results reflect poorly on the proposed method. The single line that is present has accuracies that, for CIFAR-10, are not good.

[W3] I am not convinced that the orthogonality is maximized in practice (Theorem A.2), since the loss term that would maximize orthogonality is entangled with maximizing the distance. In practice, the distance may be sufficiently large that the orthogonality is not so different compared to other works. Detailed analysis and comparison with baselines would have been very helpful to verify that both terms are handled well (orthogonality and distance) in practice.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Is there a reason why the proposed loss is applied in the supervised contrastive learning domain, given that contrastive loss is predominantly used in self-supervised learning?
soundness: 2
strengths: The paper proposed a novel loss that incorporates orthogonality in embedding space to enhance class separation and interpretability. The proposed method shows its effectiveness within a limited CIFAR-10 benchmark.
summary: The paper introduces SimO Loss, an anchor-free contrastive learning loss for supervised learning, designed to create class-specific neighborhoods in embedding spaces that maintain intra-class cohesion while achieving inter-class orthogonality. By leveraging Euclidean distance and orthogonality measures, the method aims to address dimensionality collapse and improve interpretability in contrastive learning models.
weaknesses: While the approach shows potential in the CIFAR-10 dataset, the limited scope and high computational requirements raise questions about its broader applicability.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See first point weaknesses.
soundness: 3
strengths: 1. The paper is easy to read.
2. The paper validate SimO by embedding visualizations which clearly shows formation of semantic structure on CIFAR10.
summary: The paper introduce SimO loss, an anchor-free contrastive loss objective which optimizes both orthogonality and euclidean distance of the embeddings. The authors argue that this is a departure from existing approaches which typically have focused on either orthogonality or euclidean distance. The paper pertain SimO loss to supervised and semi-supervised settings. The paper include visualizations to demonstrate that SimO can find embeddings with clear separations between classes. The authors argue that SimO advances the state-of-the-art in terms of embedding explainability of contrastive learning methods.
weaknesses: 1. Model performance metrics over 1 epoch of fine-tuning on frozen backbone features is not standard practice. It would be more convincing if you ran this in a standard setting to SimCLR[1] and SupCon[2], and give a proper reference to benchmarks of related work. 
2. The 2 year old contrastive method t-SimCNE can give interpretable cluster visualizations with semantic structure [3], unsupervised without class labels and in raw 2d space on both CIFAR10 and CIFAR100.  SimO pertain to supervised settings which limits the applicability. 





[1] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning,pp. 1597–1607. PMLR, 2020a.

[2] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673, 2020.

[3] Jan Niklas Bohm, Philipp Berens, and Dmitry Kobak. Unsupervised visualization of image datasets using contrastive learning. In International Conference on Learning Representations, 2023.