One-Class SVM-guided Negative Sampling for Enhanced Contrastive Learning

Venue: NLDL_org_2025_Conference
Published: 2024-09-07 12:00:24

Abstract: Recent studies on contrastive learning have emphasized carefully sampling and mixing negative samples.
This study introduces a novel and improved approach for generating synthetic negatives.
We propose a new method using One-Class Support Vector Machine (OCSVM) to guide in the selection process before mixing named as  **Mixing OCSVM negatives (MiOC)**.
Our results show that our approach creates more meaningful embeddings, which lead to better classification performance.
We implement our method using publicly available datasets (Imagenet100, Cifar10, Cifar100, Cinic10, and STL10). We observed that MiOC exhibit favorable performance compared to state-of-the-art methods across these datasets.
 By presenting a novel approach, this study emphasizes the exploration of alternative mixing techniques that expand the sampling space beyond the conventional confines of hard negatives produced by the ranking of the dot product.

Main paper: 
 introduction: 
 Empirical evidence has demonstrated that unsupervised contrastive learning is a highly effective technique for acquiring high-quality features, making optimal use of a vast unlabeled dataset. It has gained considerable popularity as a pre-training strategy for a range of tasks such as classification, segmentation, and generative modeling like in [1] [2] [3] . Recent studies indicate that contrastive learning yields better performance than supervised learning [4, 5] . The core concept of contrastive learning is to bring similar features closer together in the feature space while highlighting the differences between dissimilar features. In this context, an "anchor or query" image embedding is intended to share similarities with a "positive or key" image embedding, while it is designed to be distinct from the "negative" image embedding ensuring a clear separation. The selection process for positive and negative samples plays a crucial role in this domain, prompting continuous investigation into diverse methodologies. Momentum Contrast, or MoCo [6] , is presented as a state-of-the-art baseline method in this paper utilizing two encoders: one for query and one for key. Visual illustration of the contrastive learning pipeline. In MoCov2 [5] the embeddings in the memory buffer/queue are used as negatives.Instead of backpropagation, the key encoder's pa-047 rameters are updated using a momentum-based 048 method from the query encoder as shown in Fig- 049 ure 1. This causes the key encoder's parameters to 050 change slowly, ensuring more consistent and stable 051 representations of the negative samples. A dynamic 052 dictionary of encoded data samples is constructed, 053 functioning as a queue of negatives for contrastive 054 learning. Typically, the positive pairs consist of dif-055 ferent augmentations of the same image, whereas 056 the negatives are sourced from distinct images. This 057 paper investigates approaches to identify optimal 058 negatives that could be interpolated and added to 059 the existing queue to enhance the contrastive perfor-060 mance. There have been considerable efforts in iden-061 tifying hard negative samples that are closely related 062 to the query and hence harder to distinguish [7] [8] [9] , 063 however, there has been a lack of research dedicated 064 to exploring different types of negative samples that 065 are preferable for mixing. Focusing only on using 066 hard negative samples for mixing can have a few 067 issues:• Hard negatives might not encapsulate the 069 broader patterns inherent in the data [10] . The 070 synthetic negatives should possess the capacity 071 to be non-redundant, in order to construct a 072 more resilient representation.• When engaging in the process of mixing, it is 074 essential to construct harder negatives that are 075 in proximity to the query [9] . Additionally, 076 there should be a focus on accentuating the captured from different perspectives or modalities to learn more robust and generalizable representations. Momentum Contrast (MoCo) [6] was another approach that was proposed, which utilized a memory buffer as a queue to store negative samples and updated the weights of one of the encoders through momentum averaging, ensuring that the feature space does not exhibit significant disparities. Enhancement has been made to MoCo by several methods like MoCov2 [5] , Relational Self Supervised Learning (ReSSL) [14] and Similarity Contrastive Estimation (SCE) [15] . A method described in [16] emphasized the importance of focusing on only the top 5% of the hardest negative samples to achieve optimal models. Additionally, the authors found that the most challenging 0.1% of negative samples are unnecessary and can hinder the training process in some cases, as they often consisted of pseudo-negative samples. There are some works like Student-t distribution with a neighbor consistency constraint(TNCC) and contrastive learning loss based on the Student-t distribution (CLT) [17] who introduced a novel loss that emphasizes prioritizing weak negatives over hard negatives. Alternative techniques have also been explored, such as [18, 19] which do not rely on negative samples. 
mixup: 
 Several mixing approaches have enhanced the robustness of the learning process. MixCo [20] is based on the principle of understanding the relative similarity between representations, indicated how much the mixed images retain the characteristics of the original samples. Another method, iMix [21] , involved mixing images in a controlled manner, challenging the learning model to disentangle and identify the individual components of the mixed images. MoCHI [11] is another approach that generates two groups of synthetic negative samples. The first group z k z k z k z k z k z k z k z k z q z q z q z q z q z q z q z q z -z - z - z - z - z - z - z - z - z - z - z - z - z - z -z - z - z - z - z - z - z - z - z - z - z - z - z - z - z - z -Step 1:Step 2: Inside the Hyperspherez - z - z -z -z - z - z - z - z q z q z q z q x k = β k * (z q i ) + (1 -β k ) * (z - j ) z q ∈ P z - ∈ O Figure 3. Illustration of our approach to create the synthetic set So: With every incoming batch, Step 1, OCSVM is trained on query z q and key z k belonging to a batch of embeddings to build the surrounding hypersphere. InStep 2, the inlier negative embeddings z -are randomly chosen and interpolated with a randomly chosen z q . Here, P represents the set containing all z q in a batch, and O is the set of z -located within the OCSVM hypersphere (i.e., z q ∈ P and z -∈ O).β is a hyperparameter which is randomly chosen between [0, 0.5]. (Recommended to view in color)is created by mixing hard negatives, while the second group is created by mixing hard negatives with the anchor. Another approach, called SynCo [22] , introduced six strategies for generating diverse synthetic hard negatives in real-time.3 Method 
principles of one class svm (ocsvm): 
 OCSVM can be considered as a class density estimation problem. These algorithms are widely employed in anomaly detection and outlier detection.OCSVM detects the smallest possible hyper-sphere that encompasses all the points belonging to a specific class [23, 24] . It can alternatively be viewed as a margin separator from the origin. The hypersphere is characterized by its center, c, and radius, r. The optimization problem can be expressed as follows: EQUATIONsubject to ∥Φ (x i ) -c∥ 2 ≤ r 2 + ζ i for all i = 1,EQUATIONwhere w is a normal vector to the hyperplane, b 181 is the bias term, and ρ is the threshold. Here f (s i ) 182 can have one of the three ranges of values:183 • f (s i ) > 0: s i is inside the decision boundary. 184 • f (s i ) < 0: s i is outside the decision boundary. 185 • f (s i ) = 0: s i is on the decision boundary.The function f (s i ) will be used to sample hard neg-187 atives, which are instances located near the query. 188We propose to construct additional synthetic nega-190 tives (inspired by MoCHI [11] ) by the linear interpo-191 lation of a randomly chosen query and a randomly 192 chosen negative as shown in Equation 3.EQUATIONHere, β ranges from 0 to 0.5, interpolating a nega-195 tive embedding z - j with a query z q i . Two synthetic 196 groups of negatives S n and S o are created as shown 197 in Figure 4 . Each group consist of a number of 198 synthetic negatives of type x k from Equation 3. In 199 the case of S n , the negative is randomly chosen 200 from the queue as described in Equation 4and then 201 interpolated with a randomly chosen query, where queue comprises of K samples in the nega-204 tive memory buffer. s i is the i th negative sample. img: image from the loader, f q and f k: encoder networks for query and key, C: embedding dimension, queue: dictionary as a queue of K keys (C × K), t: temperature, O: set of inlier ocsvm negatives, f (s i , hypersphere): returns orientation of s i OCSVM: One-Class SVM 1: for each image in loader do 2:EQUATIONimg 1 ← aug(img) #augmented img 3:img 2 ← aug(img) # another augmented image 4:EQUATION#Obtaining the hypersphere parameters 7: hypersphere = OCSVM(cat(z q , z k ))# Finding samples inside the hypersphereO ← {s i , ∀i ∈ queue|f (s i , hypersphere) > 0}#First set of synthetic negatives 9: These hard-negatives are identified from the inlier 209 negatives located inside the hypersphere that encom-210 passes a batch of query z q and key z k embeddings 211 as shown in Figure 3 . We denote the set O for these 212 hard-negatives as in Equation 5.Sn ← {x k = β k z q i + (1 -β k )z - j | z - j ∈ queue} #Second set of synthetic negatives 10: So ← {x k = β k z q i + (1 -β k )z - j | z - j ∈ O} #ConcatenateEQUATIONTo summarize a batch of (query (z q ) + key (z k )) embeddings are used to train a high dimensional OCSVM hypersphere. Subsequently, we search for the negative embeddings that fall within the bounds of the hypersphere to create the set O by utilizing the decision function outlined in Equation 2. We use the InfoNCE loss as mentioned in MoCo [6] with our modification of the synthetic negatives as shown in Algorithm 1. 
experiments: 
 We conducted reproducible experiments on Ima-genet100, a subset of Imagenet1k [25] . Moreover, we conducted supplementary experiments to evaluate the overall performance of pre-training models under standard conditions, utilizing smaller datasets for linear evaluation.The training was conducted using a single Tesla A100. The images were resized to 224×224 and subjected to MoCov2 [5] augmentations. The pretraining and linear classification was done on the training set, while the results were reported on the validation set. The linear evaluation stage was conducted three times to show the standard deviation. We employed a MoCov2 [5] setup with a pre-training learning rate of 0.03 and a linear warm-up scheduler spanning ten epochs during which only S n negatives were generated. This allowed MiOC to include a certain number of samples within the hypersphere. Subsequently, a cosine scheduler was employed, and the pre-training process was conducted for 200 epochs using a ResNet50, which was trained from scratch.The embedding dimension and batch size were kept at 128. During the linear evaluation phase, we fixed the encoder, appended a linear layer on top, and conducted training for 60 epochs with a learning rate of 10 (as in [11] ), employing a multistep scheduler with a factor of 0.1 at [30, 40, 50 ] epochs.The results for the linear evaluation on the Ima-genet100 dataset are presented in CLT [17] 68.17 MoCo [6] MoCo + iMix [21] 74.2 CMC [13] 75.7 CMC + iMix [21] 75.9 MiOC demonstrated the best k-NN score and Top1 % Accuracy , while SCE [15] outperformed MoCoV2 [6] and MoCHI [11] in both metrics.SynCo [22] with a shorter pretraining time had lower k-NN and no significant improvement of Top1 % Accuracy than in MoCov2 [5] . We incorporated an expanded queue of 17K in MoCov2 [5] . Our findings demonstrated that the augmented queue length was not the primary factor contributing to the enhanced performance in MiOC. The computational load in MiOC mainly involved fitting the OCSVM and classifying points (inside or outside the hypersphere).This process is moderately resource-intensive when handling a small number of points, such as a batch of (z q + z k ). Despite the increased computation, it remains faster than MoCHI [11] .Assessing performance on smaller datasets provides insights into the model's capacity to generalize to new data. A strong performance on a small dataset implies that the model has acquired useful representations applicable across various tasks and datasets.We employed the pre-trained models trained on Ima-genet100 for linear evaluation on four datasets-(Ci-far10 [26] , Cifar100 [26] , STN10 [27] , Cinic10 [28] ).The initial three datasets are widely recognized as benchmark datasets, whereas Cinic10 [28] is a newly introduced dataset designed to serve as an intermediary between Cifar10 [26] and Imagenet [25] . The images were resized to 224×224 for Cifar10 [26] , Ci-far100 [26] , Cinic10 [28] , and 96×96 for STL10 [27] .We used a learning rate of 3 with a batch size of 128 and trained for 100 epochs with a multistep scheduler with a factor of 0. results for linear evaluation on the smaller datasets 301 in Table 2 . MoCHI [11] outperformed MoCov2 [5] in 302 both STL10 [27] and Cinic10 [28] , whereas SCE [15] 303 showed a slight improvement over MoCov2 [5] in 304 Cifar10 [26] , and Cinic10 [28] , although the differ-305 ence was not significant. MiOC demonstrated supe-306 rior performance relative to all other models which 307 clearly shows the benefit of our sampling and mixing 308 strategy of negative embeddings. This insight sheds 309 light on the importance of negative sample diversity 310 and suggests that future research could explore more 311 nuanced approaches to refine model performance fur-312 ther. Figure 5 exhibits the visualization of the ten (a) MoCov2 [5] (b) MoCHI [11] (c) MiOC MoCHI [11] which is based on creating new sets of negative embeddings, i.e., s k and s ′ k , by linear interpolation. (Equations using the same naming convention as in [11] )s k = sk ∥s k ∥ 2, where sk =α k n i + (1 -α k ) n j (6)Here, α represents a random variable ranging from 0 to 1. The variables n i and n j denote randomly selected hard negatives from the set N, which comprises hard negatives obtained by ranking the negative sample's dot product with a query. An additional set of more challenging negatives, denoted as s ′ k , is generated using a similar method.s ′ k = s′ k s′ k 2 , where s′ k = β k q i + (1 -β k ) n j (7)Here, β ranges from 0 to 0.5, interpolating the hard negative embedding n j with query q i . The authors represent each model as [N, s, s ′ ], where N represents the number of hard negatives from which s synthetic hard negatives and s ′ synthetic harder negatives are derived. Although it was noticed that a higher value of N could lead to improved outcomes, the sorting of negatives was found to increase processing time. For each query they created the synthetic negatives which inturn increased the pretraining time and the effective queue-size. As the pretraining time increased to upto greater than 100 hours, we modified MoCHI [11] to perform interpolation on randomly chosen query and generate a total of (s + s ′ ) synthetic negatives. Though this method is not comparable with the original MoCHI [11] method, it is closer to our method and shows the importance of sampling with OCSVM, hence we used it in our experiments.We experimented with various settings for the different hyperparameter configurations for the Ima-genet100 dataset. First, we conducted experiments with the OCSVM hyperparameters, including nu, gamma, and kernel, which significantly affect the hypersphere. We conducted the pre-training using a larger queue size of 65K to compare the pre-training time more efficiently. 4, though instead of mixing it with a randomly chosen query, we mix 2 negatives belonging to this set as in Equation 6. Here, we observe that the combination of S n and S o works the best and gives an advantage over MoCov2 [5] .We conducted further experiments wherein we restricted the number of samples per class to ranges between 10-1000 images for the Cifar10 [26] and Cinic10 [28] datasets. This approach is particularly useful in real-world situations where labeled data can be scarce or expensive to obtain. Linear evaluation with few images enables practitioners to use limited labeled data resources efficiently. in practical applications faced with data scarcity. We recognize the importance of the runtime for 653 MiOC. We fit OCSVM to z k +z q embeddings, which, 654 with our batch size, amounts to 128 + 128 = 256. 655Ideally, the algorithm can scale effectively up to a  
 
 Appendix:  


Decision: This paper presents MiOC, a novel method that leverages One-Class SVM (OCSVM) to improve contrastive learning through the generation of challenging synthetic negative samples. The approach is promising, showing competitive performance against other methods on several benchmark datasets. However, the paper would benefit from clearer articulation of its methodology and its placement within the existing literature. Additionally, more extensive comparisons to alternative sampling strategies and a deeper examination of the SVM’s computational costs would strengthen the study’s claims of efficiency and scalability.

While the approach shows potential, these improvements would enhance its accessibility and confirm the scalability of MiOC. Overall, the paper contributes a fresh perspective on negative sampling for contrastive learning, though more comprehensive validation is needed to maximise its impact.

Reviews: review: title: title: Good read, sound approach, a few things could still be improved
paper_summary: paper_summary: The authors present MiOC, an approach for generating synthetic negatives to improve classification performance. They describe the method and evaluate MiOC using various data sets. The results show that it performs better than existing approaches.
main_review: main_review: I already accepted the paper in my initial review, so my rating remains the same.
strength_weakness: strength_weakness: The paper does not have a dedicated section on related work. I suppose this is due to the fact that little research has been done in this area yet, but at the same time, this makes it hard to assess the paper's contribution. If there's anything the authors can do to improve on this, they should do it, as it would improve the paper's scientific quality! (e.g. not only compare quantitatively with existing approaches but also qualitatively or on a conceptual level).

Some remarks:

1 Introduction: The term "they" is used two times: "They utilise two encoders" and "They construct a dynamic dictionary". The authors should make it clear to what or whom 'they' refers in these cases.

1 Introduction: "[...] in the Figure 2" should be "[...] in Figure 2"

4 Experiments: The text switches back and forth between past and present tense (especially at the beginning). The authors should decide on a tense and then stick to it.

Appendix: I did not really understand how the appendix contributes to the paper. The only reference to it is in the caption of Table 1. Is the appendix really necessary? In particular, since MoCHI has already been described in a previous work [11]? The authors should make this clear and perhaps remove redundant information that has already been published.
title: title: Interesting method, but a few concerns on originality and importance
paper_summary: paper_summary: The paper proposes to use one-class support-vector machine  (OCSVM) to guide the selection process for generating synthetic negatives for contrastive learning. The authors base their analysis on the idea of MoCHI
of interpolation between the randomly chosen query and a randomly chosen negative.
main_review: main_review: I've checked the authors rebuttal, as well as discussion, and I think the authors addressed my comments and the comments from other reviewers. Therefore I suggest acceptance.
strength_weakness: strength_weakness: - Originality: in essence, the paper proposes a combination of two well-known approaches, MoCHI and OCSVM
- Importance: it seems to me that the proposed solution only marginally improves the performance, with the experimental results comparing the solution exclusively with MoCHI

Questions:
To expand upon the importance part, I suggest the following improvements:
- In this work, the authors focus their efforts on comparing the proposed method with MoCHI; it may be important to also present the comparison with other strategies for synthetic negatives generation, for example MixCo and iMix.
- It is also interesting that the results with MoCov2 + MoCHI do not improve upon MoCov2. Does it mean that the impact of MoCHI in this setting is actually negative? What would  be the reason for this, and could the same also be a problem for the proposed, closely linked, method? 
- Should Davies-Bouldin score and Calinski-Harabasz scores be given with the confidence interval? The same comment applies to Table 2.
- The text around Eq 5-6 could be given as an algorithm to facilitate reading
- Figure 1’s samples of hard negatives do not (obviously) look convincingly better than the dot-product ones. I wonder if the authors could clarify 
- (Minor question) Figure 2 comes before Figure 1, the authors need to fix it
- Given that the authors seem to be focus on OCSVM, I wonder if the alternative anomaly detection techniques could be used (e.g., Nizan & Tal, 2024; Guille-Escuret et al (2024)), and what would be their benefit?

Nizan & Tal, k-NNN: Nearest Neighbors of Neighbors for Anomaly Detection, WACV workshops, 2024
Guille-Escuret, C., Rodriguez, P., Vazquez, D., Mitliagkas, I., & Monteiro, J. (2024). CADet: Fully Self-Supervised Out-Of-Distribution Detection With Contrastive Learning. Advances in Neural Information Processing Systems, 36.