Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration
Authors: Yu Wang
Jingyang Lin
Jingjing Zou
Yingwei Pan
Ting Yao
Tao Mei
Venue: NeurIPS 2021 Poster (2021)
Keywords: Contrastive Learning
Self-supervised Learning
Unsupervised Learning

Abstract: Our work reveals a structured shortcoming of the existing mainstream self-supervised learning methods. Whereas self-supervised learning frameworks usually take the prevailing perfect instance level invariance hypothesis for granted, we carefully investigate the pitfalls behind. Particularly, we argue that the existing augmentation pipeline for generating multiple positive views naturally introduces out-of-distribution (OOD) samples that undermine the learning of the downstream tasks. Generating diverse positive augmentations on the input does not always pay off in benefiting downstream tasks. To overcome this inherent deficiency, we introduce a lightweight latent variable model UOTA, targeting the view sampling issue for self-supervised learning. UOTA adaptively searches for the most important sampling region to produce views, and provides viable choice for outlier-robust self-supervised learning approaches. Our method directly generalizes to many mainstream self-supervised learning approaches, regardless of the loss's nature contrastive or not. We empirically show UOTA's advantage over the state-of-the-art self-supervised paradigms with evident margin, which well justifies the existence of the OOD sample issue embedded in the existing approaches. Especially, we theoretically prove that the merits of the proposal boil down to guaranteed estimator variance and bias reduction. Code is available: https://github.com/ssl-codelab/uota.

Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration
Yu Wang'
Jingyang Lin
Jingjing Zou Yingwei Pan
Ting Yao
Tao Mei!
JD Al Research, Beijing, China
Sun Yat-sen University, Guangzhou, China University of Califoriia, San Diego, USA
{feather1014, gung.linjyl (panyu. uste, tingyao ustc)@gmaill com, tmei@jd. com
Linjyn @gmail com, j2zouduced. edu lig.iiinjyamililll....
Abstract
Our work reveals structured shortcoming of the existing mainstream self- supervised learning methods. Whereas self-supervised learning frameworks usually take the prevailing perfect instance level invariance hypothesis for granted, we carefully investigate the pitfalls behind. Particularly, we argue that the existing augmentation pipeline for generating multiple positive views naturally introduces out-of-distribution (OOD) samples that undermine the learning the downstream tasks. Generating diverse positive augmentations on the input does not always pay off benefiting downstream tasks. To overcome this inherent deficiency, we intro- duce lightweight latent variable model UOTA, targeting the view sampling issue for self-supervised learning. UOTA adaptively searches for the most important sampling region produce views, and provides viable choice for outlier-robust self-supervised learning approaches. Our method directly generalizes to many mainstream supf-suue learning approaches, regardless of the loss's s nature contrastive or not. We empirically show UOTA's advantage over the state-of-the-art self-supervised paradigms with evident margin, which well justifies the existence of the OOD sample issue embedded in the existing approaches. Especially, we theoretically prove that the merits of the proposal boil down guaranteed estimator variance and bias reduction. Code is available: http://gghhbb..oo//slcoobbbbbbbl
Introduction
Self-supervised learning is an increasingly appealing direction in learning effective deep repre- sentations. In the regime of computer vision, natural language processing and machine learning tasks, multiple milestones under such self-supervised learning frameworks have been established
48 49 The premise here is that self-supervised learning methods usually generate multiple views and assume one view be predictive of another.
12 1g 32 33
39
43
47
So far, the most prevailing assumption is to force views from the same instance invariant in the feature space
18 33 47 While certainly natural consider generating diverse augmentations to spread the feature distribution and force consistency in between views, we reveal structured shortcoming of such popular augmentation pipeline: excessive distortions applied on original image would produce samples that deviate drastically in the semantics. We find these produced views have semantically deviated from the original instance and thus behave like out of distribution (OOD) samples. The model therefore fail to generalize in some certain parameter region due to the interference of the OOD samples.
We support our claim by empirically verifying the non-negligible detrimental impact of the OOD noise on downstream task performances. Particularly, we conduct extensive experiments and show that importance sampling technique and the associated analysis can help effectively differentiate noises. The proposed new model suppresses the effect of noisy samples and efficiently improve the downstream task performance over broad spectrum of state-of-the-art self-supervised learning
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

approaches. This indicates the evident failure of baseline model in some parameter space owing to the OOD noise. Our contribution in this paper are summarized as follows: 1. We present the first formal analysis of OOD issue under the currently prevailing self-supervised learning framework. 2. We propose a lightweight latent variable model UOTA that is able to differentiate noise from original instance's ss semantics. The new model does not introduce extra computational complexity whereas it can efficiently suppress the influence of OOD samples. In contrast to existing SSL paradigms, the proposal method is able to automatically balance the bias-variance trade-off in the mean squared error (MSE) of the deep estimator: We do desire large data variance through strong augmentations, on the other hand, we should be careful not introduce extra estimator bias through extremely large data distortions. We argue that in the context of self- sel--uueeeeeee learning. the hidden OOD samples and such bias-variance trade-off have a clear and non non- non-neglig impact that should be attended to.
2 Related Work
Self-supervised learning. Without access any data annotations, self-supervised learning (SSL) approaches impose additional constraints or hypothesis among distinct views of the original input data
52]. Recently, mainstream approaches require that different views out of the same instance should be predicative of each others. Such line of research can be roughly classified into several sub-directions. The first family is inspired from the seminal NCE approach [16] and capitalizes on contrastive loss [5] 7
2773 [3] 14
32
B3
39
41
44
These approaches usually rely on the construction of both the positive and negative views of a single instance. In these work, positive samples usually are augmented views of the same reference instance: Negative samples are defined as any views from a distinct instance other than that reference instance. Contrastive loss is then expected to learn useful semanties when forced to distinguish between positive and negative views on each instance. The second line of research further simplify the contrastive assumption, which only imposes invariant constraints between paired positive samples [9] 15 46 47 negative samples.
33
in the absence of the
Bias correction for contrastive learning. Existing literature prioritize the sampling bias issue for negative samples, while positive samples are usually considered clean. In [TO), debiased contrastive learning corrects the sampling bias in the negative samples in order to reduce the effect of false negatives. In 24 37,, harder negatives are created for training in order to improve the learning. The work in 30 is the only work we are aware of, that touches the noise issue in positive pairs. But [30] is motivated exclusively for observed caption-framee misalignments issues in narrated videos, and the loss is tailored for applications that are based on contrastive loss only. In contrast, we reveal that OOD noise actually exists in a broader coverage of self- cuf-viuee learning frameworks and is influencing the downstream tasks with nen-negligible strength.
OOD detection and noisy label problem. Outlier detection is an important problem in machine learning. A popular and intuitive framework for detecting OOD inputs is to train a generative model on training data and use that to detect OOD inputs at test time. For instance. in [34), likelihood ratio method is proposed to correct for the background statistics between inliers and outliers. In the meanwhile, for supervised learning tasks, robust training in the presence of noisy labels is also a practical and challenging issue. In (50), a generalized cross entropy loss is used to mitigate the label noise issue. In 233. Mentor-net capitalizes on tailored network architecture to down- down-weight weight the samples likely to be associated with noisy labels. All of these work were targeting conventional supervised learning approaches, whereas the noise under the SSL framework is rarely explored.
Comparison to existing works. We unveil that the intrinsic positive sample noises exist broadly behind various mainstream SSL frameworks. To both justify and mitigate the issue, we propose latent variable framework that is capable improving the SSI. optimization via automatic Unsupervised OuTlier Arbitration (UOTA). We are also aware of contemporary work in [5l] that attempts to change the sampling distribution via assigning sample weights to each positive sample. But motivation in [5I] is completely opposite to ours, as they oversample the region whenever a positive sample turns out to be remote to its query, while they still assume a perfect invariance on positive keys as critical supporting prerequisite. In (26), Mahalanobis distance- based confidence score is defined to modify sampling distribution for supervised learning tasks. Note our work differentiates from [26] in many aspects: we analyze the OOD issue particularly for unsupervised SSL frameworks, whereas (26) strongly hinges on data labels to compute confidence. We model the associated sampling ratio as a latent variable model which is exclusively suitable for the SSL problem, and supported by rigorous analysis. Our model also naturally jettisons the need of any threshold as used in (26).
2

---

Figure 1: Illustration of OOD samples sampled from distribution p. Figures ranked according to descent order its associated wi.j. The biggest image in the red box is the original instance input æ;,
3 Method
In this section, we reveal that OOD noise is inherently existing in positive view sampling pipelines for a broad range of mainstream SSL approaches. Most importantly, we showcase that these OOD samples are highly structured, and disrupt the downstream tasks with a non-negligible strength. We borrow basic analytic tool from importance sampling to investigate the source of OOD problem specifically for SSL frameworks. We identify an important bias-variance trade-off behind the sampling distribution options for positive views (augmentations), and we analyze the impact such bias-vane variance trade-off on the estimator MSE (mean squared error) via rigorous justification. Our model UOTA is motivated to adaptively balance the sampling distribution trade-off via a novel latent variable model, and the proposed method is agnostic to the form of SSL loss. We show that the model is able to effectively suppress the OOD sampling noise for positive samples.
3.1 Problem Formulation of SSL
Mainstream SSL approaches, especially contrastive learning methods, often rely series trans- formation functions & = g( n&y) to produce multiple "views" Ti, of input instance æ; The (T&x 1,...; K} is the support of "augmentation nuisances variable" n, representing group of transformations such as flipping, scaling, cropping, or other augmentations acting on æ; via function g. Subscript k indexes all possible augmentations, i indicates the i-th instance (each instance corresponds to specific image input). Without labeling information, SSL relies on the invariance assumption so that deep feature extractor 2 f(
k 0) is at least insensitive to such group of augmentation nuisances T& w.r.t. the network parameters O. Existing work [5] 7 (8) 18 33] show that, while the deep extractor f 0) trained learn the shared semantics from distorted views Ti,k- such pretrained network parameters & effectively benefit further downstream tasks.
SSL approaches based such pairwised instance invariance assumptions can be generally summa- rized as penalizing some form of inconsistency between paired of features Zi,j Zi,k:
Lo (n.; m=s)) T.(zi.j, 2.,k)
where j. indexes over the augmented views on instance x;, Without loss of generality, we expand
the support of random variable to pairwised (n2,, 72+) j, k 1..... K} to incorporate pairs of augmentation actions. Loss (x (nz, n2)) represents such invariance loss under the pair of augmentation actions (Nz, n.). Here, the generic form of loss function L(zi,j; Zi,&) imposes the desired cross-view invariance assumptions between ži,j and z₁,6 for the i-th instance. Take for instance, T((zi.) Zi,A) can be defined as contrastive loss in MoCo [18, clustered loss as in SwAV (5), or simply &2 norm as in BYOL [15]. Our idcalized goal then to sample from independent latent generative processes H; Ps and p* such that 2i,k f 0) fg( n2.), 0) captures the same semantics as from original x;.
Assume we have access to such oracle distribution p", and with equal sampling probability for æ., the objective function is:
N ΣΣΣ
E Ea [La n1]
LZzs z1,)) (n).
(2)
N
8=1 3.k
Optimizing Eq. parameters 0.
then corresponds to learning an M-estimator 122 with regard to network
3.2 Sampling Distribution p": a Heuristic
Augmented views are the bread and butter for self-supervieed learning approaches. Intuitively, the network needs to see diverse augmentations from p" (m) in order to train for invariance. But in the meanwhile, the learning process needs to also avoid training on those 2i having excessive
3

---

distortions who have drastically deviated from the semantics of the original instance. There exists an implicit trade-off in our mind that controls the strength (hyperparameter) of the augmentation actions. The elusive definition on optimal distribution p" (n) therefore seems to be best considered as a constrained distribution offering most data variance subject to the least semantic deviation from the original instance.
We illustrate this point in Fig. [ The left-most picture is the original instance input, whereas the two rows of smaller images display the augmented views generated via the multi-crop augmentation pipeline as in AW [5]. Apparently, some of augmented views are highlighting the bunny, whereas the others capture the straw only. Under such generative process of augmentations, there is good chance that a pair of (bunny, straw) features are imposed be close in feature space. Such noisy invariance might intuitively disrupt the optimization, leading to suboptimal downstream generalization ability. Actually, the work in [36] has theoretically discussed the generalization ability of contrasive pretraining approaches on downstream tasks: if a negative sample is not from different class of the positive, the upperbound of error on downstream linear classification provably guaranteed to increase. Similarly, we can derive from [36] via mild modification that if a pair of constructed positive views are not from the same semantic class, the upperbound of linear classification error in the downstream tasks would also increase.
Nevertheless, in practice, while the optimal desired p" (n) remains unknown and elusive, we usually take it for granted that all generated augmentations are equally good, and we tentatively sample n from some heuristically chosen p(n), surrogate distribution that needs not coincide with the desired p" (n) having the proper augmentation variance-bias trade-off. And instead of optimizing Eq. ( we often optimize:
N
E= (x, n)]
Σ j.k
Z1,k) n2).
(3)
As per our discussion above, since we ultimately hope to sample from the desired (n) instead of the surrogate distribution ÿú, we rewrite Eq. (2) using the basic change of measure rule from the importance sampling technique, and yield:
C' Σ
1 N
Σ ((z)),22,K)) ( n2) 2 j.k
1 N
EE zi,e) im1
((1)) (1))))))))))))))))))))) p" (n,: mk) p(n,, na)
(4)
3.f
Comparing Eq. and Eq. (3). we notice that if we desire to optimize Eq. (2). whereas we insist sampling from instead of p', we should adjust for this change and compensate each loss term via the multiplicative likelihood ratio p" (nj, nx) //(rr n,,, nkg), rather than optimizing Eq. B This likelihood ratio detects how much chance we would miss the optimal sampling distribution If instead using the surrogate distribution p. If p" = p. then the objective function recovers the optimal sampling scheme as in Eq. 2 and no adjustment using likelihood ratio is needed.
In practice, SSL approaches usually freeze the optimization on either feature Zi,j zLk, while only the other one is optimized via backpropagation [1] [8] Therefore it is also reasonable to simplify the objective Lo (x. (n,,, n2)) to Lo (zr, n,), and the likelihood ratio p" (mj, mx)/ ns) to p" (/n,)) In addition, for n = Id which is the identity transformation, Le (æ,, Id) degenerates to the objective function on the original instance 2; without augmentation, we denote it as Co (xx))
3.3 Sampling Distribution p": Lemma
Unfortunately, we do not have access to ratio p" /p (as hard as knowing p²). But we could define w(n) = p² (n)/p(nn) which is essentially the Radon Nikodym derivative, and we tentatively analyze how w would influence the MSE (mean squared error) of deep estimator. To reach this goal, we extend the result in (6, and have the following proposed Lemma on the MSE of deep estimator:
Lemma Define Ou = arg ming Ez)o(z), 0G = arg ming Ez Co n n'ypp" (n)), and θG
arg ming L" Let Vo be the Hessian of O + Lo(x()) at do and Ve the Hessian of and Ma = where Vo, (a) and Vo (m) are gradients of Lo(x) at 0 = fo and 0 Og respectively. Similarly, denote M Mc V Lea (x, V (x, n)T Denote the trace
+ Ez Lo(æ, n) n)dp(n))((n) at 0G. Let Mo(x) = VLa VL,
v Lo (x)VLo

---

of matrix X as tr tr(X). Then. with C' constant invariant of w, under mild conditions, we have:
MSE(OG) ~ + 110G - 0oll2 + t.l tr(Vo - Mo(z)))V (d())))a""d((())unn (p)))w(n))
(5)
+ N
Be (Mo())
(6)
+
-
N (v F,z (Cav (a. Va')
where Covw (V VLa (2, is the covariance matrix of Lø(x, n) at Og under measure p" (m).
Lemma D Tis very busy. But we can still summarize clear variance-bias trade-off implication from it. The key term in Eq. (8) is the covariance matrix of the gradient of loss w.r.t. augmentation measures. In the special case of one-dimensional 0, this term is reflective of the Fisher information and thus larger values lead to better variance reduction for the estimator 0g and smaller MSE. This means we desire more diverse data. In the meanwhile, terms in Eq. (5) and (6) measure the difference of squared loss gradient between augmented view and its original instance. If an augmented view is showing a much different gradient M ( n) in loss than the original clean instance, i.c., gradient Mc(#) (as measured in Eq. )5: ); or showing very different gradient Mc(x) than the gradient on original instance Mo(w) (x) (as in Eq. (6)), then the biases represented in Eq. (5) combined with (6) will increase, and thus the MSE will also increase.
Lemma D also explains our intuition in the previous section: we desire large data variance via sampling from ("n)). such that we can reduce the estimator variance Eq. (8) But data variance is a double-edged sword. A radical sampling distribution on n with excessive distortions can simply contradict the desired prerequisite hypothesis (invariance) and increases estimator bias. Through Lemma]) D we are now clear why we would tune the strength of the augmentations to reach the sweet spot for downstream tasks!
Balancing the variana trade-off to reduce MSE Lemma 1 is tricky. Fortunately, we still have w(n) variable to associate with each term in Eq. (5) Eq. and Eq. 8 Recall, the message from Lemma| D lis: if certain ži.j under the augmentation action of n, shows a strongly deviating direction from original instance 2;;s 's semantics, it is potentially not representative of the i-th instance at all, perhaps is signaling an out of distribution outlier, and might increase the estimation bias. The variable w(n) should be suppressing the effect of these samples. But what is the semantics that original instance x, mostly cares, the bunny or the straw? The most efficient way is to let all views vote, and to compute mean feature μe; using these augmented views. This μe, feature hopefully represents the original semantics. Eq. (5) also implics that, if there is only a single augmented view available, we can additionally forward the original instance without augmentation (i.e., 2;,) and use feature z; = fa,, 0) to approximate μæ: (more discussion in Section 4.2p. Note, deep neural networks eventually will fit whatever OOD noises via memorization (1], even it the paired invariance is noisy and wrong. We expect to avoid this happening through the influence of w to at least reduce bias term in Eq. (5) and (6) without sacrificing much data diversity.
3.4 Optimizing Ratio = p²/ / as Latent Variable
Under measures p" and ğ for n., the probabilities of sampling T.,j = g( n2j), where g is invertible. are (2x:) -p" (nt,) and (x::) p((2)) respectively, due to independence of sampling procedure of æ; and nyy. And the likelihood ratio at æi, (x.)) -p" (nz))/ (p.() p(n,)) = p" (n,)/ wn,)). Therefore, to find the desired w (n,) that reduces the MSE in Lemma 1, it suffices to find likelihood ratio in sampling zt.j. By the one-one correspondence between Ti,j and z, it further boils down to calculating weights wi, for Zi.j-
=
With all of these implications from Lemma 1, we correspondingly define the ratio was as:
p'
x wi, = expl expp-(22,, - -1; () - H.)l.
(9)
p
where the μe and E are defined as:
M
N M
11 M
Σ
Zi.jo
Σ
NM Σ
Σ
(ii M.) -.)²
(10)
5

---

Here, Zi,.. E {l,.., } are features out of M augmentations on zạ. Each (zi,j - 14,) measures the difference between a specific augmentation feature ztj and μ.: Hyperparameter T is the temperature that controls the dependency strength of Wa,j on such distance. The distance is then normalized through Σ among view augmentations. Σ normalizes the distance measurement such that all dimensions in (zi.) (4,)) are of similar magnitude, avoiding risking single dimension dominating the distance measure. Eq. (I0) computes the covariance using all the augmentation of all instances in the current batch and requires all instances share the same Σ. This seemingly strong hypothesis is absolutely not a simplification. It is rather intended to unveil an appealing unique signature of self-super learning that helps shrink the search region of the parameters: the source of variance on all instances are only explained by the same set of augmentation actions. Supervised learning does not enjoy similar assumption, variance of data comes from wild possibilities across classes. Note the construction of the Radon Nikodym derivative Wa. are also supported by the observations in [[] real data examples are consistently shown be easier to fit than noise during the training, i.e., the deep model usually first learns the simple and general patterns of the real data before fitting the noise. We wouldn't 't sacrifice learning true general patterns during the process of suppressing noise.
3.5 The Unsupervised OuTlier Arbitration Approach (UOTA)
According to the discussion above, the complete UOTA training procedure of wi, is described as follows (see pseudocode in the supplementary material). We further normalize the ratio value Wi across the whole batch of samples into T.,y:
Wi,j Σn
x Wa
(11)
WA,j
Our loss function Eq.
eventually boils down to:
N
Lours
Wy.y Co (2:; nz).
(12)
During training, one can replace the placeholder loss Lo (a n2y) in Eq. (12) by particular form of losses defined in variety of SSL approaches such the contrastive loss MoCo. pairwise & loss in BYOL or any alternative losses based on pairwise feature invariance assumption. During each training iteration, we firstly forward the training data Z;, and update Was according Eq. (9) and Eq. (11). where f1; and Σ are computed using Eq. (TO). The value of Wi, is then freezed and we backpropogate the loss defined by Eq. (1) only through feature variable zi,j (depending on the SSL loss), to update the network parameters 0. The whole optimizing procedure is an alternating optimization procedure and iterates between updating the latent variable Way (Eq. ( and Eq. (TT) given fixed 0; and updating 0 (backpropogate Eq. (12) given fixed w.y)). Note, the update of Wi,i in Eq. [T] is motivated from our Lemma I and the importance sampling technique, which conveniently jettisons any threshold selection issues in [25] 26 50). Particularly we further have the following theorem:
Theorem Given mild condition, the optimization of loss Eq. (2) according to the was definition in Eq. ( leads to a reduced MSE defined as in Lemma 7 than having constant Wi.j.
The proof is in the supplementary file. The above theorem implies that UOTA procedure and the associated loss function in Eq. (12) are able to effectively correct the estimator bias. We train a network under UOTA procedure, and present the augmentations in Fig. accordinn descending order of each associated wij. Interestingly. most augmentations clearly capturing bunny object has higher wi. values, whereas the straw is arbitrated highly possible OOD samples (smaller wiz)).
Empirical Analysis
In this section, we empirically evaluate the effectiveness of the proposed UOTA algorithm and justify
correctiveness of Theorem [[ In detail, we replace the term Le r2,)) in Eg. (2) by specific losses
defined by a variety of state-of-the-art SSL methods, i.e., MoCo [18], BYL [SS and SWAV (5). We then implement our proposed UOTA algorithm (see supplementary file for applying (2 to these methods) according to the training procedure in Section 3 For each "X+UOTA" model, we firstly
train its baseline X for N Nwarm warm up epochs, and then we resume the "X+UOTA" training till end

---

Table 1: Performance comparison under different training losses using corresponding architectures. All models pretrained for 200 epochs on ImageNet 100 [9]] Test accuracy reported in %. Standard deviation (+x)) is reported with 10 different runs in pretraining the model.
MoCo-v2
MoCo-v2*
BYOL
SWAV
Model
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
without UOTA
73.0 (+0.1)
92.0 (+0.1)
80.3 (+0.1)
95.5 (40.1)
75.3 (+0.2)
93.4 (40.1)
81.1 (40.2)
95.9
(40.1)
+UOTA
74.0 (+0.2)
92.4 (#0.1)
81.4 (+0.1)
95.7 (40.1)
76.2 (+0.1)
93.7 (40.1)
82.2 (40.2)
96.1 (+0.))
Table 2: Performance comparison under dif- ferent training losses and using corresponding architectures. All models pretrained for 200 epochs on ImageNet100 dataset.
Table 3: Performance comparison using different estimate strategies for Covariance and Mean Eq. (10) of UOTA. All models pretrained for 200 epochs on ImageNet100 dataset.
Model
Top-1 I (%)
Top-5 (%)
Model
Covariance
Mean Top-1 (%) Top-5 (%)
MoCo-v2"
80.3
95.5
MoCo-v2*
/
80.3 (40.1) 95.5 (+0.1)
+Focal 28
79.6
95.2
+GCE 50
80.8
95.5
+UOTA
Σ
μ4
80.7 (#0.2) 95.5 (#0.3)
+MIL-NCE 30
80.4
95.5
+UOTA
Local E;
M4
80.7 (40.2) 95.6 (20.1)
+Debiased 10
80.7
95.6
+UOTA
Glohal E
81.3 (#0.1) 95.6 (#0.1)
+UOTA (Ours)
81.4
95.7
+UOTA
Global E
p4 81.4 (#0.1) 95.7 (#0.1)
(see supplementary file for Nwarm values used for each experiment). Total number of training epochs of X+UOTA including warm up is the same as that for training X. We follow the standard evaluation protocols as in [1] and evaluate the algorithms as follows: (1) Linear evaluation on frozen pretrained features via UOTA or other various methods on both ImageNet100 (39] and ImageNet1 K [T; (2) Extensive ablation studies against hyperparmeter variations; (3) More downstream task accuracy (i.e., object detection, instance segmentation and keypoint detection) on MS COCO dataset [29] by finetuning pretrained network obtained via UOTA or other SSL approaches. Without specification, all tables with standard deviation (=r) are reported with 10 different runs pretraining the model.
4.1 Evaluating the Unsupervised Features on ImageNet100
In this section, we compare the result of various SSL approaches pretrained with the ResNet- 18 Network [20] on ImageNet100 dataset, a subset of ImageNetl K dataset. This is because the suitable data scale allows to reimplement and frequently tune all the models reach their best performances. On ImageNet100 00, we pretrain all the approaches for 200 epochs with a batch size 128. We use each approach's 's default optimizer (e.g.. LARS [45) SGD) and architecture according to their official codes. Following the protocol in [18), we firstly pretrain the ResNet-18 18 with various SSL losses. We then train linear classifier on frozen features out of each pretrained ResNet-18 18 to evaluate the classification accuracy test data. For detailed implementation of all approaches in this section, please see supplementary material.
Implementing UOTA on various SSL algorithms. In this section, we implement a variety of SSL algorithms and plug the corresponding losses into Eq. (12 to pretrain the network. We respectively evaluate the linear classification performances on the original baseline models in row "without UOTA" of Table T and we present each baseline's performance with UOTA approach in the row "+UOTA". The model MoCo-v2" is an additional baseline we implemented by applying the multi-crop augmentation (8 views 2 x 224 + 96) and other training techniques from SWAV MoCo-v2 (LARS optimization, threshold, Ir and etc., see supplementary file). Therefore, Table Daleeaacee actually covers a broad type of SSL approaches: plain contrasive loss based approach (MoCo-v2); non-contrastive loss with cross view & norm penalty YOL))); cluster based multi-view contrastive loss (SwAV, 8 views); and finally contrastive loss with additional augmented views that are trained simultancously (MoCo-v2*). As Table demonstrates. UOTA is showing evident OOD removal effect with robust performance boost (always 2 0.9% gain) against the potential OOD samples in the training data.
Comparison to other related works. We compare with other existing OOD detection techniques, some of which originally proposed for supervised learning scenarios. Focal loss 28 is a typical algorithm overweighting hard samples during training; In contrast, GCE [50] is shown to be robust against label noises for the supervised learning similar to self-paced raining procedure 255 MIL- NCE (30 loss is particularly designed for caption-frane frames misalignments issues in narrated videos, and is tailored for contrastive loss. Debiased contrastive learning [10 (abbreviated in as Debiased in
Table 2 only considers removing the sampling bias from the negative samples. In this section,
7

---

Table 4: All models pretrained for 200 epochs on ImageNet100 dataset. MoCo-v2 and MoCo+ UOTA are trained according to setup in [37. NDA result directly from (37
Table 5: The 2 class (OOD vs Non-OOD) lin- ear classification aceuracy on ImageNet100 data. Group numbers index different strength of augmen- tations detected by Wij.
Model
Top-1 (%)
Top-5 (%)
Model/Group
2
4
6
MoCo-v2
69.7
90.0
Training acc.
99.9 97.0 92.2
87.9
86.5
51.5
MoCo-V2+NDA
70.0
Test acc.
99.8 96.3
91.1
86.7
84.9
49.7
MoCo-v2+UOTA
70.6
90.2
use MoCo-v2* (defined in previous paragraph) as the baseline model, and implement all the relevant approaches on MoCo-v2* applicable. We defer the implementation details into the supplementary material. In Table MoCo-v2*+UOTA, has achieved the best test accuracy across all the algorithms. Focal loss overweights samples that has the highest loss, thus fails when compared with the baseline model. In contrast, MILL-NCE does not adaptively update the sampling distribution depending on the data, therefore showing an inferior performance than ours. This verifies that OOD) issue is indeed unique issue for the SSL approaches having unique embedded noise structures compared to conventional supervised learning approaches. UOTA outperforms "Debiased", which at least shows that addressing the noise issue in positive samples is equally important in negative samples.
While UOTA tries mute the intrinsic OOD samples from positive, the NDA approach in [ even considers manually generating OOD samples as negatives. These two methods differ in motivation, but both would be provably shown result in reduced error bound on the downstream linear classification tasks, as per the discussion in paper [36]. We compare scores reported in [37 by following the ResNet50 training setups of Table showing the effectiveness of both methods while UOTA is relatively better.
37 The results are reported in Table
4.2 Ablation Studies
In this section, we examine the impact of different hyperparamters in the UOTA framework on ImageNer100. We evaluate different settings on linear classification task as defined in Section 4.1 with the same pretraining set up for UOTA as described in Section 4. The impact of r, Nwarm and other hyperparameters of UOTA is included in supplementary material.
Impact of crop-min strength in augmentation We first investigate the effect of the crop-min value. The hyperparameter crop-min defines the minimal size of an augmentation via random cropping from the original instance. According Lemmal [ there should be a sweet point of crop-min value best balancing the trade-off for the estimator MSE. Interestingly, we spot the expected sweetpoint for the MoCo-v2 baseline at value of 0.16, as in Fig. 2(a) Notice that MoCO-V2+UOTA has evident effect in flattening the influence of crop-min against the accuracy. This well corroborates the efficient role of UOTA, which adaptively adjust for the sampling distribution to combat the OOD samples. In this way. UOTA guards the SSL approaches without suffering too much OOD noise.
Impact of number of views for each instance. We adopt the MoCo-v2* framework as the bascline to associate with UOTA. Fig. [(b) shows, as the view numbers for MoCo-v2* increase, the UOTA approach consistently outperforms MoCo-v2" by clear margin. This advantage is even slightly promoted as view number increases. We speculate that as the diversity of views grows, both algorithm benefit from increased data variance and reduced estimator variance through Lemma T while UOTA enjoys better performance gain via Theorem
Impact of covariance estimate. In Section 3.4 we impose the constraint that all instances share the identical covariance. This implicit constraint is critical to practically reduce MSE defined in Lemma I. In this section, justify our hypothesis by comparing with two baselines views). The "local Σ, baseline computes a unique Σ, for each instance by only involving its own augmentations. "Σ- = I" bascline further removes the whitening step (effectively squared euclidean distance) when evaluating the feature distance (between Z,,, and (4)). In contrast, "global Σ" is our default option for UOTA as defined in Eq. 110) i.e., "global Σ" is estimated via all augmentations of all instances (all instances share the same E). As TaalleBssowe "global Σ" outperforms both "local E" and "Σ- I" which demonstrates the benefit of proposed constraint.
MoCo-v2* (8
Impact of mean estimate. In Section 3.3 we hypothesize that μ, around the value of original zi original z, feature can be regarded effecttivlly integrating out the augmentation nuisance
8

---

Top-1 Acc
75.0
74.5
MoCo-V2+U0TA
MoCo-v2
Top-1 Acc.
82
MoCoo22"
MoCovv2 HUDTA
74.0
81
13.5
80
13.0
79
0.10
0.12
D.14
0.16
0.18
0.20
8
Crop-min strength
Number of views
(a) Impact of crop-min strength in augmentation.
(b) Impact of number of views for each instance.
Figure 2: Impact of several hyperparameters of UOTA: (a) crop-min strength, (b) number of views.
variable n given certain 2;. We therefore examine the UOTA performance under the MoCo-v2* * (8 views) framework using these two different mean estimates. "z; "oodel means we replace the μe; by using the original feature (without augmentation) 2; in Eq. (10 The cost of this replacement an extra forward of original æ,, though. As observed in Table 3] there is slight difference between the two models, showing effectiveness using average μ4; to represent 2,.
4.3 Evaluating the Unsupervised Features on ImageNet1K
Implementation details. In this section, we compare the result of various SSL approaches pretrained on the ResNet-50 (20) Following the protocol in [18] we firstly pretrain the ResNet-50 with various SSL losses. We then implement further supervised downstream tasks by either finetuning on the pretrained network, or by directly using the frozen feature out of the pretrained network depending on the task. In this section, we implement the proposed UOTA algorithm by using the SwAV loss (multi-crop with 8 views 2 x 224 + 6 x 96), the exactly architecture and augmentation pool as in [5] to optimize Eq. (2). We train all relevant algorithms on V100 GPUs, with a batch size 256. For other training details, please refer to supplementary task.
Evaluating the linear classification task on frozen features. We further train linear classifier on frozen features out of each pretrained ResNet-50 network to evaluate the classification accuracy on test data. As Table 6 displays, our proposed UOTA algorithm (associated with the SWAV approach) has achieved the best Top-1 accuracy (73.5%) and Top-5 accuracy (91.8%) across all the presented SSL pretraining algorithms. Particularly, the proposed SwAV+UOT: pretrained for 200 epochs even surpasses SimCLR (7) and Barlow Twin [47] trained for 1000 epochs. Besides, the SWAV+UOTA framework outperforms the original SWAV baseline by 0.8% accuracy, In the meanwhile, the total training time for SWAV+UOTA is 181.5 hours, with only negligible 2 hours more than SwAV's training time 179.2 hours on 4 V100 GPUs These results demonstrate clear evidence of proposed hypothesis: OOD samples broadly exist in the training data. Importantly, the OOD issue is not vanishing as the the number of instances in training significantly grows (much more than in ImageNet100), Fortunately, our lightweight UOTA procedure is able to modify the sampling distribution and reduce the estimator bias, guaranteed by our Lemma aand Theorem | We think this 0.8% improvement effectively raises warning flag to the community, and verifies that OOD samples has a clear and non-negligible impact that should be attended to.
Binary classification between OOD data and clean data. In this section, we empirically verify that OOD samples and original image are linear separable on frozen features learned by UOTA. For this purpose, we choose ImageNet100 training data as training set and ImageNetIK val set (50,000 images) as test set. Each of the training data is randomly augmented into 5 views. We reorder all the samples according their computed Wiy value in an ascending order (Group I contains the lowest Wij valued samples, showing strongest sign of OOD sample; Group 5 has the highest Wij samples, best resembling the semantic of the original training sample). We sequentially split such reordered augmentation set into groups with equal sample size. Different groups then reflect the detected semantic uncertainty via UOTA. The test data are augmented in the same way and split into 5 groups, too. We then train the binary classifier on top of the frozen pre-trained model (Sw (SwAV+UOTA model producing Table 6) by respectively treating each Group n class I (OOD class), with clean data considered as class 0 (class O and 1 have equal number of training data). Group 6 is a baseline with equal size of clean data in two subsets representing class and class respectively. More training details are included in the supplementary file. As Tabos, the binary linear classifier between OOD and clean data on their frozen features always show clear statistical significance and better separability against the reference baseline Group 6. Notably, as the data exhibits weaker OOD signs (e.g., Group 5), the training becomes harder and harder, returning lower training accuracy, while the OOD samples still demonstrate much stronger linear separability against the baseline model Group 6.

---

Table Accuracy of linear classification model on ImageNetIK. Bold numbers are the best perfor- mance among models trained for 200 epochs. Numbers (+x%) denotes additional gain compared to the baseline model (i.e., SwAV here in the table) without UOTA approach. t denotes results means results of reproduced reproduced based on SwAV official code.
represented from 9
Method
Epochs
Batch size
Top-1 (%)
Top-5 (%)
CPC v2 21
200
512
63.8
85.3
CMC
240
64.8
86.1
MoCo 18
200
256
60.6
83.1
MoCo X
200
256
67.61
88.01
JCI 8
200
256
68.7
89.0
SimCLR 7
1000
4096
69.3
89.0
SimSiam 9
200
256
70.0
InfoMin Aug.
200
256
89.4
BYOL 15
200
4096
70.6
Barlow Twin 47
1000
2048
73.2
91.0
SwAV 5
200
256
72.77
91.5
SWAV=UOTA (Ours)
200
256
73.5
(+0.8%)
91.8 (+0.3%)
Table 7: Performance downstream tasks: object detection B3 (left), instance segmentation [9
(middle) and keypoint detection [19] (right). Accuracy in %. All models pretrained 200 epochs and finetuned on MS COCO with 1 schedule.
R.CNN RS0-FPN
Mask R.CNN RSO-FPN
Keypoint R-CNN + RS0-FPN
Model
Apd
AP
AP
Apm
APSI
AP
AP4
AP
AP
random
30.1
48.6
31.9
28.5
46.8
30.4
63.5
85.3
69.3
supervised
38.2
59.1
41.5
65.4
87.0
71.0
37.1
37.4
40.2
33.1
53.9
37.7
63.6
87.1
71.3
MoCo-y2
8
37.6
57.9
40.8
35.3
55.9
37.9
66.0
87.2
71.4
JCL S
38.1
58.3
41.3
35.6
56.2
38.3
66.2
87.2
72.3
InfoMin Aug. 40
/
/
36.7
57.7
39,4
/
/
SWAV
38.5
60.5
41.4
36.3
57.7
38.9
65.6
86.9
71.6
SwAV+UOTA
39.0
61.0
42.0
36.7
58.4
39.4
66.3
87.4
72.3
Finetuning on pretrained network for more downstream tasks. In this section, we evaluate the presented SSL algorithms by finetuning the pretrained ResNet50 equipped with FPN 27 on more downstream tasks. Table respee//espectiveef, reports the performance on object detection, instance segmentation and keypoint detection tasks. These tasks, to some extent, evaluate the generalization capability of the pretrained features when transferred to other tasks and training datasets. According to Table 7 SwAV+UOTA still outperforms its baseline model SwAV on all evaluation metrics for the object detection and instance segmentation tasks. Notably, SwAV baseline drops drastically on the keypoint detection tasks in comparison to MoCo-v2. We speculate that the multi-resolution multi-crop augmentation has introduced more OOD samples into SwAV than in MoCO-v2, leading to vague semantic feature that critical for successful key-point detection tasks. This well reveals that, even if we further finetune the pretrained network using alternative dataset, SwAV+UOTA is still able to benefit tasks by suppressing the OOD samples and balancing the estimator bias-variance trade-off during the pretrain. The advantage of UOTA is not erased due the finetuning. Table concludes that the down- down-weight weighted OOD samples through training using UOTA approach successfully the true invariant semantics shared across multi-views, which plays pivotal importance on downstream task performance.
forced the network to focus
Conclusion
In this paper, we explore the potential OOD noise issue for SSL approaches. Most importantly, demonstrate that these OOD samples are highly structured, and are interfering the downstream tasks with non-neglnn nog-neglig impact. We use importance sampling technique investigate the source of OOD problem via estimator MSE decomposition, and we identify an important bias-variance trade-off behind the sampling distribution options for positive views. We propose a useful latent variable model UOTA to effectively balance such trade-off. Empirical study well corroborates our hypothesis and shows strong advantage of our UOTA approach over the state-of-the-ar SSL approaches.
Social Impact and Limitation. With our SSL approach, one can conveniently construct her own pretraining dataset by random crawling data. This economic and beneficial for research purpose, but would potentially risk privacy and license issues. Also, given the easy access to data via unsupervised learning, one might risk abusing the use of technique and applying in appropriate occasions. Our approach also requires tuning extra hyperparameters depending dataset.
10

---

Acknowledgments and Disclosure of Funding
Funding direct support of this work: the National Key R&D Program of China under Grant No. 2020AAA0108600.
References
[1] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In ICML, 2017.
[2] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS, 2019.
[3] Qi Cai, Yu Wang, Yingwei Pan, Ting Yao, and Tao Mei. Joint contrastive learning with infinite possibilities. In NewrIPS, 2020.
[4] Fabio M Carlucci, Antonio D' D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles, In CVPR, 2019.
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
[6] Shuxiao Chen, Edgar Dobriban, and Jane H. Lee. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21(245):1-71, 2020.
[7] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. In ICML, 2020.
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv:2003.00 042927 2020.
[9] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CVPR, 2021.
[10] Ching- Bao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In NeurIPS, 2020.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[12] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
[13] Adam Foster, Rattana Pukdee, and Tom Rainforth. Improving transformation invariance in contrastive representation learning. In ICLR, 2021.
[14] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018
[15] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your latent new approach to self-supervised learning. In NewrIPS, 2020.
[16] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
[17] R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
[18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick, Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.

---

[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[21] Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. In ICML, 2020.
[22] Peter Huber. Robust statistics, volume 523. John Wiley & Sons, 2004.
[23] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICMI., 2018.
[24] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In NeurlPS, 2020.
[25] M. Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In NIPS, 2010.
[26] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting
samples and adversarial attacks. In NeurlPS, 2018.
[27] Tsung-Yt Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[28] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[29] Tsung-Y- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[30] Antoine Miech, Jean-Baptiste Alayrac, Lueas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020.
[31] Ishan Misra and I Laurens van der Maaten. Self-supervised learning of pretext-invariant repre- sentations. In CVPR, 2020.
[32] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
[33] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv:1807.05745 2018.
[34] Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for duitof-diittibution detection. In NeurlPS, 2019.
[35] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
[36] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
[37] Abhishek Sinha, Kumar Ayush, Jiaming Song, Burak Uzkent, Hongxia Jin, and Stefano Ermon. Negative data augmentation. In ICLR, 2021.
[38] Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding. In NewrlPS, 2020.
[39] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
12

---

[40] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. NeurIPS, 2020.
[41] Feng Wang, Huaping Liu, Guo, and Sun Fuchun. Unsupervised representation learning by invariance propagation. In NeurIPS, 2020.
[42] Yu Wang, Jingyang Lin, Qi Cai, Yingwei Pan, Ting Yao, Hongyang Chao, and Tao Mei. A low rank promoting prior for unsupervised contrastive learning. arXiv preprint 100.0966,, 2021.
[43] Ting Yao, Yiheng Zhang, Zhaofan Qiu, Yingwei Pan, and Tao Mei. Seco: Exploring sequence supervision for unsupervised representation learning, In AAAI, 2020.
[44] Mang Ye, Xu Zhang, Pong Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, 2019.
[45] Yang You. Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv:1708.000 033888, 2017.
[46] Xinlei Chen Yuandong Tian and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICML, 2021.
[47] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-
supervised learning via redundancy reduction. In ICML, 2021.
[48] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
[49] Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In CVPR, 2017.
[50] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS, 2018.
[51] Huangjie Zheng, Xu Chen, Jiangchao Yao, Hongxia Yang, Chunyuan Li, Ya Zhang, Hao Zhang, Ivor Tsang, Jingren Zhou, and Mingyuan Zhou. Contrastive conditional transport for representation learning. arXiv:2105.03774 2021.
[52] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In ICCV, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5 (Social Impact and Limitation).
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
See
Section 5 (Social Impact and Limitation).
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs all theoretical results? [Yes] See supplementary file.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or a URL.)? [Yes] See supplementary file.
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplementary file.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.3 (Implementation details).
4. If you using existing assets (c.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "tfP0KX2LmqI", "reviewer": "Reviewer_wYZX", "summary": "This paper presents an algorithm named UOTA for self-supervised representation learning. The method is based on the observation that some low-quality negative samples are generated during the learning procedure and these bad samples can confuse the deep networks. So, a Gaussian distribution is built to estimate the weight of each sample and added to the loss term. Experiments show consistent accuracy gain.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an algorithm named UOTA for self-supervised representation learning. The method is based on the observation that some low-quality positive samples are generated during the learning procedure and these bad samples can confuse the deep networks. So, a Gaussian distribution is built to estimate the weight of each sample and added to the loss term. Experiments show consistent accuracy gain.\n\nThe major concern of this paper lies in the technical contribution. Though some ablations are made to validate the effectiveness of introducing a Gaussian distribution to estimate the quality of positive samples, this is actually a small add-on to the existing self-supervised learning approaches. I buy the motivation, the method, and the mathematical analysis, but I just think that the contribution does not achieve the standard of NeurIPS.\n\nTo add more words, I do think that the current self-supervised learning methods rely too much on the negative samples, which is not good. However, I really expect some efforts of alleviating this issue by (1) finding key objects in the image so that the positive samples can largely preserve the quality of these key objects; or (2) finding a new proxy that bypasses the requirement of positive or negative pairs. I know that it is unfair to ask the authors to follow the suggested path, yet I just think that adding a weight to each positive sample forms a limited contribution, so I cannot suggested acceptance for now.\n\nBTW, there are some efforts in adding stronger augmentations to achieve better performance of self-supervised learning. Do you have any comments of applying the proposed approach, UOTA, to these stronger augmented samples?\n[A] Wang et al., Contrastive Learning with Stronger Augmentations, 2020.\n[B] Xu et al., Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "bo9I5j1CzUb", "reviewer": "Reviewer_iGgP", "summary": "This contribution is based on two main claims: i- using data augmentation to create different views from an image might result in out-of-distribution instances; and ii- employing such out-of-distribution views as positive instances in self-supervised approaches that rely on comparing representations from positive and negative views of an example, might be harmful for downstream performance. To address these issues, the authors proposed to apply an out-of-distribution detection approach to estimate how likely it is for a positive view to be out-of-distribution and use this information to weight the contribution of each positive pair when computing the learning criterion. The proposed strategy, named Unsupervised OuTlier Arbitration (UOTA), is then incorporated to three previously proposed self-supervised learning approaches and empirically validated on ImageNet and MS-COCO. Comparisons with other out-of-distribution detection techniques were also provided. Overall, UOTA showed to marginally improve the downstream performance of self-supervised pretraining both with respect to the compared detection approaches as well as self-supervised models without any mechanism to compensate for potential out-of-distribution positive views. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main positive aspect of this work is to raise a discussion on using out-of-distribution detection as a tool for improving self-supervised methods that rely on contrastive learning between multiple views. I also appreciated the large number of experiments and ablation studies presented in the empirical analysis. In spite of that, I think this contribution is not ready for publication, mostly due to a lack of clear evidence to support its main claims and conclusions. In addition to that, I found the manuscript quite hard to follow (it was necessary to read some sections a few times to grasp at least their main content). In the following, I further detail my concerns and provide suggestions to address them.  \n\n\n- Unsupported claims: \n  \n  - The authors claim in line 30 \"We find these produced views even semantically deviate from the original instance and thus behave like out of distribution (OOD) samples\" and in line 92  \"we reveal that OOD noise is inherently existing in positive view sampling pipelines for a broad range of mainstream SSL approaches\". However, it is not clear to me which part (if any) of the discussion presented in Sections 3.1, 3.2, and 3.3 supports this claim. As far as I understood, the example provided in Figure 1 is the only evidence showing that augmentation approaches for generating positive views are introducing out-of-distribution instances. To address this issue in the current version of the manuscript, experiments such as showing whether linear classifiers trained on top of features extracted by self-supervised pretrained models are able to distinguish augmented and original examples could be interesting (i.e. estimating the $\\mathcal{H}$-divergence [1] between augmented and original data distributions).\n\n  - Moreover, in line 32:  \"The model therefore fail to generalize in some certain parameter region due to the interference of the OOD samples\". I also couldn't find in the manuscript no piece of evidence to support such claim. How exactly out-of-distribution views are hurting generalization performance? Aren't there cases where enforcing the model to match positive views that might be considered as coming from different distributions could actually be helping the encoder to discard irrelevant information for the downstream the task?\n\n  - Line 369: \"Empirical study well corroborates our hypothesis and shows strong advantage of our UOTA approach over the state-of-the-art SSL approaches.\" The results are not presented along with error bars or any statistical significance analysis, therefore, I don't think it is possible to conclude from the empirical evaluation that UOTA shows a *strong* advantage with respect to previous approaches. In Table 4, for example, the improvement obtained by SwAV+UOTA was less than 1% in comparison to SwAV when training both models during 200 epochs, which might be due to other factors rather than due to the use of UOTA. To further illustrate my point, in Figure 3b of [3], it is shown that after training SwAV for 200 epochs it reaches 73.9% top-1 accuracy on ImageNet1k, surpassing SwAV+UOTA with the same budget of epochs. To address this issue, I suggest the authors to make it clear in the manuscript that the experiments do not indicate that UOTA provides a *strong* advantage with respect to the considered baselines. \n\n\n- Questions and other concerns:\n  -  Surprisingly, the introduction contains no reference to support several statements related to previous work. For example, \"So far, the most prevailing and effective assumption, is to force views from the same instance to be invariant in the feature space.\" (line 25) need to be followed by the corresponding references. \n\n  - It seems to me that different augmentations will shift the data distribution in different ways and this aspect should also be discussed in the manuscript. More specifically, I'm looking forward to understanding whether it would be helpful to only consider in the pipeline augmentations that generate small shifts. In case only these augmentations are used to generate positive views, how would this affect the downstream performance of the self-supervised model, *i.e.* what is the interplay between the distribution shift induced by an augmentation pipeline and the generalization capability of the self-supervised model on downstream tasks? \n\n  - Although I understand that contrastive learning approaches are very common for developing self-supervised methods, I believe the authors should also acknowledge in the manuscript the related literature which is based on different strategies, such as clustering. Please cite, discuss, and include for comparison in the experiments approaches such as DeepClusterv2 [2, 3] (it showed strong performance on several experimental settings, and pretrained models along with the original implementation are available on github).\n\n  - Would the proposed importance sampling approach be also useful to weight the importance of augmentations on purely supervised tasks?  \n\n  - Line 109: specify that $\\theta$ corresponds to the model's parameters.\n\n  - Line 126: what is an M-estimator?\n\n   - Line 180 \"A radical sampling distribution on n can simply contradict the desired prerequisite hypothesis (invariance) at hand and increases bias\". It is not clear to me what radical sampling means. Please clarify. \n\n- Minor (not affected the final score):\n  - Typo in line 277: \"raining\" -> training\n  - Line 374: \"appropriate\" -> inappropriate?\n\n\n[1] Ben-David, Shai, et al. \"A theory of learning from different domains\", 2010.\n\n[2] Caron, Mathilde, et al. \"Deep clustering for unsupervised learning of visual features\", 2018.\n\n[3] Caron, Mathilde, et al. \"Unsupervised learning of visual features by contrasting cluster assignments\", 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "E1dK6uW9Mj", "reviewer": "Reviewer_MNMm", "summary": "The paper is generally concerned with self-supervised contrastive learning, such as SimCLR. The authors posit that existing ways to generate (contrastive) views introduce out-of-distribution examples that actually harm performance. To remedy this, the authors propose a latent variable model they term UOTA that, based on importance sampling, suggest good view candidates. They show UOTA's theoretical properties and conduct extensive experiments, showing gains using UOTA against a large number of recent baselines. ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation for the paper is good -- contrastive learning is a hot topic recently, and as far as I'm aware there is little work describing the harm of using existing view generation / data augmentation strategies. The paper is a bit hard to follow -- the flow and general exposition can be improved, and presenting UOTA as an algorithm block could improve readability. I felt the experimental evaluation was solid -- testing using MoCo, BYOL, and SwAV, with Focal loss, Generalized Cross Entropy, MIL-NCE, Debiased, etc. I felt the ablations were complete as well. The strong and complete empirical results push me to lean in favor of acceptance, however I encourage the authors to improve the flow of the first half of the paper. I did not thoroughly check the proofs in the Appendix, and I'm not too confident about the related works on this particular topic.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "99MebAaGMTq", "reviewer": "Reviewer_ifnT", "summary": "The paper suggests that the different augmentations used as a positive pair during self-supervised training often lack the same semantic structure as the original image, and thus should be treated as an OOD sample. Using such OOD samples, they believe can hurt the downstream task performance. The aim should be to choose augmentations that offer the most data variance but are subject to the least semantic deviation. For this, they propose an importance sampling approach to weigh the loss for different augmentations differently. Empirically they obtain improvement for different downstream tasks across different datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of certain augmentations samples lacking semantic consistency and thus being weighed differently seems interesting. The authors do a good job of motivating the paper and showing results across different datasets and downstream tasks. The ablation studies are also nicely conducted.\n\nMy main doubts/concerns regarding the paper are the following - \n\n1. While it is true that strong augmentations (such as RandomResizedCrop) often create OOD samples, but even for supervised settings people have been using such strong augmentations[1]. I believe the proposed approach can be extended even for such supervised settings. Can the reviewers show results over the supervised setting too, or otherwise comment on why their approach is not suitable for this direction.\n\n2. Another work suggests explicitly generating OOD samples by augmentations such as Jigsaw, and then treats them as negatives during SSL training [2]. Can the authors compare their results to this paper?\n\n3. The weights computed in the initial stages might not be useful, as the features extracted from the network are not well trained. What happens if you instead use a pre-trained network to determine the weights of the augmentations. Say for instance, you take a pre-trained MoCo-v2 and use the features extracted from this network to computed the weights of samples. How does this perform?\n\n\nI will be willing to increase my score if the above points are addressed.\n\nReferences - \n[1] https://github.com/pytorch/examples/blob/master/imagenet/main.py \n[2] Negative Data Augmentation. Sinha et al.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes the authors have addressed their work's societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a robust discussion amongst the reviewers about the merits of this work. There was some disagreement as to whether this work's potential impact on the field of semi-supervised learning is big enough. I decided to side with the reviewers that argued that this contribution could in fact be important. The reviewers have unanimously suggested that the new experimental results be included in the final version, as well improved flow & readability.
Decision: Accept (Poster)