Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration

1 Introduction . Self-supervised learning is an increasingly appealing direction in learning effective deep representations . In the regime of computer vision , natural language processing and machine learning tasks , multiple milestones under such self-supervised learning frameworks have been established [ 4 , 5 , 14 , 12 , 18 , 32 , 31 , 33 , 39 , 43 , 47 , 48 , 49 ] . The premise here is that self-supervised learning methods usually generate multiple views and assume one view be predictive of another . So far , the most prevailing assumption is to force views from the same instance invariant in the feature space [ 7 , 18 , 33 , 47 ] . While it is certainly natural to consider generating diverse augmentations to spread the feature distribution and force consistency in between views , we reveal a structured shortcoming of such popular augmentation pipeline : excessive distortions applied on original image would produce samples that deviate drastically in the semantics . We find these produced views have semantically deviated from the original instance and thus behave like out of distribution ( OOD ) samples . The model therefore fail to generalize in some certain parameter region due to the interference of the OOD samples . We support our claim by empirically verifying the non-negligible detrimental impact of the OOD noise on downstream task performances . Particularly , we conduct extensive experiments and show that importance sampling technique and the associated analysis can help effectively differentiate the noises . The proposed new model suppresses the effect of noisy samples and efficiently improve the downstream task performance over a broad spectrum of state-of-the-art self-supervised learning 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . approaches . This indicates the evident failure of baseline model in some parameter space owing to the OOD noise . Our contribution in this paper are summarized as follows : 1 . We present the first formal analysis of OOD issue under the currently prevailing self-supervised learning framework . 2 . We propose a lightweight latent variable model UOTA that is able to differentiate noise from original instance ’ s semantics . The new model does not introduce extra computational complexity whereas it can efficiently suppress the influence of OOD samples . In contrast to existing SSL paradigms , the proposal method is able to automatically balance the bias-variance trade-off in the mean squared error ( MSE ) of the deep estimator : We do desire large data variance through strong augmentations , on the other hand , we should be careful not to introduce extra estimator bias through extremely large data distortions . We argue that in the context of self-supervised learning , the hidden OOD samples and such bias-variance trade-off have a clear and non-negligible impact that should be attended to . 2 Related Work . Self-supervised learning . Without access to any data annotations , self-supervised learning ( SSL ) approaches impose additional constraints or hypothesis among distinct views of the original input data [ 2 , 7 , 13 , 14 , 32 , 33 , 39 , 41 , 44 , 52 ] . Recently , mainstream approaches require that different views out of the same instance should be predicative of each others . Such line of research can be roughly classified into several sub-directions . The first family is inspired from the seminal NCE approach [ 16 ] and capitalizes on contrastive loss [ 5 , 7 , 17 , 18 , 24 , 33 , 38 , 42 ] . These approaches usually rely on the construction of both the positive and negative views of a single instance . In these work , positive samples usually are augmented views of the same reference instance ; Negative samples are defined as any views from a distinct instance other than that reference instance . Contrastive loss is then expected to learn useful semantics when forced to distinguish between positive and negative views on each instance . The second line of research further simplify the contrastive assumption , which only imposes invariant constraints between paired positive samples [ 9 , 15 , 46 , 47 ] in the absence of the negative samples . Bias correction for contrastive learning . Existing literature prioritize the sampling bias issue for negative samples , while positive samples are usually considered clean . In [ 10 ] , debiased contrastive learning corrects the sampling bias in the negative samples in order to reduce the effect of false negatives . In [ 24 , 37 ] , harder negatives are created for training in order to improve the learning . The work in [ 30 ] is the only work we are aware of , that touches the noise issue in positive pairs . But [ 30 ] is motivated exclusively for observed caption-frames misalignments issues in narrated videos , and the loss is tailored for applications that are based on contrastive loss only . In contrast , we reveal that OOD noise actually exists in a broader coverage of self-supervised learning frameworks and is influencing the downstream tasks with a non-negligible strength . OOD detection and noisy label problem . Outlier detection is an important problem in machine learning . A popular and intuitive framework for detecting OOD inputs is to train a generative model on training data and use that to detect OOD inputs at test time . For instance , in [ 34 ] , a likelihood ratio method is proposed to correct for the background statistics between inliers and outliers . In the meanwhile , for supervised learning tasks , robust training in the presence of noisy labels is also a practical and challenging issue . In [ 50 ] , a generalized cross entropy loss is used to mitigate the label noise issue . In [ 23 ] , Mentor-net capitalizes on tailored network architecture to down-weight the samples likely to be associated with noisy labels . All of these work were targeting conventional supervised learning approaches , whereas the noise under the SSL framework is rarely explored . Comparison to existing works . We unveil that the intrinsic positive sample noises exist broadly behind various mainstream SSL frameworks . To both justify and to mitigate the issue , we propose a latent variable framework that is capable improving the SSL optimization via automatic Unsupervised OuTlier Arbitration ( UOTA ) . We are also aware of a contemporary work in [ 51 ] that attempts to change the sampling distribution via assigning sample weights to each positive sample . But motivation in [ 51 ] is completely opposite to ours , as they oversample the region whenever a positive sample turns out to be remote to its query , while they still assume a perfect invariance on positive keys as critical supporting prerequisite . In [ 26 ] , a Mahalanobis distance-based confidence score is defined to modify sampling distribution for supervised learning tasks . Note our work differentiates from [ 26 ] in many aspects : we analyze the OOD issue particularly for unsupervised SSL frameworks , whereas [ 26 ] strongly hinges on data labels to compute confidence . We model the associated sampling ratio as a latent variable model which is exclusively suitable for the SSL problem , and is supported by rigorous analysis . Our model also naturally jettisons the need of any threshold as used in [ 26 ] . 3 Method . In this section , we reveal that OOD noise is inherently existing in positive view sampling pipelines for a broad range of mainstream SSL approaches . Most importantly , we showcase that these OOD samples are highly structured , and disrupt the downstream tasks with a non-negligible strength . We borrow basic analytic tool from importance sampling to investigate the source of OOD problem specifically for SSL frameworks . We identify an important bias-variance trade-off behind the sampling distribution options for positive views ( augmentations ) , and we analyze the impact of such bias-variance trade-off on the estimator MSE ( mean squared error ) via rigorous justification . Our model UOTA is motivated to adaptively balance the sampling distribution trade-off via a novel latent variable model , and the proposed method is agnostic to the form of SSL loss . We show that the model is able to effectively suppress the OOD sampling noise for positive samples . 3.1 Problem Formulation of SSL . Mainstream SSL approaches , especially contrastive learning methods , often rely on a series transformation functions xi , k = g ( xi , nk ) to produce multiple “ views ” xi , k of input instance xi . The { nk : k = 1 , . . . , K } is the support of an “ augmentation nuisances variable ” n , representing a group of transformations such as flipping , scaling , cropping , or other augmentations acting on xi via function g. Subscript k indexes all possible augmentations , i indicates the i-th instance ( each instance corresponds to a specific image input ) . Without labeling information , SSL relies on the invariance assumption so that deep feature extractor zi , k = f ( xi , k , θ ) is at least insensitive to such group of augmentation nuisances nk w.r.t . the network parameters θ . Existing work [ 5 , 7 , 18 , 33 ] show that , while the deep extractor f ( xi , k , θ ) is trained to learn the shared semantics from distorted views xi , k , such pretrained network parameters θ effectively benefit further downstream tasks . SSL approaches based on such pairwised instance invariance assumptions can be generally summarized as penalizing some form of inconsistency between paired of features zi , j , zi , k : Lθ ( xi , ( nj , nk ) ) = L ( zi , j , zi , k ) ( 1 ) where j , k indexes over the augmented views on instance xi . Without loss of generality , we expand the support of random variable n to pairwised { ( nj , nk ) : j , k = 1 , . . . , K } to incorporate pairs of augmentation actions . Loss Lθ ( xi , ( nj , nk ) ) represents such invariance loss under the pair of augmentation actions ( nj , nk ) . Here , the generic form of loss function L ( zi , j , zi , k ) imposes the desired cross-view invariance assumptions between zi , j and zi , k for the i-th instance . Take for instance , L ( zi , j , zi , k ) can be defined as contrastive loss in MoCo [ 18 ] , clustered loss as in SwAV [ 5 ] , or simply ` 2 norm as in BYOL [ 15 ] . Our idealized goal then is to sample from independent latent generative processes xi ∼ px and n ∼ p∗ such that zi , k = f ( xi , k , θ ) = f ( g ( xi , nk ) , θ ) captures the same semantics as from original xi . Assume we have access to such oracle distribution p∗ , and with equal sampling probability for xi , the objective function is : L∗ = Ex∼px , n∼p∗ [ Lθ ( x , n ) ] ≈ 1 N N∑ i=1 ∑ j , k L ( zi , j , zi , k ) p ∗ ( nj , nk ) . ( 2 ) Optimizing Eq . ( 2 ) then corresponds to learning an M-estimator [ 22 ] with regard to network parameters θ . 3.2 Sampling Distribution p∗ : a Heuristic . Augmented views are the bread and butter for self-supervised learning approaches . Intuitively , the network needs to see diverse augmentations from p∗ ( n ) in order to train for invariance . But in the meanwhile , the learning process needs to also avoid training on those xi , k having excessive distortions who have drastically deviated from the semantics of the original instance . There exists an implicit trade-off in our mind that controls the strength ( hyperparameter ) of the augmentation actions . The elusive definition on optimal distribution p∗ ( n ) therefore seems to be best considered as a constrained distribution offering most data variance subject to the least semantic deviation from the original instance . We illustrate this point in Fig . 1 . The left-most picture is the original instance input , whereas the two rows of smaller images display the augmented views generated via the multi-crop augmentation pipeline as in SwAV [ 5 ] . Apparently , some of augmented views are highlighting the bunny , whereas the others capture the straw only . Under such generative process of augmentations , there is a good chance that a pair of ( bunny , straw ) features are imposed to be close in feature space . Such noisy invariance might intuitively disrupt the optimization , leading to suboptimal downstream generalization ability . Actually , the work in [ 36 ] has theoretically discussed the generalization ability of contrasive pretraining approaches on downstream tasks : if a negative sample is not from a different class of the positive , the upperbound of error on downstream linear classification is provably guaranteed to increase . Similarly , we can derive from [ 36 ] via mild modification that if a pair of constructed positive views are not from the same semantic class , the upperbound of linear classification error in the downstream tasks would also increase . Nevertheless , in practice , while the optimal desired p∗ ( n ) remains unknown and elusive , we usually take it for granted that all generated augmentations are equally good , and we tentatively sample n from some heuristically chosen p̃ ( n ) , a surrogate distribution that needs not coincide with the desired p∗ ( n ) having the proper augmentation variance-bias trade-off . And instead of optimizing Eq . ( 2 ) , we often optimize : L̃ = Ex∼px , n∼p̃ [ Lθ ( x , n ) ] ≈ 1 N N∑ i=1 ∑ j , k L ( zi , j , zi , k ) p̃ ( nj , nk ) . ( 3 ) As per our discussion above , since we ultimately hope to sample from the desired p∗ ( n ) instead of the surrogate distribution p̃ , we rewrite Eq . ( 2 ) using the basic change of measure rule from the importance sampling technique , and yield : L∗ ≈ 1 N N∑ i=1 ∑ j , k L ( zi , j , zi , k ) p ∗ ( nj , nk ) ≈ 1 N N∑ i=1 ∑ j , k L ( zi , j , zi , k ) p̃ ( nj , nk ) · p∗ ( nj , nk ) p̃ ( nj , nk ) . ( 4 ) Comparing Eq . ( 4 ) and Eq . ( 3 ) , we notice that if we desire to optimize Eq . ( 2 ) , whereas we insist sampling from p̃ instead of p∗ , we should adjust for this change and compensate each loss term via the multiplicative likelihood ratio p∗ ( nj , nk ) /p̃ ( nj , nk ) , rather than optimizing Eq . ( 3 ) . This likelihood ratio detects how much chance we would miss the optimal sampling distribution if instead using the surrogate distribution p̃ . If p∗ = p̃ , then the objective function recovers the optimal sampling scheme as in Eq . ( 2 ) and no adjustment using likelihood ratio is needed . In practice , SSL approaches usually freeze the optimization on either feature zi , j or zi , k , while only the other one is optimized via backpropagation [ 15 , 18 ] . Therefore it is also reasonable to simplify the objective Lθ ( x , ( nj , nk ) ) to Lθ ( x , nj ) , and the likelihood ratio p∗ ( nj , nk ) /p̃ ( nj , nk ) to p∗ ( nj ) /p̃ ( nj ) . In addition , forn = Id which is the identity transformation , Lθ ( xi , Id ) degenerates to the objective function on the original instance xi without augmentation , we denote it as Lθ ( xi ) .

Summary: This paper presents an algorithm named UOTA for self-supervised representation learning. The method is based on the observation that some low-quality negative samples are generated during the learning procedure and these bad samples can confuse the deep networks. So, a Gaussian distribution is built to estimate the weight of each sample and added to the loss term. Experiments show consistent accuracy gain.