Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration

1 Introduction . Self-supervised learning is an increasingly appealing direction in learning effective deep representations . In the regime of computer vision , natural language processing and machine learning tasks , multiple milestones under such self-supervised learning frameworks have been established [ 4 , 5 , 14 , 12 , 18 , 32 , 31 , 33 , 39 , 43 , 47 , 48 , 49 ] . The premise here is that self-supervised learning methods usually generate multiple views and assume one view be predictive of another . So far , the most prevailing assumption is to force views from the same instance invariant in the feature space [ 7 , 18 , 33 , 47 ] . While it is certainly natural to consider generating diverse augmentations to spread the feature distribution and force consistency in between views , we reveal a structured shortcoming of such popular augmentation pipeline : excessive distortions applied on original image would produce samples that deviate drastically in the semantics . We find these produced views have semantically deviated from the original instance and thus behave like out of distribution ( OOD ) samples . The model therefore fail to generalize in some certain parameter region due to the interference of the OOD samples . We support our claim by empirically verifying the non-negligible detrimental impact of the OOD noise on downstream task performances . Particularly , we conduct extensive experiments and show that importance sampling technique and the associated analysis can help effectively differentiate the noises . The proposed new model suppresses the effect of noisy samples and efficiently improve the downstream task performance over a broad spectrum of state-of-the-art self-supervised learning 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . approaches . This indicates the evident failure of baseline model in some parameter space owing to the OOD noise . Our contribution in this paper are summarized as follows : 1 . We present the first formal analysis of OOD issue under the currently prevailing self-supervised learning framework . 2 . We propose a lightweight latent variable model UOTA that is able to differentiate noise from original instance ’ s semantics . The new model does not introduce extra computational complexity whereas it can efficiently suppress the influence of OOD samples . In contrast to existing SSL paradigms , the proposal method is able to automatically balance the bias-variance trade-off in the mean squared error ( MSE ) of the deep estimator : We do desire large data variance through strong augmentations , on the other hand , we should be careful not to introduce extra estimator bias through extremely large data distortions . We argue that in the context of self-supervised learning , the hidden OOD samples and such bias-variance trade-off have a clear and non-negligible impact that should be attended to . 2 Related Work . Self-supervised learning . Without access to any data annotations , self-supervised learning ( SSL ) approaches impose additional constraints or hypothesis among distinct views of the original input data [ 2 , 7 , 13 , 14 , 32 , 33 , 39 , 41 , 44 , 52 ] . Recently , mainstream approaches require that different views out of the same instance should be predicative of each others . Such line of research can be roughly classified into several sub-directions . The first family is inspired from the seminal NCE approach [ 16 ] and capitalizes on contrastive loss [ 5 , 7 , 17 , 18 , 24 , 33 , 38 , 42 ] . These approaches usually rely on the construction of both the positive and negative views of a single instance . In these work , positive samples usually are augmented views of the same reference instance ; Negative samples are defined as any views from a distinct instance other than that reference instance . Contrastive loss is then expected to learn useful semantics when forced to distinguish between positive and negative views on each instance . The second line of research further simplify the contrastive assumption , which only imposes invariant constraints between paired positive samples [ 9 , 15 , 46 , 47 ] in the absence of the negative samples . Bias correction for contrastive learning . Existing literature prioritize the sampling bias issue for negative samples , while positive samples are usually considered clean . In [ 10 ] , debiased contrastive learning corrects the sampling bias in the negative samples in order to reduce the effect of false negatives . In [ 24 , 37 ] , harder negatives are created for training in order to improve the learning . The work in [ 30 ] is the only work we are aware of , that touches the noise issue in positive pairs . But [ 30 ] is motivated exclusively for observed caption-frames misalignments issues in narrated videos , and the loss is tailored for applications that are based on contrastive loss only . In contrast , we reveal that OOD noise actually exists in a broader coverage of self-supervised learning frameworks and is influencing the downstream tasks with a non-negligible strength . OOD detection and noisy label problem . Outlier detection is an important problem in machine learning . A popular and intuitive framework for detecting OOD inputs is to train a generative model on training data and use that to detect OOD inputs at test time . For instance , in [ 34 ] , a likelihood ratio method is proposed to correct for the background statistics between inliers and outliers . In the meanwhile , for supervised learning tasks , robust training in the presence of noisy labels is also a practical and challenging issue . In [ 50 ] , a generalized cross entropy loss is used to mitigate the label noise issue . In [ 23 ] , Mentor-net capitalizes on tailored network architecture to down-weight the samples likely to be associated with noisy labels . All of these work were targeting conventional supervised learning approaches , whereas the noise under the SSL framework is rarely explored . Comparison to existing works . We unveil that the intrinsic positive sample noises exist broadly behind various mainstream SSL frameworks . To both justify and to mitigate the issue , we propose a latent variable framework that is capable improving the SSL optimization via automatic Unsupervised OuTlier Arbitration ( UOTA ) . We are also aware of a contemporary work in [ 51 ] that attempts to change the sampling distribution via assigning sample weights to each positive sample . But motivation in [ 51 ] is completely opposite to ours , as they oversample the region whenever a positive sample turns out to be remote to its query , while they still assume a perfect invariance on positive keys as critical supporting prerequisite . In [ 26 ] , a Mahalanobis distance-based confidence score is defined to modify sampling distribution for supervised learning tasks . Note our work differentiates from [ 26 ] in many aspects : we analyze the OOD issue particularly for unsupervised SSL frameworks , whereas [ 26 ] strongly hinges on data labels to compute confidence . We model the associated sampling ratio as a latent variable model which is exclusively suitable for the SSL problem , and is supported by rigorous analysis . Our model also naturally jettisons the need of any threshold as used in [ 26 ] . 3 Method . In this section , we reveal that OOD noise is inherently existing in positive view sampling pipelines for a broad range of mainstream SSL approaches . Most importantly , we showcase that these OOD samples are highly structured , and disrupt the downstream tasks with a non-negligible strength . We borrow basic analytic tool from importance sampling to investigate the source of OOD problem specifically for SSL frameworks . We identify an important bias-variance trade-off behind the sampling distribution options for positive views ( augmentations ) , and we analyze the impact of such bias-variance trade-off on the estimator MSE ( mean squared error ) via rigorous justification . Our model UOTA is motivated to adaptively balance the sampling distribution trade-off via a novel latent variable model , and the proposed method is agnostic to the form of SSL loss . We show that the model is able to effectively suppress the OOD sampling noise for positive samples . 3.1 Problem Formulation of SSL . Mainstream SSL approaches , especially contrastive learning methods , often rely on a series transformation functions xi , k = g ( xi , nk ) to produce multiple “ views ” xi , k of input instance xi . The { nk : k = 1 , . . . , K } is the support of an “ augmentation nuisances variable ” n , representing a group of transformations such as flipping , scaling , cropping , or other augmentations acting on xi via function g. Subscript k indexes all possible augmentations , i indicates the i-th instance ( each instance corresponds to a specific image input ) . Without labeling information , SSL relies on the invariance assumption so that deep feature extractor zi , k = f ( xi , k , θ ) is at least insensitive to such group of augmentation nuisances nk w.r.t . the network parameters θ . Existing work [ 5 , 7 , 18 , 33 ] show that , while the deep extractor f ( xi , k , θ ) is trained to learn the shared semantics from distorted views xi , k , such pretrained network parameters θ effectively benefit further downstream tasks . SSL approaches based on such pairwised instance invariance assumptions can be generally summarized as penalizing some form of inconsistency between paired of features zi , j , zi , k : Lθ ( xi , ( nj , nk ) ) = L ( zi , j , zi , k ) ( 1 ) where j , k indexes over the augmented views on instance xi . Without loss of generality , we expand the support of random variable n to pairwised { ( nj , nk ) : j , k = 1 , . . . , K } to incorporate pairs of augmentation actions . Loss Lθ ( xi , ( nj , nk ) ) represents such invariance loss under the pair of augmentation actions ( nj , nk ) . Here , the generic form of loss function L ( zi , j , zi , k ) imposes the desired cross-view invariance assumptions between zi , j and zi , k for the i-th instance . Take for instance , L ( zi , j , zi , k ) can be defined as contrastive loss in MoCo [ 18 ] , clustered loss as in SwAV [ 5 ] , or simply ` 2 norm as in BYOL [ 15 ] . Our idealized goal then is to sample from independent latent generative processes xi ∼ px and n ∼ p∗ such that zi , k = f ( xi , k , θ ) = f ( g ( xi , nk ) , θ ) captures the same semantics as from original xi . Assume we have access to such oracle distribution p∗ , and with equal sampling probability for xi , the objective function is : L∗ = Ex∼px , n∼p∗ [ Lθ ( x , n ) ] ≈ 1 N N∑ i=1 ∑ j , k L ( zi , j , zi , k ) p ∗ ( nj , nk ) . ( 2 ) Optimizing Eq . ( 2 ) then corresponds to learning an M-estimator [ 22 ] with regard to network parameters θ . 3.2 Sampling Distribution p∗ : a Heuristic . Augmented views are the bread and butter for self-supervised learning approaches . Intuitively , the network needs to see diverse augmentations from p∗ ( n ) in order to train for invariance . But in the meanwhile , the learning process needs to also avoid training on those xi , k having excessive distortions who have drastically deviated from the semantics of the original instance . There exists an implicit trade-off in our mind that controls the strength ( hyperparameter ) of the augmentation actions . The elusive definition on optimal distribution p∗ ( n ) therefore seems to be best considered as a constrained distribution offering most data variance subject to the least semantic deviation from the original instance . We illustrate this point in Fig . 1 . The left-most picture is the original instance input , whereas the two rows of smaller images display the augmented views generated via the multi-crop augmentation pipeline as in SwAV [ 5 ] . Apparently , some of augmented views are highlighting the bunny , whereas the others capture the straw only . Under such generative process of augmentations , there is a good chance that a pair of ( bunny , straw ) features are imposed to be close in feature space . Such noisy invariance might intuitively disrupt the optimization , leading to suboptimal downstream generalization ability . Actually , the work in [ 36 ] has theoretically discussed the generalization ability of contrasive pretraining approaches on downstream tasks : if a negative sample is not from a different class of the positive , the upperbound of error on downstream linear classification is provably guaranteed to increase . Similarly , we can derive from [ 36 ] via mild modification that if a pair of constructed positive views are not from the same semantic class , the upperbound of linear classification error in the downstream tasks would also increase . Nevertheless , in practice , while the optimal desired p∗ ( n ) remains unknown and elusive , we usually take it for granted that all generated augmentations are equally good , and we tentatively sample n from some heuristically chosen p̃ ( n ) , a surrogate distribution that needs not coincide with the desired p∗ ( n ) having the proper augmentation variance-bias trade-off . And instead of optimizing Eq . ( 2 ) , we often optimize : L̃ = Ex∼px , n∼p̃ [ Lθ ( x , n ) ] ≈ 1 N N∑ i=1 ∑ j , k L ( zi , j , zi , k ) p̃ ( nj , nk ) . ( 3 ) As per our discussion above , since we ultimately hope to sample from the desired p∗ ( n ) instead of the surrogate distribution p̃ , we rewrite Eq . ( 2 ) using the basic change of measure rule from the importance sampling technique , and yield : L∗ ≈ 1 N N∑ i=1 ∑ j , k L ( zi , j , zi , k ) p ∗ ( nj , nk ) ≈ 1 N N∑ i=1 ∑ j , k L ( zi , j , zi , k ) p̃ ( nj , nk ) · p∗ ( nj , nk ) p̃ ( nj , nk ) . ( 4 ) Comparing Eq . ( 4 ) and Eq . ( 3 ) , we notice that if we desire to optimize Eq . ( 2 ) , whereas we insist sampling from p̃ instead of p∗ , we should adjust for this change and compensate each loss term via the multiplicative likelihood ratio p∗ ( nj , nk ) /p̃ ( nj , nk ) , rather than optimizing Eq . ( 3 ) . This likelihood ratio detects how much chance we would miss the optimal sampling distribution if instead using the surrogate distribution p̃ . If p∗ = p̃ , then the objective function recovers the optimal sampling scheme as in Eq . ( 2 ) and no adjustment using likelihood ratio is needed . In practice , SSL approaches usually freeze the optimization on either feature zi , j or zi , k , while only the other one is optimized via backpropagation [ 15 , 18 ] . Therefore it is also reasonable to simplify the objective Lθ ( x , ( nj , nk ) ) to Lθ ( x , nj ) , and the likelihood ratio p∗ ( nj , nk ) /p̃ ( nj , nk ) to p∗ ( nj ) /p̃ ( nj ) . In addition , forn = Id which is the identity transformation , Lθ ( xi , Id ) degenerates to the objective function on the original instance xi without augmentation , we denote it as Lθ ( xi ) .

Summary: The paper suggests that the different augmentations used as a positive pair during self-supervised training often lack the same semantic structure as the original image, and thus should be treated as an OOD sample. Using such OOD samples, they believe can hurt the downstream task performance. The aim should be to choose augmentations that offer the most data variance but are subject to the least semantic deviation. For this, they propose an importance sampling approach to weigh the loss for different augmentations differently. Empirically they obtain improvement for different downstream tasks across different datasets.