For self-supervised learning, Rationality implies generalization, provably

1 INTRODUCTION . The current standard approach for classification is “ end-to-end supervised learning ” where one fits a complex ( e.g. , a deep neural network ) classifier to the given training set ( Tan & Le , 2019 ; He et al. , 2016 ) . However , modern classifiers are heavily over parameterized , and as demonstrated by Zhang et al . ( 2017 ) , can fit 100 % of their training set even when given random labels as inputs ( in which case test performance is no better than chance ) . Hence , the training performance of such methods is by itself no indication of their performance on new unseen test points . In this work , we study a different class of supervised learning procedures that have recently attracted significant interest . These classifiers are obtained by : ( i ) performing pre-training with a selfsupervised task ( i.e. , without labels ) to obtain a complex representation of the data points , and then ( ii ) fitting a simple ( e.g. , linear ) classifier on the representation and the labels . Such “ Self-Supervised + Simple ” ( SSS for short ) algorithms are commonly used in natural language processing tasks ( Devlin et al. , 2018 ; Brown et al. , 2020 ) , and have recently found uses in other domains as well ( Ravanelli et al. , 2020 ; Liu et al. , 2019 ) . Compared to standard “ end-to-end supervised learning ” , SSS algorithms have several practical advantages . In particular , SSS algorithms can incorporate additional unlabeled data , the representation obtained can be useful for multiple downstream tasks , and they can have improved out-of-distribution performance ( Hendrycks et al. , 2019 ) . Moreover , recent works show that even without additional unlabeled data , SSS algorithms can get close to state-of-art accuracy in several classification tasks ( Chen et al. , 2020b ; He et al. , 2020 ; Misra & Maaten , 2020 ; ∗Equal contribution . Email : { ybansal , galkaplun } @ g.harvard.edu †Email : b @ boazbarak.org . Tian et al. , 2019 ) . For instance , SimCLRv2 ( Chen et al. , 2020b ) achieves 79.8 % top-1 performance on ImageNet with a variant of ResNet-152 , on par with the end-to-end supervised accuracy of this architecture at 80.5 % . We show that SSS algorithms have another advantage over standard supervised learning—they often have a small generalization gap between their train and test accuracy , and we prove non-vacuous bounds on this gap . We stress that SSS algorithms use over-parameterized models to extract the representation , and reuse the same training data to learn a simple classifier on this representation . Thus , the final classifier they produce has high complexity by most standard measures , and it is by no means apriori evident that their generalization gap will be small . Our bound is obtained by first noting that the generalization gap of every training algorithm is bounded by the sum of three quantities , which we name the Robustness gap , Rationality gap , and Memorization gap ( we call this the RRM bound , see Fact I ) . We now describe these gaps at a high level , deferring the formal definitions to Section 2 . All three gaps involve comparison with a setting where we inject label noise by replacing a small fraction η of the labels with random values . The robustness gap corresponds to the amount by which training performance degrades by noise injection . That is , it equals the difference between the standard expected training accuracy ( with no label noise ) and the expected training accuracy in the noisy setting ; in both cases , we measure accuracy with respect to the original ( uncorrupted ) labels . The robustness gap is nearly always small , and sometimes provably so ( see Section 3 ) . The rationality gap corresponds to the difference between performance on the noisy training samples ( on which the training algorithm gets the wrong label ) and test samples ( on which it doesn ’ t get any label at all ) , again with respect to uncorrupted labels . An optimal Bayesian procedure would have zero rationality gap , and indeed this gap is typically zero or small in practice . Since it is a nonstandard quantity , We discuss the rationality gap in Section 3.1 , and explain assuming it is small is both well-founded and does not trivialize the question of generalization . The memorization gap , which often accounts for the lion ’ s share of the generalization gap , corresponds to the difference in the noisy experiment between the training accuracy on the entire train set and the training accuracy on the samples that received the wrong label ( both measured with respect to uncorrupted labels ) . The memorization gap can be thought of as quantifying the extent to which the classifier can “ memorize ” noisy labels , or act differently on the noisy points compared to the overall train set . The memorization gap is large in standard “ end-to-end supervised training ” . In contrast , our main theoretical result is that for SSS algorithms , the memorization gap is small if the simple classifier has small complexity , independently of the complexity of the representation . As long as the simple classifier is under-parameterized ( i.e. , its complexity is asymptotically smaller than the sample size ) , our bound on the memorization gap tends to zero . When combined with small rationality and robustness , we get concrete non-vacuous generalization bounds for various SSS algorithms on the CIFAR-10 and ImageNet datasets ( see Figures 1 and 4 ) . In a nutshell , our results are the following : Theoretical contributions .. 1 . Our main theoretical result ( Theorem II ) is that the memorization gap of an SSS algorithm is bounded byO ( √ C/n ) whereC is the complexity of the simple classifier produced in the “ simple fit ” stage . This bound is oblivious to the complexity of the representation produced in the pre-training and does not make any assumptions on the relationship between the representation learning method and the supervised learning task . One way to interpret this result is that we give a rigorous bound on the generalization gap of SSS algorithms , under the assumptions that the robustness and rationality gaps are bounded by some small constant ( e.g. , 5 % ) . As mentioned below , these assumptions hold widely in practice across many different classifiers . Moreover , these assumptions are nontrivial and do not “ assume away the difficulty ” . Indeed , there are many natural examples of training algorithms for which these assumptions hold but the generalization gap is large . Last , making some assumptions is necessary for a generalization bound to hold for SSS algorithms ; see Remark 3.1 and Appendix E. 2 . We also give a theoretical justification for the assumption of a small rationality gap , by proving that a positive rationality gap corresponds to “ leaving performance on the table ” , in the sense that we can transform a learning procedure with a large rationality gap into a procedure with better test performance ( Theorem 3.2 ) . Empirical contributions . We complement the theoretical results above with an extensive empirical study of several SSS and end-to-end algorithms on both the CIFAR-10 and ImageNet datasets . 1 . We study several top-performing SSS architectures , and show that they all exhibit relatively small generalization gaps on both CIFAR-10 and ImageNet . We stress that we consider the case where the same data is used for both representation learning and classification , and hence it is by no means a-priori obvious that these algorithms should have small generalization gaps . See Figures 1 and 4 for sample results and Section 4 for more details . 2 . We also show that the results of Zhang et al . ( 2017 ) do not replicate to SSS algorithms , in the sense that such algorithms , despite using an over-parameterized representation , are not able to fit random label noise . 3 . We understake an empirical study of the robustness , rationality , and memorization gaps for both SSS and end-to-end supervised learning algorithms . We show that the robustness and rationality gaps are small for all these algorithms , while the memorization gap is small for SSS algorithms but can be large for end-to-end supervised learning . We show that the RRM bound is typically non-vacuous , and in fact , often close to tight , for a variety of SSS algorithms on the CIFAR-10 and ImageNet datasets , including SimCLR ( which achieves test errors close to its supervised counterparts ) . 4 . We demonstrate that replacing the memorization gap with the upper bound of Theorem II yields a non-vacuous generalization bound for a variety of SSS algorithms on CIFAR-10 and ImageNet . Moreover , this bound gets tighter with more data augmentation . Related Work . There are many works on generalization bounds for supervised learning ( e.g. , Golowich et al . ( 2018 ) ; Neyshabur et al . ( 2017 ) ; Bartlett et al . ( 2017 ) ; Dziugaite & Roy ( 2017 ) ; Neyshabur et al . ( 2018 ) ; Cao & Gu ( 2019 ) , and references therein ) . The related work section of Arora et al . ( 2019 ) contains an extensive discussion of such bounds , and why more often than not the assumptions used do not hold in practice . Indeed , many such bounds give vacuous guarantees for modern architectures ( such as the ones considered in this paper ) that have the capacity to memorize their entire training set ( Zhang et al. , 2017 ) . Some non-vacuous bounds are known ; e.g. , Zhou et al . ( 2019 ) gave a 96.5 % bound on the error of MobileNet on ImageNet . Belkin et al . ( 2019 ) ; Nagarajan & Kolter ( 2019 ) showed some barriers for generalization gaps for standard end-to-end supervised learning . Similarly , standard approaches such as Rademacher complexity can not directly bound SSS algorithms ’ generalization gap ( see Remark 3.1 ) . Recently , Saunshi et al . ( 2019 ) and Lee et al . ( 2020 ) gave generalization bounds for self-supervised based classifiers . The two works considered special cases of SSS algorithms , such as contrastive learning and pre-text tasks . Both works make strong statistical assumptions of ( exact or approximate ) conditional independence relating the pre-training and classification tasks . For example , if the pre-training task is obtained by splitting a given image x into two pieces ( x1 , x2 ) and predicting x2 from x1 , then Lee et al . ( 2020 ) ’ s results require x1 and x2 to be approximately independent conditioned on their class y . However , in many realistic cases , the two parts of the same image will share a significant amount of information not explained by the label . Our work applies to general SSS algorithms without such statistical assumptions , at the expense of assuming bounds on the robustness and rationality gaps . There have been works providing rigorous bounds on the robustness gap or related quantities ( See Section 3. ) . However , as far as we know , the rationality gap has not been explicitly defined or studied before . We provide a brief exposition of the various types of SSS methods in Section 4 , and a more detailed discussion in Appendix D.1 . Paper Organization . Section 2 contains formal definitions and statements of our results . Section 3 provides an overview of prior work and our new results on the three gaps of the RRM bound . In Section 4 , we describe our experimental setup and detail our empirical results . Section 5 concludes the paper and discusses important open questions . We defer proofs and additional experimental results to the appendix . Appendix B contains the proof of Theorem II , while Appendix C contains the proof of Theorem 3.2 . Appendix D fully details our experimental setup.1 Notation . We use capital letters ( e.g. , X ) for random variables , lower case letters ( e.g. , x ) for a single value , and bold font ( e.g. , x ) for tuples ( which will typically have dimension corresponding to the number of samples , denoted by n ) . We use xi for the i-th element of the tuple x . We use calligraphic letters ( e.g. , X , D ) for both sets and distributions .

Summary: This paper gives a new perspective on generalization, motivated by the success of self-supervised learning, especially on noisy data. They view the generalization error as consisting of 3 independent components: robustness, rationality and memorization. Informally, robustness measures the degradation in training accuracy due to the addition of noise. rationality measures the gap between noisy training and test accuracy and memorization is the gap between the training error on the uncorrupted and corrupted examples. The main point of the paper is that the memorization gap is smaller for self-supervised, simple algorithms compared to full supervised algorithms. They prove that when the noise is defined by a small fraction of labels being randomly flipped, then the memorization error of such simple algorithms can be bounded in terms of their information-theoretic complexity, independent of the representation they produce. (The proof is simple once the definitions are set up properly).