For self-supervised learning, Rationality implies generalization, provably

1 INTRODUCTION . The current standard approach for classification is “ end-to-end supervised learning ” where one fits a complex ( e.g. , a deep neural network ) classifier to the given training set ( Tan & Le , 2019 ; He et al. , 2016 ) . However , modern classifiers are heavily over parameterized , and as demonstrated by Zhang et al . ( 2017 ) , can fit 100 % of their training set even when given random labels as inputs ( in which case test performance is no better than chance ) . Hence , the training performance of such methods is by itself no indication of their performance on new unseen test points . In this work , we study a different class of supervised learning procedures that have recently attracted significant interest . These classifiers are obtained by : ( i ) performing pre-training with a selfsupervised task ( i.e. , without labels ) to obtain a complex representation of the data points , and then ( ii ) fitting a simple ( e.g. , linear ) classifier on the representation and the labels . Such “ Self-Supervised + Simple ” ( SSS for short ) algorithms are commonly used in natural language processing tasks ( Devlin et al. , 2018 ; Brown et al. , 2020 ) , and have recently found uses in other domains as well ( Ravanelli et al. , 2020 ; Liu et al. , 2019 ) . Compared to standard “ end-to-end supervised learning ” , SSS algorithms have several practical advantages . In particular , SSS algorithms can incorporate additional unlabeled data , the representation obtained can be useful for multiple downstream tasks , and they can have improved out-of-distribution performance ( Hendrycks et al. , 2019 ) . Moreover , recent works show that even without additional unlabeled data , SSS algorithms can get close to state-of-art accuracy in several classification tasks ( Chen et al. , 2020b ; He et al. , 2020 ; Misra & Maaten , 2020 ; ∗Equal contribution . Email : { ybansal , galkaplun } @ g.harvard.edu †Email : b @ boazbarak.org . Tian et al. , 2019 ) . For instance , SimCLRv2 ( Chen et al. , 2020b ) achieves 79.8 % top-1 performance on ImageNet with a variant of ResNet-152 , on par with the end-to-end supervised accuracy of this architecture at 80.5 % . We show that SSS algorithms have another advantage over standard supervised learning—they often have a small generalization gap between their train and test accuracy , and we prove non-vacuous bounds on this gap . We stress that SSS algorithms use over-parameterized models to extract the representation , and reuse the same training data to learn a simple classifier on this representation . Thus , the final classifier they produce has high complexity by most standard measures , and it is by no means apriori evident that their generalization gap will be small . Our bound is obtained by first noting that the generalization gap of every training algorithm is bounded by the sum of three quantities , which we name the Robustness gap , Rationality gap , and Memorization gap ( we call this the RRM bound , see Fact I ) . We now describe these gaps at a high level , deferring the formal definitions to Section 2 . All three gaps involve comparison with a setting where we inject label noise by replacing a small fraction η of the labels with random values . The robustness gap corresponds to the amount by which training performance degrades by noise injection . That is , it equals the difference between the standard expected training accuracy ( with no label noise ) and the expected training accuracy in the noisy setting ; in both cases , we measure accuracy with respect to the original ( uncorrupted ) labels . The robustness gap is nearly always small , and sometimes provably so ( see Section 3 ) . The rationality gap corresponds to the difference between performance on the noisy training samples ( on which the training algorithm gets the wrong label ) and test samples ( on which it doesn ’ t get any label at all ) , again with respect to uncorrupted labels . An optimal Bayesian procedure would have zero rationality gap , and indeed this gap is typically zero or small in practice . Since it is a nonstandard quantity , We discuss the rationality gap in Section 3.1 , and explain assuming it is small is both well-founded and does not trivialize the question of generalization . The memorization gap , which often accounts for the lion ’ s share of the generalization gap , corresponds to the difference in the noisy experiment between the training accuracy on the entire train set and the training accuracy on the samples that received the wrong label ( both measured with respect to uncorrupted labels ) . The memorization gap can be thought of as quantifying the extent to which the classifier can “ memorize ” noisy labels , or act differently on the noisy points compared to the overall train set . The memorization gap is large in standard “ end-to-end supervised training ” . In contrast , our main theoretical result is that for SSS algorithms , the memorization gap is small if the simple classifier has small complexity , independently of the complexity of the representation . As long as the simple classifier is under-parameterized ( i.e. , its complexity is asymptotically smaller than the sample size ) , our bound on the memorization gap tends to zero . When combined with small rationality and robustness , we get concrete non-vacuous generalization bounds for various SSS algorithms on the CIFAR-10 and ImageNet datasets ( see Figures 1 and 4 ) . In a nutshell , our results are the following : Theoretical contributions .. 1 . Our main theoretical result ( Theorem II ) is that the memorization gap of an SSS algorithm is bounded byO ( √ C/n ) whereC is the complexity of the simple classifier produced in the “ simple fit ” stage . This bound is oblivious to the complexity of the representation produced in the pre-training and does not make any assumptions on the relationship between the representation learning method and the supervised learning task . One way to interpret this result is that we give a rigorous bound on the generalization gap of SSS algorithms , under the assumptions that the robustness and rationality gaps are bounded by some small constant ( e.g. , 5 % ) . As mentioned below , these assumptions hold widely in practice across many different classifiers . Moreover , these assumptions are nontrivial and do not “ assume away the difficulty ” . Indeed , there are many natural examples of training algorithms for which these assumptions hold but the generalization gap is large . Last , making some assumptions is necessary for a generalization bound to hold for SSS algorithms ; see Remark 3.1 and Appendix E. 2 . We also give a theoretical justification for the assumption of a small rationality gap , by proving that a positive rationality gap corresponds to “ leaving performance on the table ” , in the sense that we can transform a learning procedure with a large rationality gap into a procedure with better test performance ( Theorem 3.2 ) . Empirical contributions . We complement the theoretical results above with an extensive empirical study of several SSS and end-to-end algorithms on both the CIFAR-10 and ImageNet datasets . 1 . We study several top-performing SSS architectures , and show that they all exhibit relatively small generalization gaps on both CIFAR-10 and ImageNet . We stress that we consider the case where the same data is used for both representation learning and classification , and hence it is by no means a-priori obvious that these algorithms should have small generalization gaps . See Figures 1 and 4 for sample results and Section 4 for more details . 2 . We also show that the results of Zhang et al . ( 2017 ) do not replicate to SSS algorithms , in the sense that such algorithms , despite using an over-parameterized representation , are not able to fit random label noise . 3 . We understake an empirical study of the robustness , rationality , and memorization gaps for both SSS and end-to-end supervised learning algorithms . We show that the robustness and rationality gaps are small for all these algorithms , while the memorization gap is small for SSS algorithms but can be large for end-to-end supervised learning . We show that the RRM bound is typically non-vacuous , and in fact , often close to tight , for a variety of SSS algorithms on the CIFAR-10 and ImageNet datasets , including SimCLR ( which achieves test errors close to its supervised counterparts ) . 4 . We demonstrate that replacing the memorization gap with the upper bound of Theorem II yields a non-vacuous generalization bound for a variety of SSS algorithms on CIFAR-10 and ImageNet . Moreover , this bound gets tighter with more data augmentation . Related Work . There are many works on generalization bounds for supervised learning ( e.g. , Golowich et al . ( 2018 ) ; Neyshabur et al . ( 2017 ) ; Bartlett et al . ( 2017 ) ; Dziugaite & Roy ( 2017 ) ; Neyshabur et al . ( 2018 ) ; Cao & Gu ( 2019 ) , and references therein ) . The related work section of Arora et al . ( 2019 ) contains an extensive discussion of such bounds , and why more often than not the assumptions used do not hold in practice . Indeed , many such bounds give vacuous guarantees for modern architectures ( such as the ones considered in this paper ) that have the capacity to memorize their entire training set ( Zhang et al. , 2017 ) . Some non-vacuous bounds are known ; e.g. , Zhou et al . ( 2019 ) gave a 96.5 % bound on the error of MobileNet on ImageNet . Belkin et al . ( 2019 ) ; Nagarajan & Kolter ( 2019 ) showed some barriers for generalization gaps for standard end-to-end supervised learning . Similarly , standard approaches such as Rademacher complexity can not directly bound SSS algorithms ’ generalization gap ( see Remark 3.1 ) . Recently , Saunshi et al . ( 2019 ) and Lee et al . ( 2020 ) gave generalization bounds for self-supervised based classifiers . The two works considered special cases of SSS algorithms , such as contrastive learning and pre-text tasks . Both works make strong statistical assumptions of ( exact or approximate ) conditional independence relating the pre-training and classification tasks . For example , if the pre-training task is obtained by splitting a given image x into two pieces ( x1 , x2 ) and predicting x2 from x1 , then Lee et al . ( 2020 ) ’ s results require x1 and x2 to be approximately independent conditioned on their class y . However , in many realistic cases , the two parts of the same image will share a significant amount of information not explained by the label . Our work applies to general SSS algorithms without such statistical assumptions , at the expense of assuming bounds on the robustness and rationality gaps . There have been works providing rigorous bounds on the robustness gap or related quantities ( See Section 3. ) . However , as far as we know , the rationality gap has not been explicitly defined or studied before . We provide a brief exposition of the various types of SSS methods in Section 4 , and a more detailed discussion in Appendix D.1 . Paper Organization . Section 2 contains formal definitions and statements of our results . Section 3 provides an overview of prior work and our new results on the three gaps of the RRM bound . In Section 4 , we describe our experimental setup and detail our empirical results . Section 5 concludes the paper and discusses important open questions . We defer proofs and additional experimental results to the appendix . Appendix B contains the proof of Theorem II , while Appendix C contains the proof of Theorem 3.2 . Appendix D fully details our experimental setup.1 Notation . We use capital letters ( e.g. , X ) for random variables , lower case letters ( e.g. , x ) for a single value , and bold font ( e.g. , x ) for tuples ( which will typically have dimension corresponding to the number of samples , denoted by n ) . We use xi for the i-th element of the tuple x . We use calligraphic letters ( e.g. , X , D ) for both sets and distributions .

Summary: The present paper aims to understand the generalization capability of self-supervised learning algorithms that fine-tune a simple linear classifier to the labels. Analyzing generalization in this case is challenging due to a data re-use problem: the same training data that is used for self-supervised learning is also used to fit the labels. The paper addresses this issue by implicitly conditioning on the training covariates x and then deriving generalization bounds that depend only on (hypothetical) noise to the labels y. The paper show that, empirically, the dominant factor in generalization error is a certain quantity called the "memorization gap", which can also be upper-bounded via theoretical analysis (the theoretical bound seems to be loose by about a factor of 4 compared to the empirical measurement, but is still non-vacuous in many cases). Interestingly, this is *not* the case for standard supervised learning, likely to the higher-complexity models used to fit the labels; in that case the memorization gap is high, but a different gap (called the "rationality gap") is large in magnitude but negative.