Co-complexity: An Extended Perspective on Generalization Error

1 INTRODUCTION . In the context of supervised classification , a major factor for consideration is the generalization error of the classifier , i.e. , how good a classifier generalizes to test ( unseen ) data points . The notion of overfitting describes the case when the test error significantly exceeds the training error . Naturally , the objective for building a robust classifier entails the minimization of this generalization gap , to avoid overfitting . To that end , statistical studies on generalization error ( Blumer et al . ( 1989 ) ; Bartlett & Mendelson ( 2003 ) ) find that complexity measures on the classifier function space , F , often directly control the generalization gap of a classifier . Two prominent examples of such measures include the Rademacher Complexity Rm ( F ) ( Bartlett & Mendelson ( 2003 ) ) and the VC dimension ( Blumer et al . ( 1989 ) ) V C ( F ) . Both measures directly estimate the flexibility of a function space , i.e. , how likely is it for F to contain functions that can fit any random labelling over a set of data points . In this paper , we work with Rademacher complexity and propose extensions that provide a new perspective on generalization error . From a statistical perspective , the generalization gap can be understood through convergence bounds on the error function , i.e. , the expected deviation of the error function on the test data compared to the training data . Traditional generalization error bounds ( Bartlett & Mendelson , 2003 ) state that function complexity ( i.e. , Rm ( F ) ) directly corresponds to the generalization gap . Thus , higher Rm ( F ) usually leads to a greater generalization gap and slower convergence . Although the originalRm ( F ) was proposed for binary classification , similar results have been shown for multi-class settings and a larger variety of loss functions ( Xu et al. , 2016 ; Liao et al. , 2018 ) . Note thatRm ( F ) is over the entire function space and thus global in nature . Local forms of Rademacher complexity , which involve restricting the function space and lead to minimum error on the training data samples , have been proposed ( Bartlett et al. , 2005 ; 2002 ) . Apart from function complexity based measures , there is also considerable work which uses an information theoretic perspective ( Xu & Raginsky , 2017 ; Russo & Zou , 2020 ; Bu et al. , 2020 ; Haghifam et al. , 2020 ) in treating the subject . 1.1 WHY THE GROUND TRUTH LABEL GENERATING FUNCTION MATTERS . We define the label generating function for a classification problem as the function which generates the true labels for all possible datapoints . Note that most generalization error bounds , including the traditional ones , primarily are introspective in nature , i.e. , they consider the size and flexibility of the classifier ’ s function space F . The main direction proposed in this work is the investigation of the unknowability of the ground truth label generating functions ( LGF ) , using another function space which we call the generator space . The generalization error bounds in Bartlett & Mendelson ( 2003 ) state that the difference in test and training performance is roughly bounded above by the Rademacher Complexity of the classifier ’ s function space ( i.e. , Rm ( F ) ) . In other words , whatever the training error , the test error will always be likely to be greater by an amountRm ( F ) on average . We note that , in deriving the original bound , a major assumption is that the LGF g is fixed and knowable from the data . We now outline our main argument for taking the generator space into account . The LGF is indeed fixed , i.e. , there can not be two different ground truth label generating functions applicable to the same problem . However , our primary emphasis is on the fact that the true LGF will always be unknown , i.e. , for any finite training data containing data-label pairs ( zi , g ( zi ) ) , we would only truly know the output of the label generating function on the given training data samples . Only when we have infinite training data samples , the values of LGF are known at each z ∈ Rd . In this work , we denote the function space of all possible LGFs , within which the true LGF is contained , as the generator space . Note that the generator space arises due to the unknowability of the LGF . We show in this work that due to the generator space , the true generalization gap is greater than the Rademacher complexity of the classifier , and also depends on the Rademacher complexity of the generator . Note that the size of the generator space , which dictates its complexity , will be dependent on the amount of constraints that the LGF has : if the LGF has no constraints at all , then generator spaces are larger , whereas if the LGF is constrained to be smooth/invariant to many transformations , generator spaces are smaller ( as the set of functions which are smooth and invariant are also much smaller , for instance in vision ) . Let us consider the case where the LGF g has no constraints at all , i.e. , it is sampled from a generator space G which contains all possible functions g : Rd −→ { −1 , 1 } . In this case , the function g is expected to have no structure and behaves like a random function , and thus the expected test accuracy of any classifier will be 50 % ( i.e. , random chance ) . Therefore , even if the classifier function f ∈ F produces a very good fit on the training data and F happens to have a low complexity measure Rm ( F ) , the generalization performance will still be poor , as no knowledge of the LGF values on the unseen datapoints is available from the training samples . This is in contrast to the generalization error bounds based on Rademacher complexity , which would estimate that the classifier should have good generalization performance ( i.e. , low test error ) , as both Rm ( F ) and training error are low . Note that although a typical training dataset extracted from this LGF may be hard to fit using a low-complexity classifier F , there will be examples of training instances with non-zero probability on which a low-complexity classifier can have a good fit . The takeaway from this example is that the structure of the data ( here represented using the complexity of the generator space ) can additionally dictate whether a classifier can generalize . Note that , in this scenario , the expected generalization performance would be better if the LGF had more structure . Figure 1 illustrates the role of both generator and classifier spaces in generalization via the two example scenarios discussed earlier . In both examples , the same low-complexity classifier has a good fit on the training data , thusRm ( F ) is low . In example ( a ) , the LGF has no constraints ; while , in example ( b ) , the LGF has constraints such as local smoothness . In example ( a ) , the classifier clearly exhibits poor generalization performance on the test data . This agrees with our previous argument that if the LGF has no constraints , generalization is essentially impossible . In example ( b ) , the classifier shows much more robust generalization performance on the test data , due to the LGF having significantly more structure . Furthermore , it is intuitively clear that a classifier function space F which has a high overlap with the generator space G ( and therefore its constraints ) should yield good generalization performance . This shows , that in addition to the function spaces F and G individually affecting the generalization gap , the similarities between F and G are also important . Both of these perspectives play leading roles in our construction of generator-and-classifier aware complexity measures and the associated novel generalization error bounds . 1.2 RELATED WORK . To the best of our knowledge , the approach we are proposing to study generalization performance is novel and there is not much directly related work . Here , we describe some examples of works which discuss relevant concepts . The no free lunch theorem proposed in Wolpert & Macready ( 1997 ) indirectly sheds light on the behaviour of the LGFs . However , it does not incorporate ways to reduce variability in the LGFs by considering constraints related to the classification problem . Invariance constraints in learning algorithms were studied in Sokolic et al . ( 2017 ) , where the input space was factored into invariant transformations , similar to what we also do in this work . In doing so , the complexity of the data was indirectly explored , based on the number of invariant transformations present in the input space . However , the generalization bounds were derived with an assumption of perfectly invariant classifier function spaces , which is not applicable for CNNs and their variants ( as shown in Kauderer-Abrams ( 2017 ) ) . In another relevant study ( Jin et al . ( 2019 ) ) , a cover-complexity measure of a single dataset was proposed , and the generalization error of fully connected networks was analyzed with respect to the same . However , invariances in the dataset and the learning algorithm were neglected , i.e. , the similarities between the generator and the classifier spaces were not studied . 1.3 KEY CONTRIBUTIONS . The contributions of this work are as follows : 1 . We propose a novel complexity measure between the classifier function space F and the generator function space G called co-complexity , which we use to derive new , more accurate global estimates of generator-aware generalization error bounds ( Theorem 3 and 4 ) . Co-complexity considers not only the complexity of the generator and classifier function spaces , but also the similarities between them . Doing so allows for the a more exhaustive look into generalization error . 2 . We decompose co-complexity into two different measures of complexity , invariance co-complexity and dissociation co-complexity , which are used to derive new generalization error bounds , including bounds on the expected training error ( Theorem 6 ) . We find that reducing invariance co-complexity while keeping the dissociation co-complexity unchanged , helps reduce the generalization gap ( low variance ) while maintaining low training error ( low bias ) . This emphasizes the importance of having classifiers that share invariance properties with generator spaces , e.g. , rotation-invariant CNNs ( Cohen & Welling ( 2016b ) ) on MNIST-Rot ( Larochelle et al . ( 2007 ) ) . 3 . We present empirical validation of co-complexity measures of CNN and its scale-equivariant and rotation-equivariant counterparts ( SE-CNN in Sosnovik et al . ( 2019 ) , RE-CNN in Cohen & Welling ( 2016b ) ) , which explains their superior generalization ability compared to MLPs . Our proposed error bounds are easily specialized to the case where ground truth label function is fixed , leading to potentially tighter generalization error bounds ( see Appendix A ) . Although our proposed measures are global in nature , local variants can be derived via similar extensions used in Bartlett et al . ( 2005 ) . 2 DEFINITIONS . Assume that we have m number of d-dimensional i.i.d sampled instances S = [ z1 , z2 , .. , zm ] drawn from some distribution P , and another set ofm i.i.d sampled instances S′ = [ z′1 , z ′ 2 , .. , z ′ m ] also drawn from P . Define σ = [ σ1 , σ2 , .. , σm ] where σi are i.i.d random variables following the Rademacher distribution ( Pr ( σi = +1 ) = Pr ( σi = −1 ) = 0.5 ) . F and G are two function spaces from Rd −→ { −1 , 1 } and we assume that they are defined at all points in Rd . In the context of our problem , F will be the classifier ’ s function space , whereas G will be the generator space . Rademacher Complexity ( Bartlett & Mendelson ( 2003 ) ) : First , we provide the definition of Rademacher Complexity : Rm ( F ) = E σ , S [ sup f∈F ( 1 m m∑ i=1 f ( zi ) σi ) ] ( 1 ) It can be seen thatRm ( F ) indicates the noisy-label fitting ability of the classifier ’ s function space F . Correlated Rademacher Complexity : We propose a modified form of the original Rademacher complexity , called Correlated Rademacher Complexity , RCm ( F ) , which is defined as follows : RCm ( F ) = 1 2 × E σ , S , S′ [ sup f∈F ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) σi ) ] ( 2 ) We have 0 ≤ RCm ( F ) ≤ 1/2 . We also show thatRCm ( F ) ≤ Rm ( F ) ( see Appendix C.1 ) . It has been argued thatRm ( F ) ( and thereforeRCm ( F ) ) can be considered as ” entropy ” measures of the entire function space F ( Anguita et al . ( 2014 ) ) . Note that , likeRm ( F ) , RCm ( F ) is also eventually depends on the noisy label fitting ability of F , but computes it via ability of the function space to assign the same or a different label to two random points zi and z′i ( via f ( zi ) f ( z ′ i ) ) . Thus we denote it as the correlated Rademacher complexity of F . Co-Complexity : Now we propose various complexity measures between two separate function spaces F and G. Similar toRm ( F ) andRCm ( F ) , these measures assess the noisy label fitting abilities of the union of the function spaces F and G. In doing so , these measures compute a joint-entropy like metric over the two function spaces . First , we define the Co-complexity between F and G as follows . Rm ( F , G ) = 1 2 × E σ , S , S′ [ sup f∈F , g∈G ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] ( 3 ) Some of the properties of co-complexity are as follows : P1 Rm ( F , G ) = Rm ( G , F ) , i.e. , co-complexity is symmetric . P2 Rm ( F , G ) ≥ RCm ( F ) and Rm ( F , G ) ≥ RCm ( G ) , i.e. , the co-complexity between F and G is always greater than the individual correlated Rademacher complexities of F and G. P3 Rm ( F , G ) ≤ RCm ( F ) +RCm ( G ) ≤ Rm ( F ) +Rm ( G ) , i.e. , the co-complexity between F and G is upper bounded by the sum of the Rademacher complexities of F and G. P4 Rm ( F , G ) behaves like a joint-entropy measure ofF and G. To see this , let us define Im ( F , G ) = RCm ( F ) +RCm ( G ) − Rm ( F , G ) , called the mutual co-complexity between F and G. We later find that Im ( F , G ) behaves like a mutual-information measure . This , coupled with the fact that RCm ( F ) andRCm ( G ) can be construed as entropy measures of F and G , implies the result . Invariance Co-Complexity : Next , we quantify some of the properties of the ground truth generator space G in terms of its invariance transformations . We define the Invariance Classes of G as follows : IC ( G ) = { τ1 ( . , θ1 ) , τ2 ( . , θ2 ) , ... , τn ( . , θn ) } , ( 4 ) where { τi ( . , θi ) |i = 1 , 2 , · · · , n } are functions from Rd −→ Rd , and θi represents the extent of the transformation τi . Each θi ∈ R is a scalar and takes on a set of admissible values depending on the transformation ( possibly infinite ) . The functions are constrained such that , ∀τ ∈ IC ( G ) , ∀g ∈ G , g ( τ ( z ) , t ) = g ( z ) for all data points z ∈ Rd , and for all admissible values of the transformation parameter t. Also , note that setting θi = 0 leads to the identity function , i.e. , τi ( z , 0 ) = 1 for all z ∈ Rd and all i . Based on the above we define a transformation-indicator function I ( z , zi ) , such that I ( z , zi ) = 1 if z = τ1 ( . , t1 ) ◦ τ2 ( . , t2 ) ◦ .... ◦ τn ( . , tn ) ( zi ) , for a certain t1 , t2 , ... tn and otherwise I ( z , zi ) = 0 . Additionally , with respect to a generator space G , we also define the invariance extended set of a datapoint zi ∈ Rd as follows : IG ( zi ) = { z ∈ Rd | I ( z , zi ) = 1 } ( 5 ) Note that that invariance extended set of z will always have the same ground truth label as z . Next , we define the Invariance Co-complexity between two function spaces F and G as follows : RIm ( F , G ) = 1 2 × E S , S′ [ sup f∈F ( 1− ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) ) ) ] , where z′i ∼ P ( z ) I ( z , zi ) , ∀i . ( 6 ) Note that RIm ( F , G ) ≤ 1 . Each data point z′i in S′ is contained within the invariance extended set IG ( zi ) of zi , and is sampled according to the un-normalized distribution P ( z ) I ( z , zi ) over z ∈ Rd . The invariance co-complexity between F and G indicates the degree to which F obeys the invariance transformations within G. For instance , a low RIm ( F , G ) would indicate that f ( zi ) f ( z′i ) = 1 ( i.e. , f ( zi ) = f ( z ′ i ) ) for most zi and z ′ i which are related via some invariance transformation in G. Dissociation Co-Complexity : We now define the Dissociation Co-complexity between F and G when the corresponding datapoints in S and S′ are not related by any invariance transformation . That is , for all zi ∈ S and z′i ∈ S′ , z′i 6∈ IG ( zi ) , and is sampled from the distribution P ( z ) ( 1− I ( z , zi ) ) over z ∈ Rd . The dissociation co-complexity can then be defined as : RDm ( F , G ) = 1 2 × E σ , S , S′ [ sup f∈F ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) σi ) ] , where z′i ∼ P ( z ) ( 1− I ( z , zi ) ) , ∀i . ( 7 ) Note thatRDm ( F , G ) measures the average flexibility of F in its label assignment to any two points zi and z′i which are not related via any invariance transformation in G. Thus , a largerRDm ( F , G ) would indicate that the classifier function space F is able to easily assign separate labels to datapoints which are not related via any invariance transformation in G. We also define a variant of RDm ( F , G ) , in which S contains only one instance , instead of m instances , denoted asRD,1m ( F , G ) . RD,1m ( F , G ) is defined as follows . RD,1m ( F , G ) = 1 2 × E σ , z0∼P , S′ [ sup f∈F ( 1 m m∑ i=1 f ( z0 ) f ( z ′ i ) σi ) ] , where z′i ∼ P ( z ) ( 1− I ( z , z0 ) ) , ∀i . ( 8 ) Note thatRDm ( F , G ) ≤ 0.5 andRD,1m ( F , G ) ≤ 0.5 . Rademacher Smoothness : For our final result in Theorem 6 , we assume that the function space F is Rademacher smooth w.r.t . G. We define Rademacher Smoothness as follows . If F is Rademacher smooth w.r.t . G , then any quantity of the form ES , S′ [ supf∈F ( 1 m ∑m i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] or ES , S′ [ supf∈F , g∈G ( 1 m ∑m i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] , where the expectation is over datapoint permutations S , S′ , should lie between the two following cases : ( i ) S′ ∈ IG ( S ) ( i.e. , z′i ∼ P ( z ) ( I ( z , zi ) ) for all i ) and ( ii ) S′ /∈ IG ( S ) ( i.e. , z′i ∼ P ( z ) ( 1− I ( z , zi ) ) for all i ) . Then , for some non-zero value of α , 0 ≤ α ≤ 1 , the following holds : E S , S′ [ sup f∈F 1 m ( m∑ i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] = ( α ) E S , S′ , S′∈IG ( S ) [ sup f∈F ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] + ( 1− α ) E S , S′ , S′ /∈IG ( S ) [ sup f∈F ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] . ( 9 ) Similarly , the same applies to ES , S′ [ supf∈F , g∈G ( 1 m ∑m i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] .

Summary: This paper studied a novel perspective on generalization error bounds, by introducing the "label generating function "(LGF). Several new complexity measures (correlated Rademacher complexity, co-complexity, invariance co-complexity, dissociation co-complexity, Rademacher smoothness) were proposed. The properties of the measures and generalization error bound with respect to these complexity measures are studied.