Co-complexity: An Extended Perspective on Generalization Error

1 INTRODUCTION . In the context of supervised classification , a major factor for consideration is the generalization error of the classifier , i.e. , how good a classifier generalizes to test ( unseen ) data points . The notion of overfitting describes the case when the test error significantly exceeds the training error . Naturally , the objective for building a robust classifier entails the minimization of this generalization gap , to avoid overfitting . To that end , statistical studies on generalization error ( Blumer et al . ( 1989 ) ; Bartlett & Mendelson ( 2003 ) ) find that complexity measures on the classifier function space , F , often directly control the generalization gap of a classifier . Two prominent examples of such measures include the Rademacher Complexity Rm ( F ) ( Bartlett & Mendelson ( 2003 ) ) and the VC dimension ( Blumer et al . ( 1989 ) ) V C ( F ) . Both measures directly estimate the flexibility of a function space , i.e. , how likely is it for F to contain functions that can fit any random labelling over a set of data points . In this paper , we work with Rademacher complexity and propose extensions that provide a new perspective on generalization error . From a statistical perspective , the generalization gap can be understood through convergence bounds on the error function , i.e. , the expected deviation of the error function on the test data compared to the training data . Traditional generalization error bounds ( Bartlett & Mendelson , 2003 ) state that function complexity ( i.e. , Rm ( F ) ) directly corresponds to the generalization gap . Thus , higher Rm ( F ) usually leads to a greater generalization gap and slower convergence . Although the originalRm ( F ) was proposed for binary classification , similar results have been shown for multi-class settings and a larger variety of loss functions ( Xu et al. , 2016 ; Liao et al. , 2018 ) . Note thatRm ( F ) is over the entire function space and thus global in nature . Local forms of Rademacher complexity , which involve restricting the function space and lead to minimum error on the training data samples , have been proposed ( Bartlett et al. , 2005 ; 2002 ) . Apart from function complexity based measures , there is also considerable work which uses an information theoretic perspective ( Xu & Raginsky , 2017 ; Russo & Zou , 2020 ; Bu et al. , 2020 ; Haghifam et al. , 2020 ) in treating the subject . 1.1 WHY THE GROUND TRUTH LABEL GENERATING FUNCTION MATTERS . We define the label generating function for a classification problem as the function which generates the true labels for all possible datapoints . Note that most generalization error bounds , including the traditional ones , primarily are introspective in nature , i.e. , they consider the size and flexibility of the classifier ’ s function space F . The main direction proposed in this work is the investigation of the unknowability of the ground truth label generating functions ( LGF ) , using another function space which we call the generator space . The generalization error bounds in Bartlett & Mendelson ( 2003 ) state that the difference in test and training performance is roughly bounded above by the Rademacher Complexity of the classifier ’ s function space ( i.e. , Rm ( F ) ) . In other words , whatever the training error , the test error will always be likely to be greater by an amountRm ( F ) on average . We note that , in deriving the original bound , a major assumption is that the LGF g is fixed and knowable from the data . We now outline our main argument for taking the generator space into account . The LGF is indeed fixed , i.e. , there can not be two different ground truth label generating functions applicable to the same problem . However , our primary emphasis is on the fact that the true LGF will always be unknown , i.e. , for any finite training data containing data-label pairs ( zi , g ( zi ) ) , we would only truly know the output of the label generating function on the given training data samples . Only when we have infinite training data samples , the values of LGF are known at each z ∈ Rd . In this work , we denote the function space of all possible LGFs , within which the true LGF is contained , as the generator space . Note that the generator space arises due to the unknowability of the LGF . We show in this work that due to the generator space , the true generalization gap is greater than the Rademacher complexity of the classifier , and also depends on the Rademacher complexity of the generator . Note that the size of the generator space , which dictates its complexity , will be dependent on the amount of constraints that the LGF has : if the LGF has no constraints at all , then generator spaces are larger , whereas if the LGF is constrained to be smooth/invariant to many transformations , generator spaces are smaller ( as the set of functions which are smooth and invariant are also much smaller , for instance in vision ) . Let us consider the case where the LGF g has no constraints at all , i.e. , it is sampled from a generator space G which contains all possible functions g : Rd −→ { −1 , 1 } . In this case , the function g is expected to have no structure and behaves like a random function , and thus the expected test accuracy of any classifier will be 50 % ( i.e. , random chance ) . Therefore , even if the classifier function f ∈ F produces a very good fit on the training data and F happens to have a low complexity measure Rm ( F ) , the generalization performance will still be poor , as no knowledge of the LGF values on the unseen datapoints is available from the training samples . This is in contrast to the generalization error bounds based on Rademacher complexity , which would estimate that the classifier should have good generalization performance ( i.e. , low test error ) , as both Rm ( F ) and training error are low . Note that although a typical training dataset extracted from this LGF may be hard to fit using a low-complexity classifier F , there will be examples of training instances with non-zero probability on which a low-complexity classifier can have a good fit . The takeaway from this example is that the structure of the data ( here represented using the complexity of the generator space ) can additionally dictate whether a classifier can generalize . Note that , in this scenario , the expected generalization performance would be better if the LGF had more structure . Figure 1 illustrates the role of both generator and classifier spaces in generalization via the two example scenarios discussed earlier . In both examples , the same low-complexity classifier has a good fit on the training data , thusRm ( F ) is low . In example ( a ) , the LGF has no constraints ; while , in example ( b ) , the LGF has constraints such as local smoothness . In example ( a ) , the classifier clearly exhibits poor generalization performance on the test data . This agrees with our previous argument that if the LGF has no constraints , generalization is essentially impossible . In example ( b ) , the classifier shows much more robust generalization performance on the test data , due to the LGF having significantly more structure . Furthermore , it is intuitively clear that a classifier function space F which has a high overlap with the generator space G ( and therefore its constraints ) should yield good generalization performance . This shows , that in addition to the function spaces F and G individually affecting the generalization gap , the similarities between F and G are also important . Both of these perspectives play leading roles in our construction of generator-and-classifier aware complexity measures and the associated novel generalization error bounds . 1.2 RELATED WORK . To the best of our knowledge , the approach we are proposing to study generalization performance is novel and there is not much directly related work . Here , we describe some examples of works which discuss relevant concepts . The no free lunch theorem proposed in Wolpert & Macready ( 1997 ) indirectly sheds light on the behaviour of the LGFs . However , it does not incorporate ways to reduce variability in the LGFs by considering constraints related to the classification problem . Invariance constraints in learning algorithms were studied in Sokolic et al . ( 2017 ) , where the input space was factored into invariant transformations , similar to what we also do in this work . In doing so , the complexity of the data was indirectly explored , based on the number of invariant transformations present in the input space . However , the generalization bounds were derived with an assumption of perfectly invariant classifier function spaces , which is not applicable for CNNs and their variants ( as shown in Kauderer-Abrams ( 2017 ) ) . In another relevant study ( Jin et al . ( 2019 ) ) , a cover-complexity measure of a single dataset was proposed , and the generalization error of fully connected networks was analyzed with respect to the same . However , invariances in the dataset and the learning algorithm were neglected , i.e. , the similarities between the generator and the classifier spaces were not studied . 1.3 KEY CONTRIBUTIONS . The contributions of this work are as follows : 1 . We propose a novel complexity measure between the classifier function space F and the generator function space G called co-complexity , which we use to derive new , more accurate global estimates of generator-aware generalization error bounds ( Theorem 3 and 4 ) . Co-complexity considers not only the complexity of the generator and classifier function spaces , but also the similarities between them . Doing so allows for the a more exhaustive look into generalization error . 2 . We decompose co-complexity into two different measures of complexity , invariance co-complexity and dissociation co-complexity , which are used to derive new generalization error bounds , including bounds on the expected training error ( Theorem 6 ) . We find that reducing invariance co-complexity while keeping the dissociation co-complexity unchanged , helps reduce the generalization gap ( low variance ) while maintaining low training error ( low bias ) . This emphasizes the importance of having classifiers that share invariance properties with generator spaces , e.g. , rotation-invariant CNNs ( Cohen & Welling ( 2016b ) ) on MNIST-Rot ( Larochelle et al . ( 2007 ) ) . 3 . We present empirical validation of co-complexity measures of CNN and its scale-equivariant and rotation-equivariant counterparts ( SE-CNN in Sosnovik et al . ( 2019 ) , RE-CNN in Cohen & Welling ( 2016b ) ) , which explains their superior generalization ability compared to MLPs . Our proposed error bounds are easily specialized to the case where ground truth label function is fixed , leading to potentially tighter generalization error bounds ( see Appendix A ) . Although our proposed measures are global in nature , local variants can be derived via similar extensions used in Bartlett et al . ( 2005 ) . 2 DEFINITIONS . Assume that we have m number of d-dimensional i.i.d sampled instances S = [ z1 , z2 , .. , zm ] drawn from some distribution P , and another set ofm i.i.d sampled instances S′ = [ z′1 , z ′ 2 , .. , z ′ m ] also drawn from P . Define σ = [ σ1 , σ2 , .. , σm ] where σi are i.i.d random variables following the Rademacher distribution ( Pr ( σi = +1 ) = Pr ( σi = −1 ) = 0.5 ) . F and G are two function spaces from Rd −→ { −1 , 1 } and we assume that they are defined at all points in Rd . In the context of our problem , F will be the classifier ’ s function space , whereas G will be the generator space . Rademacher Complexity ( Bartlett & Mendelson ( 2003 ) ) : First , we provide the definition of Rademacher Complexity : Rm ( F ) = E σ , S [ sup f∈F ( 1 m m∑ i=1 f ( zi ) σi ) ] ( 1 ) It can be seen thatRm ( F ) indicates the noisy-label fitting ability of the classifier ’ s function space F . Correlated Rademacher Complexity : We propose a modified form of the original Rademacher complexity , called Correlated Rademacher Complexity , RCm ( F ) , which is defined as follows : RCm ( F ) = 1 2 × E σ , S , S′ [ sup f∈F ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) σi ) ] ( 2 ) We have 0 ≤ RCm ( F ) ≤ 1/2 . We also show thatRCm ( F ) ≤ Rm ( F ) ( see Appendix C.1 ) . It has been argued thatRm ( F ) ( and thereforeRCm ( F ) ) can be considered as ” entropy ” measures of the entire function space F ( Anguita et al . ( 2014 ) ) . Note that , likeRm ( F ) , RCm ( F ) is also eventually depends on the noisy label fitting ability of F , but computes it via ability of the function space to assign the same or a different label to two random points zi and z′i ( via f ( zi ) f ( z ′ i ) ) . Thus we denote it as the correlated Rademacher complexity of F . Co-Complexity : Now we propose various complexity measures between two separate function spaces F and G. Similar toRm ( F ) andRCm ( F ) , these measures assess the noisy label fitting abilities of the union of the function spaces F and G. In doing so , these measures compute a joint-entropy like metric over the two function spaces . First , we define the Co-complexity between F and G as follows . Rm ( F , G ) = 1 2 × E σ , S , S′ [ sup f∈F , g∈G ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] ( 3 ) Some of the properties of co-complexity are as follows : P1 Rm ( F , G ) = Rm ( G , F ) , i.e. , co-complexity is symmetric . P2 Rm ( F , G ) ≥ RCm ( F ) and Rm ( F , G ) ≥ RCm ( G ) , i.e. , the co-complexity between F and G is always greater than the individual correlated Rademacher complexities of F and G. P3 Rm ( F , G ) ≤ RCm ( F ) +RCm ( G ) ≤ Rm ( F ) +Rm ( G ) , i.e. , the co-complexity between F and G is upper bounded by the sum of the Rademacher complexities of F and G. P4 Rm ( F , G ) behaves like a joint-entropy measure ofF and G. To see this , let us define Im ( F , G ) = RCm ( F ) +RCm ( G ) − Rm ( F , G ) , called the mutual co-complexity between F and G. We later find that Im ( F , G ) behaves like a mutual-information measure . This , coupled with the fact that RCm ( F ) andRCm ( G ) can be construed as entropy measures of F and G , implies the result . Invariance Co-Complexity : Next , we quantify some of the properties of the ground truth generator space G in terms of its invariance transformations . We define the Invariance Classes of G as follows : IC ( G ) = { τ1 ( . , θ1 ) , τ2 ( . , θ2 ) , ... , τn ( . , θn ) } , ( 4 ) where { τi ( . , θi ) |i = 1 , 2 , · · · , n } are functions from Rd −→ Rd , and θi represents the extent of the transformation τi . Each θi ∈ R is a scalar and takes on a set of admissible values depending on the transformation ( possibly infinite ) . The functions are constrained such that , ∀τ ∈ IC ( G ) , ∀g ∈ G , g ( τ ( z ) , t ) = g ( z ) for all data points z ∈ Rd , and for all admissible values of the transformation parameter t. Also , note that setting θi = 0 leads to the identity function , i.e. , τi ( z , 0 ) = 1 for all z ∈ Rd and all i . Based on the above we define a transformation-indicator function I ( z , zi ) , such that I ( z , zi ) = 1 if z = τ1 ( . , t1 ) ◦ τ2 ( . , t2 ) ◦ .... ◦ τn ( . , tn ) ( zi ) , for a certain t1 , t2 , ... tn and otherwise I ( z , zi ) = 0 . Additionally , with respect to a generator space G , we also define the invariance extended set of a datapoint zi ∈ Rd as follows : IG ( zi ) = { z ∈ Rd | I ( z , zi ) = 1 } ( 5 ) Note that that invariance extended set of z will always have the same ground truth label as z . Next , we define the Invariance Co-complexity between two function spaces F and G as follows : RIm ( F , G ) = 1 2 × E S , S′ [ sup f∈F ( 1− ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) ) ) ] , where z′i ∼ P ( z ) I ( z , zi ) , ∀i . ( 6 ) Note that RIm ( F , G ) ≤ 1 . Each data point z′i in S′ is contained within the invariance extended set IG ( zi ) of zi , and is sampled according to the un-normalized distribution P ( z ) I ( z , zi ) over z ∈ Rd . The invariance co-complexity between F and G indicates the degree to which F obeys the invariance transformations within G. For instance , a low RIm ( F , G ) would indicate that f ( zi ) f ( z′i ) = 1 ( i.e. , f ( zi ) = f ( z ′ i ) ) for most zi and z ′ i which are related via some invariance transformation in G. Dissociation Co-Complexity : We now define the Dissociation Co-complexity between F and G when the corresponding datapoints in S and S′ are not related by any invariance transformation . That is , for all zi ∈ S and z′i ∈ S′ , z′i 6∈ IG ( zi ) , and is sampled from the distribution P ( z ) ( 1− I ( z , zi ) ) over z ∈ Rd . The dissociation co-complexity can then be defined as : RDm ( F , G ) = 1 2 × E σ , S , S′ [ sup f∈F ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) σi ) ] , where z′i ∼ P ( z ) ( 1− I ( z , zi ) ) , ∀i . ( 7 ) Note thatRDm ( F , G ) measures the average flexibility of F in its label assignment to any two points zi and z′i which are not related via any invariance transformation in G. Thus , a largerRDm ( F , G ) would indicate that the classifier function space F is able to easily assign separate labels to datapoints which are not related via any invariance transformation in G. We also define a variant of RDm ( F , G ) , in which S contains only one instance , instead of m instances , denoted asRD,1m ( F , G ) . RD,1m ( F , G ) is defined as follows . RD,1m ( F , G ) = 1 2 × E σ , z0∼P , S′ [ sup f∈F ( 1 m m∑ i=1 f ( z0 ) f ( z ′ i ) σi ) ] , where z′i ∼ P ( z ) ( 1− I ( z , z0 ) ) , ∀i . ( 8 ) Note thatRDm ( F , G ) ≤ 0.5 andRD,1m ( F , G ) ≤ 0.5 . Rademacher Smoothness : For our final result in Theorem 6 , we assume that the function space F is Rademacher smooth w.r.t . G. We define Rademacher Smoothness as follows . If F is Rademacher smooth w.r.t . G , then any quantity of the form ES , S′ [ supf∈F ( 1 m ∑m i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] or ES , S′ [ supf∈F , g∈G ( 1 m ∑m i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] , where the expectation is over datapoint permutations S , S′ , should lie between the two following cases : ( i ) S′ ∈ IG ( S ) ( i.e. , z′i ∼ P ( z ) ( I ( z , zi ) ) for all i ) and ( ii ) S′ /∈ IG ( S ) ( i.e. , z′i ∼ P ( z ) ( 1− I ( z , zi ) ) for all i ) . Then , for some non-zero value of α , 0 ≤ α ≤ 1 , the following holds : E S , S′ [ sup f∈F 1 m ( m∑ i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] = ( α ) E S , S′ , S′∈IG ( S ) [ sup f∈F ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] + ( 1− α ) E S , S′ , S′ /∈IG ( S ) [ sup f∈F ( 1 m m∑ i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] . ( 9 ) Similarly , the same applies to ES , S′ [ supf∈F , g∈G ( 1 m ∑m i=1 f ( zi ) f ( z ′ i ) g ( zi ) g ( z ′ i ) σi ) ] .

Summary: The paper provides an interesting perspective to view generalization error for the machine learning model. In particular, it proposes to investigate the constraint on the label generating function space. They propose a concept of co-complexity analogous to the entropy-ish concept which measures complexities between two function spaces. This co-complexity can be decomposed into two parts which measure the categorization ability of the classifier in generator and extent level in classifier for the invariance transformation in the generator.