Understanding the Generalization Benefit of Model Invariance from a Data Perspective

1 Introduction . Invariance is ubiquitous in many real-world problems . For instance , categorical classification of visual objects is invariant to slight viewpoint changes [ 18 , 2 , 23 ] , text understanding is invariant to synonymous substitution and minor typos [ 53 , 36 , 27 ] . Intuitively , models capturing the underlying invariance exhibit improved generalization in practice [ 21 , 13 , 50 , 14 , 45 , 14 ] . Such generalization benefit is especially crucial when the data are scarce as in some medical tasks [ 46 ] , or when the task requires more data than usual as in cases of distribution shift [ 38 ] and adversarial attack [ 40 , 49 , 5 ] . A commonly accepted intuition attributes the generalization benefit of model invariance to the reduced model complexity , especially the reduced sensitivity to spurious features . However , a principled understanding of why model invariance helps generalization remains elusive , thus leaving many open questions . Since model invariance may come at a cost ( e.g. , compromised accuracy , increase computational overhead ) , given a task , how should we choose among various data transformations under which model invariance guarantees better generalization ? If existing data transformations are not good enough for a given task , what is the guiding principle to find new ones ? The lack of a principled understanding limits better exploitation of model invariance to further improve * Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . generalization . In addition , since identifying instructive generalization bound is a central topic in machine learning , we may expect to tighten existing generalization bounds by additionally considering the data-dependent model invariance property . The many faces of data transformations and model classes pose significant challenges to a principled understanding of model invariance ’ s generalization benefit . To address this , [ 2 , 4 , 34 , 42 , 39 ] characterize the input space and show that certain data transformations equivalently shrink the input space for invariant models , which then simplify the input and improves generalization . From another perspective , [ 19 , 30 ] directly characterize the function space and show that the volume of the invariant model class is reduced , which then simplifies the learning problem and improves generalization . These understandings provide valuable insights , yet they may become less informative on high-dimensional input data or require the model invariance to be obtained exclusively via feature averaging . Some certain assumptions on data transformations ( e.g. , finiteness , group structure with certain measure ) also make these understandings less applicable to more general data transformations . In this paper , we derive generalization bounds for invariant models based on the sample cover induced by data transformations and empirically show that the introduced notion can guide the data transformations selection . Different from previous understandings , we first identify a data-dependent property of data transformations in a model-agnostic way , and then establish its connections with the refined generalization bounds of invariant models . The analysis applies to more general data transformations regardless of how the model invariance is obtained and naturally provides a modelagnostic guidance for data transformations selection . We summarize our contributions as follows . Contributions . At the core of our understanding is the notion of sample cover induced by data transformations , defined informally as a representative subset of a dataset that can approximately recover the whole dataset using data transformations ( illustrated in Figure 1 ) . We show that this notion identifies a data-dependent property of data transformations which is related to the generalization benefit of the corresponding invariant models . Under a special setting of the sample cover , we first bound the model complexity of any invariant and output-bounded model class in terms of the sample covering numbers . Since this general bound requires a restrictive condition on data transformations in order to be informative , we then assume the model Lipschitzness to relax the requirement and refine the model complexity bound for invariant models . Finally , we outline a framework for modelinvariance-sensitive generalization bounds based on the invariant models ’ complexities , and use it to discuss the generalization benefit of model invariance . Given the usefulness of sample cover in the analysis , we propose an algorithm to empirically estimate the sample cover . This algorithm exactly verifies whether a given subset of a sample forms a valid sample cover , and always estimates a sample covering number that upper-bounds the groundtruth . Inspired by our analysis , we also propose to use the sample covering number as a suitability measurement for practical data transformation selections . This measurement is data-driven , widely applicable , and empirically correlates with invariant models ’ actual generalization performance . We discuss its limitations and the empirical mitigation . To empirically verify our propositions , we first estimate the sample covering number for some commonly used data transformations on four image datasets , including CIFAR-10 and ShapeNet ( a 3D dataset ) . Under typical settings , the 3D-view transformation induces a much smaller sample covering number than others on ShapeNet , while cropping induces the smallest sample covering number on others datasets . For those data transformations , we then train invariant models via data augmentation and invariance loss regularization to evaluate the actual generalization benefit . Results show a clear correlation between smaller sample covering numbers induced by data transformations and the better generalization benefit enjoyed by invariant models . 2 Preliminaries . Data transformations . We refer to the data transformation as a function from the input space X → X , and data transformations as a set of such functions . Unless otherwise specified , we do not assume data transformations to have group structures since many non-invertible transformations ( e.g. , cropping ) do not fit into a group structure directly . For a set of data transformations G = { g : X → X } and a data point ( also referred to as an example ) x ∈ X , we overload the notion of orbit in group theory and denote by G ( x ) the orbit of x defined as follows . The orbit of x generated by data transformations G is the collection of outputs after applying any transformation g ∈ G on x : G ( x ) = { g ( x ) ∈ X : g ∈ G } . Model invariance . Let D be the underlying data distribution and supp ( D ) be its support . A model h : X → Y is said to be invariant under data transformations G on D if h ( g ( x ) ) = h ( x ) for any x ∈ supp ( D ) and any g ∈ G. We refer to a class of invariant models as the G-invariant model class . Complexity measurements . Covering number and Rademacher complexity [ 33 ] are two commonly used complexity measurements for model classes ( including neural networks [ 6 ] ) that can provide uniform generalization bounds . The covering number can also be directly used to upper bound the Rademacher complexity via Dudley ’ s entropy integral theorem [ 17 , 32 ] . Covering number . Let ( F , d ) be a ( pseudo ) metric space with some ( pseudo ) metric1 d. An -cover of a set H ⊆ F is defined as a subset Ĥ ⊆ H such that for any h ∈ H , there exists ĥ ∈ Ĥ such that d ( h , ĥ ) ≤ . The covering number N ( , H , d ) is defined as the minimum cardinality of an -cover ( among all -covers ) ofH . In this paper , we use the concept of covering number both for measuring model class complexities and for defining the sample covering number on datasets . Empirical Rademacher complexity . Let H be a class of functions h : X → R. Given a sample S = { xi } ni=1 , the empirical Rademacher complexity of model class H is defined as : RS ( H ) = Eσ [ suph∈H 1 n ∑n i=1 σih ( xi ) ] where σ = [ σ1 , ... , σn ] > is the vector of i.i.d . Rademacher random variables , each uniformly chosen from { −1 , 1 } . Generalization error and gap . Let S = { xi } ni=1 be a sample drawn i.i.d . from some data distribution D , and H be a model class . Given a loss function ` : R → [ 0 , 1 ] , for a h ∈ H , we define the empirical error as RS ( h ) = 1n ∑n i=1 ` ( h ( xi ) , yi ) , the generalization error as R ( h ) = E ( x , y ) ∼D [ ` ( h ( x ) , y ) ] , and the generalization gap as R ( h ) −RS ( h ) . 3 Generalization Benefit of Model Invariance . In this section , we derive the generalization bounds for invariant models by identifying the model invariance properties . We start by introducing the notion of sample cover induced by data transformations and based on it bound the Rademacher complexity of any invariant models with bounded output ( Section 3.1 ) . Then , we assume model Lipschitzness to provide a more informative model complexity bound for any data transformations ( Section 3.2 ) . Finally , we provide a framework for model-invariance-sensitive generalization bounds and discuss the generalization benefit of model invariance ( Section 3.3 ) . 1A pseudometric is a metric if and only if it separates distinct points , namely d ( x , y ) > 0 for any x 6= y . 3.1 Sample Cover Induced by Data Transformations . Existing empirical results suggest that , compared with standard models , invariant models may have certain properties reducing their effective model complexities . To identify such properties , we alternatively identify the related properties of the corresponding data transformations via the notion of sample cover induced by data transformations . We now formalize the introduced notion . The definition of sample cover relies on the pseudometric induced by the data transformations G. Note that G generates an orbit G ( x ) ⊆ X for each example x ∈ S . Let ‖ · ‖ be any norm on the input space X . Given a set of transformations G , we define the G-induced pseudometric2 as ρG ( x1 , x2 ) = inf γ∈Γ ( x1 , x2 ) ∫ γ c ( r ) ds , where c ( r ) = { 0 , if r ∈ ∪x∈SG ( x ) 1 , otherwise ( 3.1 ) where ds = ‖dr‖ , and Γ denotes the set of all paths ( curves ) in X from x1 to x2 . The ρG is essentially calculating the line integral along the shortest ( if achievable ) path γ in the scalar field c , where c can also be viewed as the `` moving cost '' function depending on G. The norm ‖ · ‖ here can be selected as any meaningful norm on the input space ( e.g. , Euclidean norm as in our experiments ) and will later be used in defining the model ’ s Lipschitz constant . It can be checked that ρG satisfies pseudometric axioms . Definition 3.1 ( Sample cover induced by data transformations ) . Let ( X , ρG ) be a pseudometric space and S = { xi } ni=1 be a sample of size n. An -sample cover ŜG , of the sample S induced by data transformations G at resolution is defined as a subset of the sample S such that for any x ∈ S , there exists x̂ ∈ ŜG , satisfying ρG ( x , x̂ ) ≤ . Definition 3.2 ( Sample covering number induced by data transformations ) . The sample covering number N ( , S , ρG ) induced by data transformations G is defined as the minimum cardinality of an -sample cover : N ( , S , ρG ) = min { |ŜG , | : ŜG , is an -sample cover of S } . ( 3.2 ) Informally , the G-induced sample cover specifies a representative subset of examples which can approximately recover all the original examples using the given data transformations G. This notion is closely related to the sample compression [ 20 ] which represents a scheme to prove the learnability of concepts through a compressed set of sample . While identifying the generalization-related properties of data transformations , this notion is insensitive to other unrelated properties ( e.g. , finiteness , group structures ) and thus applies to any data transformations . The intuition behind sample cover is that G-invariant models may have consistent behaviors on an example and its associated approximation in the G-induced sample cover . As such , we can analyze the model complexities of invariant models by considering the models ’ behaviour only on the potentially small-sized sample covers . Indeed , we directly have the following model complexity result . The proof is in Appendix B . Proposition 3.3 . Let S = { xi } ni=1 be a sample of size n. LetH be a model class mapping from X to [ −B , B ] for some B > 0 and is invariant to data transformations G. Then the empirical Rademacher complexity ofH satisfy RS ( H ) ≤ 24B √ N ( 0 , S , ρG ) n . ( 3.3 ) Proposition 3.3 generally bounds the model complexity of any output-bounded and G-invariant model class in terms of the sample covering number N ( 0 , S , ρG ) induced by G. A small G-induced sample covering number at resolution = 0 thus tightens the model complexity bound for a general class of G-invariant models . Note , however , that Proposition 3.3 is informative only when the data transformations G yields N ( 0 , S , ρG ) n on the sample S — a condition requiring G to be able to exactly recover S from a small-sized subset of S. This condition is unfortunately too strict to hold for many commonly used 2Note that ρG is not a metric since it allows ρG ( x , y ) = 0 for x 6= y. data transformations which only generate orbits with measure zero ( with respect to the data measure ) at most data points . For example , the rotation transformations on CIFAR-10 do not satisfy this condition , since no two images in CIFAR-10 are rotated versions of each other . To better understand the generalization benefit brought by any data transformations ( e.g. , rotation ) , we further assume specific model properties which equivalently expand the orbits in order to get more general results . We study Lipschitz models in Section 3.2 , and relegate a sharper ( and relatively independent ) analysis for linear models under linear data transformations to Appendix C .

Summary: The paper provides a bound on the Rademacher complexity of a class of Lipschitz and transformation-invariant functions by upper bounding the covering number of the function class by a similar covering number which uses a "sample cover induced by data transformations". This is the induced covering number which accounts for transformations of the original sample set and can be smaller due to the transformation invariance of the function. This directly gives improved generalization bounds for transformation-invariant functions. The paper gives some helpful intuition and simple cases where these bounds may be applied.  Furthermore, the paper presents experiments in which the sample covering number induced by different transformations is approximated, and the generalization ability of a model which is trained to be invariant to those respective transformations is shown to be strongly correlated with the sample covering number.