Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views?

1 INTRODUCTION . The ability to robustly categorize objects under conditions and transformations that preserve the object categories is essential to animal intelligence , and to pursuits of practical importance such as improving computer vision systems . However , for general-purpose understanding and geometric reasoning , invariant representations of these objects in sensory processing circuits are not enough . Perceptual representations must also accurately encode their transformation properties . One such property is that of exhibiting equivariance to transformations of the object . When such transformations are restricted to be an algebraic group , the resulting equivariant representations have found significant success in machine learning starting with classical convolutional neural networks ( CNNs ) ( Denker et al. , 1989 ; LeCun et al. , 1989 ) and recently being generalized by the influential work of Cohen & Welling ( 2016 ) . Such representations have elicited burgeoning interest as they capture many transformations of practical interest such as translations , permutations , rotations , and reflections . Furthermore , equivariance to these transformations can be easily “ hard-coded ” into neural networks . Indeed , a new breed of CNN architectures that explicitly account for such transformations are seeing diverse and rapidly growing applications ( Townshend et al. , 2021 ; Baek et al. , 2021 ; Satorras et al. , 2021 ; Anderson et al. , 2019 ; Bogatskiy et al. , 2020 ; Klicpera et al. , 2020 ; Winkels & Cohen , 2019 ; Gordon et al. , 2020 ; Sosnovik et al. , 2021 ; Eismann et al. , 2020 ) . In addition , equivariant CNNs have been shown to capture response properties of neurons in the primary visual cortex beyond classical Gábor filter models ( Ecker et al. , 2018 ) . While it is clear that equivariance imposes a strong constraint on the geometry of representations and thus of perceptual manifolds ( Seung & Lee , 2000 ; DiCarlo & Cox , 2007 ) that are carved out by these representations as the objects transform , the implications of such constraints on their expressivity are not well understood . In this work we take a step towards addressing this gap . Our starting point is the classical notion of the perceptron capacity ( sometimes also known as the fractional memory/storage capacity ) – a quantity fundamental to the task of object categorization and closely related to VC dimension ( Vapnik & Chervonenkis , 1968 ) . Defined as the maximum number of points for which all ( or 1-δ fraction of ) possible binary label assignments ( i.e . dichotomies ) afford a hyperplane that separates points with one label from the points with the other , it can be seen to offer a quantification of the expressivity of a representation . Classical work on perceptron capacity focused on points in general position ( Wendel , 1962 ; Cover , 1965 ; Schläfli , 1950 ; Gardner , 1987 ; 1988 ) . However , understanding the perceptron capacity when the inputs are not merely points , but are endowed with richer structure , has only recently started to attract attention . For instance , work by Chung et al . ( 2018 ) ; Pastore et al . ( 2020 ) ; Rotondo et al . ( 2020 ) ; Cohen et al . ( 2020 ) considered general perceptual manifolds and examined the role of their geometry to obtain extensions to the perceptron capacity results . However , such work crucially relied on the assumption that each manifold is oriented randomly , a condition which is strongly violated by equivariant representations . With these motivations , our particular contributions in this paper are the following : • We extend Cover ’ s function counting theorem and VC dimension to equivariant representations , finding that both scale with the dimension of the subspace fixed by the group action . • We demonstrate the applicability of our result to G-convolutional network layers , including pooling layers , through theory and verify through simulation . 1.1 RELATED WORKS . Work most related to ours falls along two major axes . The first follows the classical perceptron capacity result on the linear separability of points ( Schläfli , 1950 ; Wendel , 1962 ; Cover , 1965 ; Gardner , 1987 ; 1988 ) . This result initiated a long history of investigation in theoretical neuroscience , e.g . ( Brunel et al. , 2004 ; Chapeton et al. , 2012 ; Rigotti et al. , 2013 ; Brunel , 2016 ; Rubin et al. , 2017 ; Pehlevan & Sengupta , 2017 ) , where it is used to understand the memory capacity of neuronal architectures . Similarly , in machine learning , the perceptron capacity and its variants , including notions for multilayer perceptrons , have been fundamental to a fruitful line of study in the context of finite sample expressivity and generalization ( Baum , 1988 ; Kowalczyk , 1997 ; Sontag , 1997 ; Huang , 2003 ; Yun et al. , 2019 ; Vershynin , 2020 ) . Work closest in spirit to ours comes from theoretical neuroscience and statistical physics ( Chung et al. , 2018 ; Pastore et al. , 2020 ; Rotondo et al. , 2020 ; Cohen et al. , 2020 ) , which considered general perceptual manifolds , albeit oriented randomly , and examined the role of their geometry to obtain extensions to the perceptron capacity result . The second line of relevant literature is that on group-equivariant convolutional neural networks ( GCNNs ) . The main inspiration for such networks comes from the spectacular success of classical CNNs ( LeCun et al. , 1989 ) which directly built in translational symmetry into the network architecture . In particular , the internal representations of a CNN are approximately1 translation equivariant : if the input image is translated by an amount t , the feature map of each internal layer is translated by the same amount . Furthermore , an invariant read-out on top ensures that a CNN is translation invariant . Cohen & Welling ( 2016 ) observed that a viable approach to generalize CNNs to other data types could involve considering equivariance to more general transformation groups . This idea has been used to construct networks equivariant to a wide variety of transformations such as planar rotations ( Worrall et al. , 2017 ; Weiler et al. , 2018b ; Bekkers et al. , 2018 ; Veeling et al. , 2018 ; Smets et al. , 2020 ) , 3D rotations ( Cohen et al. , 2018 ; Esteves et al. , 2018 ; Worrall & Brostow , 2018 ; Weiler et al. , 2018a ; Kondor et al. , 2018a ; Perraudin et al. , 2019 ) , permutations ( Zaheer et al. , 2017 ; Hartford et al. , 2018 ; Kondor et al. , 2018b ; Maron et al. , 2019a ; 2020 ) , general Euclidean isometries ( Weiler et al. , 2018a ; Weiler & Cesa , 2019 ; Finzi et al. , 2020 ) , scaling ( Marcos et al. , 2018 ; Worrall & Welling , 2019 ; Sosnovik et al. , 2020 ) and more exotic symmetries ( Bogatskiy et al. , 2020 ; Shutty & Wierzynski , 2020 ; Finzi et al. , 2021 ) etc . A quite general theory of equivariant/invariant networks has also emerged . Kondor & Trivedi ( 2018 ) gave a complete description of GCNNs for scalar fields on homogeneous spaces of compact groups . This was generalized further to cover the steerable case in ( Cohen et al. , 2019b ) and to general gauge fields in ( Cohen et al. , 2019a ; Weiler et al. , 2021 ) . This theory also includes universal approximation results ( Yarotsky , 2018 ; Keriven & Peyré , 2019 ; Sannai et al. , 2019b ; Maron et al. , 2019b ; Segol & Lipman , 2020 ; Ravanbakhsh , 2020 ) . Nevertheless , while benefits of equivariance/invariance in 1Some operations such as max pooling and boundary effects of the convolutions technically break strict equivariance , as well as the final densely connected layers . terms of improved sample complexity and ease of training are quoted frequently , a firm theoretical understanding is still largely missing . Some results however do exist , going back to ( Shawe-Taylor , 1991 ) . Abu-Mostafa ( 1993 ) made the argument that restricting a classifier to be invariant can not increase its VC dimension . Sokolic et al . ( 2017 ) extend this idea to derive generalization bounds for invariant classifiers , while Sannai et al . ( 2019a ) do so specifically working with the permutation group . Elesedy & Zaidi ( 2021 ) show a strict generalization benefit for equivariant linear models , showing that the generalization gap between a least squares model and its equivariant version depends on the dimension of the space of anti-symmetric linear maps . Some benefits of related ideas such as data augmentation and invariant averaging are formally shown in ( Lyle et al. , 2020 ; Chen et al. , 2020 ) . Here we focus on the limits to expressivity enforced by equivariance . 2 PROBLEM FORMULATION . Suppose x abstractly represents an object and let r ( x ) ∈ RN be some feature map of x to an N -dimensional space ( such as an intermediate layer of a deep neural network ) . We consider transformations of this object , such that they form a group in the algebraic sense of the word . We denote the abstract transformation of x by element g ∈ G as gx . Groups G may be represented by invertible matrices , which act on a vector space V ( which themselves form the group GL ( V ) of invertible linear transformations on V ) . We are interested in feature maps r which satisfy the following group equivariance condition : r ( gx ) = π ( g ) r ( x ) , where π : G → GL ( RN ) is a linear representation of G which acts on feature map r ( x ) . Note that many representations of G are possible , including the trivial representation : π ( g ) = I for all g. We are interested in perceptual object manifolds generated by the actions of G. Each of the P manifolds can be written as a set of points { π ( g ) rµ : g ∈ G } where µ ∈ [ P ] ≡ { 1 , 2 , . . . , P } ; that is , these manifolds are orbits of the point rµ ≡ r ( xµ ) under the action of π . We will refer to such manifolds as π-manifolds.2 Each of these π-manifolds represents a single object under the transformation encoded by π ; hence , each of the points in a π-manifold is assigned the same class label . A perceptron endowed with a set of linear readout weights w will attempt to determine the correct class of every point in every manifold . The condition for realizing ( i.e . linearly separating ) the dichotomy { yµ } µ can be written as yµw > π ( g ) rµ > 0 for all g ∈ G and µ ∈ [ P ] , where yµ = +1 if the µth manifold belongs to the first class and yµ = −1 if the µth manifold belongs to the second class . The perceptron capacity is the fraction of dichotomies that can be linearly separated ; that is , separated by a hyperplane . For concreteness , one might imagine that each of the rµ is the neural representation for an image of a dog ( if yµ = +1 ) or of a cat ( if yµ = −1 ) . The action π ( g ) could , for instance , correspond to the image shifting to the left or right , where the size of the shift is given by g. Different representations of even the same group can have different coding properties , an important point for investigating biological circuits and one that we leverage to construct a new GCNN architecture in Section 5 .

Summary: This paper addresses an underexamined area, the question of expressivity in equivariant neural networks.  In general, there is a large gap between our theoretical understanding of expressivity and generalization for non-equivariant networks and that for equivariant ones; this paper helps to close the gap.  In particular, given a set of equivariant representations coming from an equivariant architecture, the authors determine how expressive these features are for an invariant binary classification problem.  To do so, they define a reasonable generalization of perceptron capacity for invariant classifiers and compute it.  The answer turns out to be relatively simple and is mainly determined by the multiplicity of the trivial representation in the group representation type of the equivariant representation, which is not difficult to compute.  The authors also consider the case of useful but not quite equivariant operations such as pooling. Lastly, they provide a convincing empirical verification of their results.