Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views?

1 INTRODUCTION . The ability to robustly categorize objects under conditions and transformations that preserve the object categories is essential to animal intelligence , and to pursuits of practical importance such as improving computer vision systems . However , for general-purpose understanding and geometric reasoning , invariant representations of these objects in sensory processing circuits are not enough . Perceptual representations must also accurately encode their transformation properties . One such property is that of exhibiting equivariance to transformations of the object . When such transformations are restricted to be an algebraic group , the resulting equivariant representations have found significant success in machine learning starting with classical convolutional neural networks ( CNNs ) ( Denker et al. , 1989 ; LeCun et al. , 1989 ) and recently being generalized by the influential work of Cohen & Welling ( 2016 ) . Such representations have elicited burgeoning interest as they capture many transformations of practical interest such as translations , permutations , rotations , and reflections . Furthermore , equivariance to these transformations can be easily “ hard-coded ” into neural networks . Indeed , a new breed of CNN architectures that explicitly account for such transformations are seeing diverse and rapidly growing applications ( Townshend et al. , 2021 ; Baek et al. , 2021 ; Satorras et al. , 2021 ; Anderson et al. , 2019 ; Bogatskiy et al. , 2020 ; Klicpera et al. , 2020 ; Winkels & Cohen , 2019 ; Gordon et al. , 2020 ; Sosnovik et al. , 2021 ; Eismann et al. , 2020 ) . In addition , equivariant CNNs have been shown to capture response properties of neurons in the primary visual cortex beyond classical Gábor filter models ( Ecker et al. , 2018 ) . While it is clear that equivariance imposes a strong constraint on the geometry of representations and thus of perceptual manifolds ( Seung & Lee , 2000 ; DiCarlo & Cox , 2007 ) that are carved out by these representations as the objects transform , the implications of such constraints on their expressivity are not well understood . In this work we take a step towards addressing this gap . Our starting point is the classical notion of the perceptron capacity ( sometimes also known as the fractional memory/storage capacity ) – a quantity fundamental to the task of object categorization and closely related to VC dimension ( Vapnik & Chervonenkis , 1968 ) . Defined as the maximum number of points for which all ( or 1-δ fraction of ) possible binary label assignments ( i.e . dichotomies ) afford a hyperplane that separates points with one label from the points with the other , it can be seen to offer a quantification of the expressivity of a representation . Classical work on perceptron capacity focused on points in general position ( Wendel , 1962 ; Cover , 1965 ; Schläfli , 1950 ; Gardner , 1987 ; 1988 ) . However , understanding the perceptron capacity when the inputs are not merely points , but are endowed with richer structure , has only recently started to attract attention . For instance , work by Chung et al . ( 2018 ) ; Pastore et al . ( 2020 ) ; Rotondo et al . ( 2020 ) ; Cohen et al . ( 2020 ) considered general perceptual manifolds and examined the role of their geometry to obtain extensions to the perceptron capacity results . However , such work crucially relied on the assumption that each manifold is oriented randomly , a condition which is strongly violated by equivariant representations . With these motivations , our particular contributions in this paper are the following : • We extend Cover ’ s function counting theorem and VC dimension to equivariant representations , finding that both scale with the dimension of the subspace fixed by the group action . • We demonstrate the applicability of our result to G-convolutional network layers , including pooling layers , through theory and verify through simulation . 1.1 RELATED WORKS . Work most related to ours falls along two major axes . The first follows the classical perceptron capacity result on the linear separability of points ( Schläfli , 1950 ; Wendel , 1962 ; Cover , 1965 ; Gardner , 1987 ; 1988 ) . This result initiated a long history of investigation in theoretical neuroscience , e.g . ( Brunel et al. , 2004 ; Chapeton et al. , 2012 ; Rigotti et al. , 2013 ; Brunel , 2016 ; Rubin et al. , 2017 ; Pehlevan & Sengupta , 2017 ) , where it is used to understand the memory capacity of neuronal architectures . Similarly , in machine learning , the perceptron capacity and its variants , including notions for multilayer perceptrons , have been fundamental to a fruitful line of study in the context of finite sample expressivity and generalization ( Baum , 1988 ; Kowalczyk , 1997 ; Sontag , 1997 ; Huang , 2003 ; Yun et al. , 2019 ; Vershynin , 2020 ) . Work closest in spirit to ours comes from theoretical neuroscience and statistical physics ( Chung et al. , 2018 ; Pastore et al. , 2020 ; Rotondo et al. , 2020 ; Cohen et al. , 2020 ) , which considered general perceptual manifolds , albeit oriented randomly , and examined the role of their geometry to obtain extensions to the perceptron capacity result . The second line of relevant literature is that on group-equivariant convolutional neural networks ( GCNNs ) . The main inspiration for such networks comes from the spectacular success of classical CNNs ( LeCun et al. , 1989 ) which directly built in translational symmetry into the network architecture . In particular , the internal representations of a CNN are approximately1 translation equivariant : if the input image is translated by an amount t , the feature map of each internal layer is translated by the same amount . Furthermore , an invariant read-out on top ensures that a CNN is translation invariant . Cohen & Welling ( 2016 ) observed that a viable approach to generalize CNNs to other data types could involve considering equivariance to more general transformation groups . This idea has been used to construct networks equivariant to a wide variety of transformations such as planar rotations ( Worrall et al. , 2017 ; Weiler et al. , 2018b ; Bekkers et al. , 2018 ; Veeling et al. , 2018 ; Smets et al. , 2020 ) , 3D rotations ( Cohen et al. , 2018 ; Esteves et al. , 2018 ; Worrall & Brostow , 2018 ; Weiler et al. , 2018a ; Kondor et al. , 2018a ; Perraudin et al. , 2019 ) , permutations ( Zaheer et al. , 2017 ; Hartford et al. , 2018 ; Kondor et al. , 2018b ; Maron et al. , 2019a ; 2020 ) , general Euclidean isometries ( Weiler et al. , 2018a ; Weiler & Cesa , 2019 ; Finzi et al. , 2020 ) , scaling ( Marcos et al. , 2018 ; Worrall & Welling , 2019 ; Sosnovik et al. , 2020 ) and more exotic symmetries ( Bogatskiy et al. , 2020 ; Shutty & Wierzynski , 2020 ; Finzi et al. , 2021 ) etc . A quite general theory of equivariant/invariant networks has also emerged . Kondor & Trivedi ( 2018 ) gave a complete description of GCNNs for scalar fields on homogeneous spaces of compact groups . This was generalized further to cover the steerable case in ( Cohen et al. , 2019b ) and to general gauge fields in ( Cohen et al. , 2019a ; Weiler et al. , 2021 ) . This theory also includes universal approximation results ( Yarotsky , 2018 ; Keriven & Peyré , 2019 ; Sannai et al. , 2019b ; Maron et al. , 2019b ; Segol & Lipman , 2020 ; Ravanbakhsh , 2020 ) . Nevertheless , while benefits of equivariance/invariance in 1Some operations such as max pooling and boundary effects of the convolutions technically break strict equivariance , as well as the final densely connected layers . terms of improved sample complexity and ease of training are quoted frequently , a firm theoretical understanding is still largely missing . Some results however do exist , going back to ( Shawe-Taylor , 1991 ) . Abu-Mostafa ( 1993 ) made the argument that restricting a classifier to be invariant can not increase its VC dimension . Sokolic et al . ( 2017 ) extend this idea to derive generalization bounds for invariant classifiers , while Sannai et al . ( 2019a ) do so specifically working with the permutation group . Elesedy & Zaidi ( 2021 ) show a strict generalization benefit for equivariant linear models , showing that the generalization gap between a least squares model and its equivariant version depends on the dimension of the space of anti-symmetric linear maps . Some benefits of related ideas such as data augmentation and invariant averaging are formally shown in ( Lyle et al. , 2020 ; Chen et al. , 2020 ) . Here we focus on the limits to expressivity enforced by equivariance . 2 PROBLEM FORMULATION . Suppose x abstractly represents an object and let r ( x ) ∈ RN be some feature map of x to an N -dimensional space ( such as an intermediate layer of a deep neural network ) . We consider transformations of this object , such that they form a group in the algebraic sense of the word . We denote the abstract transformation of x by element g ∈ G as gx . Groups G may be represented by invertible matrices , which act on a vector space V ( which themselves form the group GL ( V ) of invertible linear transformations on V ) . We are interested in feature maps r which satisfy the following group equivariance condition : r ( gx ) = π ( g ) r ( x ) , where π : G → GL ( RN ) is a linear representation of G which acts on feature map r ( x ) . Note that many representations of G are possible , including the trivial representation : π ( g ) = I for all g. We are interested in perceptual object manifolds generated by the actions of G. Each of the P manifolds can be written as a set of points { π ( g ) rµ : g ∈ G } where µ ∈ [ P ] ≡ { 1 , 2 , . . . , P } ; that is , these manifolds are orbits of the point rµ ≡ r ( xµ ) under the action of π . We will refer to such manifolds as π-manifolds.2 Each of these π-manifolds represents a single object under the transformation encoded by π ; hence , each of the points in a π-manifold is assigned the same class label . A perceptron endowed with a set of linear readout weights w will attempt to determine the correct class of every point in every manifold . The condition for realizing ( i.e . linearly separating ) the dichotomy { yµ } µ can be written as yµw > π ( g ) rµ > 0 for all g ∈ G and µ ∈ [ P ] , where yµ = +1 if the µth manifold belongs to the first class and yµ = −1 if the µth manifold belongs to the second class . The perceptron capacity is the fraction of dichotomies that can be linearly separated ; that is , separated by a hyperplane . For concreteness , one might imagine that each of the rµ is the neural representation for an image of a dog ( if yµ = +1 ) or of a cat ( if yµ = −1 ) . The action π ( g ) could , for instance , correspond to the image shifting to the left or right , where the size of the shift is given by g. Different representations of even the same group can have different coding properties , an important point for investigating biological circuits and one that we leverage to construct a new GCNN architecture in Section 5 .

Summary: This paper studies the expressivity of a representation constrained by group equivariance. The authors first use a classical notion of the perceptron capacity to offer a quantification of the expressivity. Then, an algorithm is designed to efficiently classify the $\pi$-manifolds. Besides, the authors further apply the theoretical results to some practical examples and give bounds on the capacity when pooling operators exist.