Title: On the Implicit Bias of Linear Equivariant Steerable Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: implicit bias

equivariant steerable networks

data augmentation

margin

generalization bound
Abstract: We study the implicit bias of gradient flow on linear equivariant steerable networks in group-invariant binary classification. Our findings reveal that the parameterized predictor converges in direction to the unique group-invariant classifier with a maximum margin defined by the input group action. Under a unitary assumption on the input representation, we establish the equivalence between steerable networks and data augmentation. Furthermore, we demonstrate the improved margin and generalization bound of steerable networks over their non-invariant counterparts.

On the Implicit Bias of Linear Equivariant Steerable Networks

Abstract

We study the implicit bias of gradient flow on linear equivariant steerable networks in group-invariant binary classification. Our findings reveal that the parameterized predictor converges in direction to the unique group-invariant classifier with a maximum margin defined by the input group action. Under a unitary assumption on the input representation, we establish the equivalence between steerable networks and data augmentation. Furthermore, we demonstrate the improved margin and generalization bound of steerable networks over their non-invariant counterparts.

1 Introduction

Despite recent theoretical breakthroughs in deep learning, it is still largely unknown why overparameterized deep neural networks (DNNs) with infinitely many solutions achieving near-zero training error can effectively generalize on new data. However, the consistently impressive results of DNNs trained with first-order optimization methods, e.g. , gradient descent (GD), suggest that the training algorithm is implicitly guiding the model towards a solution with strong generalization performance.

Indeed, recent studies have shown that gradient-based training methods effectively regularize the solution by implicitly minimizing a certain complexity measure of the model [Vardi, 2022]. For example, Gunasekar et al. [2018b] showed that in separable binary classification, the linear predictor parameterized by a linear fully-connected network trained under GD converges in the direction of a max-margin support vector machine (SVM), while linear convolutional networks are implicitly regularized by a depth-related bridge penalty in the Fourier domain. Yun et al. [2021] extended this finding to linear tensor networks. Lyu and Li [2020] and Ji and Telgarsky [2020] established the implicit max-margin regularization for (nonlinear) homogeneous DNNs in the parameter space.

On the other hand, another line of research aims to explicitly regularize DNNs through architectural design to exploit the inherent structure of the learning problem. In recent years, there has been a growing interest in leveraging group symmetry for this purpose, given its prevalence in both scientific and engineering domains. A significant body of literature has been devoted to designing group-equivariant DNNs that ensure outputs transform covariantly to input symmetry transformations. Group-equivariant steerable networks represent a general class of symmetry-preserving models that achieve equivariance with respect to any pair of input-output group actions [Cohen et al., 2019, Weiler and Cesa, 2019, Cohen and Welling, 2017]. Empirical evidence suggests that equivariant steerable networks yield substantial improvements in generalization performance for learning tasks with group symmetry, especially when working with limited amounts of data.

There have been several recent attempts to account for the empirical success of equivariant steerable networks through establishing a tighter upper bound on the test risk for these models. This is typically accomplished by evaluating the complexity measures of equivariant and non-equivariant models under the same norm constraint on the network parameters [Sokolic et al., 2017, Sannai

et al., 2021, Elesedy, 2022]. Nevertheless, it remains unclear whether or why a GD-trained steerable network can achieve a minimizer with a parameter norm comparable to that of its non-equivariant counterpart. Consequently, the effectiveness of such complexity-measure-based arguments to explain the generalization enhancement of steerable networks in group symmetric learning tasks may not be directly applicable.

In light of the above issues, in this work, we aim to fully characterize the implicit bias of the training algorithm on linear equivariant steerable networks in group-invariant binary classification. Our result shows that when trained under gradient flow (GF), i.e. , GD with an infinitesimal step size, the steerable-network-parameterized predictor converges in direction to the unique groupinvariant classifier attaining a maximum margin with respect to a norm defined by the input group representation. This result has three important implications: under a unitary input group action,

a linear steerable network trained on the original data set converge in the same direction as a linear fully-connected network trained on the group-augmented data set. This suggests the equivalence between training with linear steerable networks and data augmentation ;

when trained on the same original data set, a linear steerable network always attains a wider margin on the group-augmented data set compared to a fully-connected network;

when the underlying distribution is group-invariant, a GF-trained linear steerable network achieves a tighter generalization bound compared to its non-equivariant counterpart. This improvement in generalization is not necessarily dependent on the group size, but rather it depends on the support of the invariant distribution.

Before we end this section, we note that a similar topic has recently been explored by Lawrence et al. [2021] in the context of linear Group Convolutional Neural Networks (G-CNNs), a special case of the equivariant steerable networks considered in this work. However, we point out that the models they studied were not truly group-invariant, and thus their implicit bias result does not explain the improved generalization of G-CNNs. We will further elaborate on the comparison between our work and [Lawrence et al., 2021] in Section 2.

2 Related work

Implicit biases : Recent studies have shown that for linear regression with the logistic or exponential loss on linearly separable data, the linear predictor under GD/SGD converges in direction to the maxL 2 -margin SVM [Soudry et al., 2018, Nacson et al., 2019, Gunasekar et al., 2018a]. These results are extended to linear fully-connected networks and linear Convolutional Neural Networks (CNNs) by Gunasekar et al. [2018b] under the assumption of directional convergence and alignment of the network parameters, which are later proved by Ji and Telgarsky [2019a,b], Lyu and Li [2020], Ji and Telgarsky [2020]. The implicit regularization of gradient flow (GF) is further generalized to linear tensor networks by Yun et al. [2021]. For overparameterized nonlinear networks in the infinite-width regime, rigorous analysis on the optimization of DNNs has also been studied from the neural tangent kernel [Jacot et al., 2018, Du et al., 2019, Allen-Zhu et al., 2019] and the mean-field perspectives [Mei et al., 2019, Chizat and Bach, 2018]. However, these models are not explicitly designed to be group-invariant/equivariant, and the implicit bias of gradient-based methods might guide them to converge to sub-optimal solutions in learning tasks with intrinsic group symmetry.

Equivariant neural networks. Since their introduction by Cohen and Welling [2016], group equivariant network design has become a burgeoning field with numerous applications from computer vision to scientific computing. Unlike the implicit bias induced by training algorithms, equivariant networks explicitly incorporate symmetry priors into model design through either group convolutions [Cheng et al., 2019, Weiler et al., 2018, Sosnovik et al., 2020, Zhu et al., 2022] or, more generally, steerable convolutions [Weiler and Cesa, 2019, Cohen et al., 2019, Worrall et al., 2017]. Despite their empirical success, it remains largely unknown why and whether equivariant networks trained under gradient-based methods actually converge to solutions with smaller test risk in group-symmetric learning tasks [Sokolic et al., 2017, Sannai et al., 2021]. In a recent study, Elesedy and Zaidi [2021] demonstrated a provably strict generalisation benefit for equivariant networks in linear regression. However, the network studied therein is non-equivariant and only symmetrized after training; such test-time data augmentation is different from practical usage of equivariant networks.

Comparison to Lawrence et al. [2021]. A recent study by Lawrence et al. [2021] also analyzes the implicit bias of linear equivariant G-CNNs, which are a special case of the steerable networks considered in this work. However, the networks studied therein are not truly equivariant/invariant. More specifically, the input space X 0 they considered is the set of functions on the group G , i.e. , X 0 = { f : G → R } , and the input G -action is given by the regular representation

In their case, due to the transitivity of the regular representation (1), G -invariant linear functions Φ : X 0 → R are constrained to the (trivial) form:

where C is a multiplicative constant. To avoid learning this trivial function, Lawrence et al. [2021] chose to parameterize Φ using a linear G-CNN, in which the final layer is replaced by a fullyconnected layer. While this fully-connected layer provides the capability to learn more complex and nontrivial functions, it simultaneously undermines the property of group invariance. Therefore, their implicit bias result does not explain the improved generalization of G-CNNs. In contrast, we assume Euclidean inputs with non-transitive group actions, allowing linear steerable networks to learn non-trivial group-invariant models.

3 Background and problem setup

We provide a brief background in group theory and group-equivariant steerable networks. We also explain the setup of our learning problem for group-invariant binary classification.

3.1 Group and group equivariance

A group is a set G equipped with a binary operator, the group product, satisfying the axioms of associativity, identity, and invertibility. We always assume in this work that G is finite, i.e. , | G | < ∞ , where | G | denotes its cardinality.

Given a vector space X , let GL ( X ) be the general linear group of X consisting of all invertible linear transformations on X . A map ρ : G → GL ( X ) is called a group representation (or linear group action ) of G on X if ρ is a group homomorphism from G to GL ( X ) , namely

.

When the representation ρ is clear from the context, we also abbreviate the group action ρ ( g ) x as g x Given a pair of vector spaces X , Y and their respective G -representations ρ X and ρ Y , a linear map Ψ : X → Y is said to be G -equivariant if it commutes with the G -representations ρ X and ρ Y , i.e. ,

Linear equivariant maps are also called intertwiners , and we denote by Hom G ( ρ X , ρ Y ) the space of all intertwiners satisfying (3). When ρ Y ≡ Id is the trivial representation, then Ψ[ ρ X ( g )( x )] = Ψ[ x ] for all x ∈ X ; namely, Ψ becomes a G -invariant linear map.

3.2 Equivariant steerable networks and G-CNNs

Let X 0 = R d 0 be a d 0 -dimensional input Euclidean space, equipped with the usual inner product. Let ρ 0 : G → GL ( X 0 ) be a G -representation on X 0 . Suppose we have an unknown target function f ∗ : X 0 = R d 0 → R that is G -invariant under ρ 0 , i.e. , f ∗ ( g x ) := f ∗ ( ρ 0 ( g ) x ) = f ∗ ( x ) for all g ∈ G and x ∈ X 0 . The goal of equivariant steerable networks is to approximate f ∗ using an L -layer neural network f = Ψ L ◦ Ψ L -1 ◦ · · · ◦ Ψ 1 that is guaranteed to be also G -invariant.

Since the composition of equivariant maps is also equivariant, it suffices to specify a collection of G -representation spaces { ( X l , ρ l ) } L l =1 , with ( X L , ρ L ) = ( R , Id ) being the trivial representation, such that each layer Ψ l ∈ Hom G ( ρ l -1 , ρ L ) : X l -1 → X l is G -equivariant. Equivalently, we want the

following diagram to be commutative,

Equivariant steerable networks. Given the representations { ( X l , ρ l ) } L l =0 , a (linear) steerable network is constructed as follows. For each l ∈ [ L ] := { 1 , · · · , L } , we choose a finite collection of N l (pre-computed) intertwiners { ψ j l } N l j =1 ⊂ Hom G ( ρ l -1 , ρ l ) . Typically, { ψ j l } N l j =1 is a basis of Hom G ( ρ l -1 , ρ l ) , but it is not necessary in our setting. The l -th layer equivariant map Ψ steer l : X l -1 → X l of a steerable network is then parameterized by

where the coefficients w l = [ w j l ] glyph[latticetop] j ∈ [ N l ] ∈ R N l are the trainable parameters of the l -th layer. An L -layer linear steerable network f steer ( x ; W ) is then defined as the composition

where W = [ w l ] L l =1 ∈ ∏ L l =1 R N l =: W steer is the collection of all trainable parameters. The network f steer ( x ; W ) defined in (6) is referred to as a steerable network because it steers the layer-wise output to transform according to any specified representation.

G-CNNs. A special case of the steerable networks is the group convolutional neural network (G-CNN), wherein the hidden representation space ( X l , ρ l ) for each l ∈ [ L -1] is set to

The representation ρ l ∈ GL ( X l ) in (7) is known as the regular representation of G . Intuitively, x l ∈ X l can be viewed as a matrix of size d l ×| G | , and ρ l ( g ) is a permutation of the columns of x l .

With this choice of { ( X l , ρ l ) } L l =0 , the first-layer equivariant map of a G-CNN is defined as

where w 1 = ( w j,k 1 ) j,k ∈ R d 0 × d 1 are the trainable parameters of the first layer. Eq. (8) is called a G -lifting map as it lifts a Euclidean signal x ∈ R d 0 to a function Ψ G-CNN 1 ( x ; w 1 ) on G . For the subsequent layers, equivariance is achieved through group convolutions , and the readers are referred to the appendix for a detailed explanation.

Assumptions on the representations. The input representation ρ 0 ∈ GL ( X 0 = R d 0 ) is assumed to be given, and ρ L ≡ 1 ∈ GL ( X L = R ) is set to the trivial representation for group-invariant outputs. In this work, we make the following special choices for the first-layer representation ( ρ 1 , X 1 ) as well as the equivariant map Ψ steer 1 ( x , w 1 ) in a general steerable network.

Assumption 3.1. We adopt the regular representation (7) for the first layer, and set the first-layer equivariant map Ψ steer 1 ( x ; w 1 ) to the G -lifting map (8) .

Remark 3.2. The rationale of Assumption 3.1 is for the network to have enough capacity to parameterize any G -invariant linear classifier; this will be further explained in Proposition 3.6. We note that the representations ( ρ l , X l ) and steerable maps Ψ steer l ( x , w l ) for all subsequent layers, l ∈ { 2 , · · · , L -1 } , can be arbitrary.

Under Assumption 3.1, we can characterize the linear steerable networks f steer ( · ; W ) in the following proposition.

Proposition 3.3. Let f steer ( x ; W ) be the linear steerable network satisfying Assumption 3.1, where W = [ w l ] L l =1 ∈ W steer is the collection of all model parameters. There exists a multi-linear map M : ( w 2 , · · · , w L ) ↦→ M ( w 2 , · · · , w L ) ∈ R d 1 such that for all x ∈ R d 0 and W ∈ W steer ,

where x := 1 | G | ∑ g ∈ G g x is the average of all elements on the group orbit of x ∈ R d 0 .

Although a straightforward proof of Proposition 3.3 can be readily derived using Schur's Lemma, we have opted to include an elementary proof of Proposition 3.3 in the appendix for the sake of completeness. If we define

where g glyph[latticetop] := ρ 0 ( g ) glyph[latticetop] , then

By the multi-linearity of M ( w 2 , · · · , w L ) , one can verify that f steer ( x ; W ) , P steer ( W ) , and P steer ( W ) are all L -homogeneous in W ; that is, for all ν > 0 and W ∈ W steer ,

Remark 3.4. In comparison, an L -layer linear fully-connected network f fc ( x ; W ) is given by

where W = [ w l ] L l =1 ∈ W fc := ( ∏ L -1 l =1 R d l -1 × d l ) × R d L -1 . It is worth noting that when G = { e } is the trivial group, a linear fully-connected network f fc ( x ; W ) is identical with a linear G-CNN, which is a special case of linear steerable networks. See Remark A.1 for details.

3.3 Group-invariant binary classification

Consider a binary classification data set S = { ( x i , y i ) : i ∈ [ n ] } , where x i ∈ R d 0 and y i ∈ {± 1 } , ∀ i ∈ [ n ] . We assume that S are i.i.d. samples from a G -invariant distribution D defined below.

Definition 3.5. A distribution D over R d 0 × {± 1 } is said to be G -invariant with respect to a representation ρ 0 ∈ GL ( R d 0 ) if

where ( ρ 0 ( g ) ⊗ Id )( x , y ) := ( ρ 0 ( g ) x , y ) , and ( ρ 0 ( g ) ⊗ Id ) ∗ D := D ◦ ( ρ 0 ( g ) ⊗ Id ) -1 is the pushforward measure of D under ρ 0 ( g ) ⊗ Id .

It is easy to verify that the Bayes optimal classifier f ∗ : R d 0 → {± 1 } (the one achieving the smallest population risk) for a G -invariant distribution D is necessarily a G -invariant function, i.e. , f ∗ ( g x ) = f ∗ ( x ) , ∀ g ∈ G . Therefore, to learn f ∗ using (linear) neural networks, it is natural to approximate f ∗ using an equivariant steerable network

After choosing the exponential loss glyph[lscript] exp : R ×{± 1 } → R + , glyph[lscript] exp (ˆ y, y ) := exp( -ˆ yy ) , as a surrogate loss function, the empirical risk minimization over S for the steerable network f steer ( x ; W ) becomes

On the other hand, since P steer ( W ) = β ∈ R d 0 always corresponds to a G -invariant linear predictorwe have slightly abused the notation by identifying β ∈ R d 0 with the map x ↦→ 〈 x , β 〉 -one can alternatively consider the empirical risk minimization directly over the invariant linear predictors β :

where R d 0 G ⊂ R d 0 is the subspace of all G -invariant linear predictors, which is characterized by the following proposition.

Proposition 3.6. Let R d 0 G ⊂ R d 0 be the subspace of G -invariant linear predictors, i.e., R d 0 G = { β ∈ R d 0 : β glyph[latticetop] x = β glyph[latticetop] g x , ∀ x ∈ R d 0 , ∀ g ∈ G } . Then

(a) R d 0 G is characterized by

(b) Let A : R d 0 → R d 0 be the group-averaging map,

Then its adjoint A glyph[latticetop] : β ↦→ 1 | G | ∑ g ∈ G g glyph[latticetop] β is a projection operator from R d 0 to R d 0 G . In other words, Range ( A glyph[latticetop] ) = R d 0 G and A glyph[latticetop] ◦ A glyph[latticetop] = A glyph[latticetop] .

(c) If G acts unitarily on X 0 , i.e., ρ 0 ( g -1 ) = ρ 0 ( g ) glyph[latticetop] , then A = A glyph[latticetop] is self-adjoint. This implies that A : β ↦→ β is an orthogonal projection from R d 0 onto R d 0 G . In particular, we have

Proposition 3.6 combined with (10) demonstrates that a linear steerable network f steer ( · ; W ) = 〈· , P steer ( W ) 〉 can realize any G -invariant linear predictor β ∈ R d 0 G ; that is, {P steer ( W ) : W ∈ W steer } = R d 0 G . Therefore (16) and (17) are equivalent optimization problems parameterized in different ways. However, minimizing (16) using gradient-based methods may potentially lead to different classifiers compared to those obtained from optimizing (17) directly.

Gradient flow. Given an initialization W (0) ∈ W steer, the gradient flow { W ( t ) } t ≥ 0 for (16) is the solution of the following ordinary differential equation (ODE),

The purpose of this work is to inspect the asymptotic behavior of the G -invariant linear predictors β steer ( t ) = P steer ( W ( t )) parameterized by the linear steerable network trained under gradient flow (21). In particular, we aim to analyze the directional limit of β steer ( t ) as t →∞ , i.e. , lim t →∞ β steer ( t ) ‖ β steer ( t ) ‖ . Before ending this section, we make the following assumption on the gradient flow W ( t ) that also appears in the prior works Ji and Telgarsky [2020], Lyu and Li [2020], Yun et al. [2021].

Assumption 3.7. The gradient flow W ( t ) satisfies L P steer ( W ( t 0 ); S ) < 1 for some t 0 > 0 .

This assumption implies that the data set S = { ( x i , y i ) : i ∈ [ n ] } can be separated by a G -invariant linear predictor P steer ( W ( t 0 )) , and our analysis is focused on the 'late phase" of the gradient flow training as t →∞ .

4 Implicit bias of linear steerable networks

Our main result on the implicit bias of gradient flow on linear steerable networks in binary classification is summarized in the following theorem.

Theorem 4.1. Under Assumption 3.1 and Assumption 3.7, let β steer ( t ) = P steer ( W ( t )) be the timeevolution of the G -invariant linear predictors parameterized by a linear steerable network trained with gradient flow on the data set S = { ( x i , y i ) : i ∈ [ n ] } ; cf. Eq. (21) . Then

(a) The directional limit β ∞ steer = lim t →∞ β steer ( t ) ‖ β steer ( t ) ‖ exists and β ∞ steer ∝ 1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ , where γ ∗ is the maxL 2 -margin SVM solution for the transformed data S = { ( x i , y i ) : i ∈ [ n ] } :

Furthermore, if G acts unitarily on the input space X 0 , i.e., g -1 = g glyph[latticetop] , then β ∞ steer ∝ γ ∗ .

(b) Equivalently, β ∞ steer is proportional to the unique minimizer β ∗ of the problem

where Proj Range ( A ) is the projection from R d 0 to Range ( A ) = Range ( 1 | G | ∑ g ∈ G g ) . Moreover, if G acts unitarily on X 0 , then

Namely, β ∞ steer achieves the maximum L 2 -margin among all G -invariant linear predictors.

Theorem 4.1 suggests that gradient flow implicitly guides a linear steerable network toward the unique G -invariant classifier with a maximum margin defined by the input representation ρ 0 .

Remark 4.2. According to Remark 3.4, when G = { e } is a trivial group, then a linear G-CNN (which is a special case of linear steerable networks) reduces to a fully-connected network f fc ( x ; W ) . Since the representation of a trivial group is always unitary, we have the following corollary which also appeared in Ji and Telgarsky [2020] and Yun et al. [2021].

Corollary 4.3. Let { W ( t ) } t ≥ 0 ⊂ W fc be the gradient flow of the parameters when training a linear fully-connected network on the data set S = { ( x i , y i ) , i ∈ [ n ] } , i.e.,

Then the classifier β fc ( t ) = P fc ( W ( t )) converges in a direction that aligns with the maxL 2 -margin SVM solution γ ∗ for the original data set S = { ( x i , y i ) : i ∈ [ n ] } ,

Remark 4.4. While Theorem 4.1 provides a complete characterization of the implicit bias exhibited by gradient flow in linear steerable networks, it is imperative to note that the convergence rate to the directional limit is, in fact, exponentially slow. This is consistent with the findings in, e.g., [Soudry et al., 2018, Yun et al., 2021]. A comprehensive analysis of gradient flow behavior in a non-asymptotic regime falls outside the scope of this current study.

5 The equivalence between steerable networks and data augmentation

Compared to hard-wiring symmetry priors into model architectures through equivariant steerable networks, an alternative approach to incorporate symmetry into the learning process is by training a non-equivariant model with the aid of data augmentation . In this section, we demonstrate that these two approaches are equivalent for binary classification under a unitary assumption for ρ 0 .

Corollary 5.1. Let β ∞ steer = lim t →∞ β steer ( t ) ‖ β steer ( t ) ‖ be the directional limit of the linear predictor β steer ( t ) = P steer ( W ( t )) parameterized by a linear steerable network trained using gradient flow on the original data set S = { ( x i , y i ) , i ∈ [ n ] } . Correspondingly, let β ∞ fc = lim t →∞ β fc ( t ) ‖ β fc ( t ) ‖ , β fc ( t ) = P fc ( W ( t )) (13) , be that of a linear fully-connected network trained on the augmented data set S aug = { ( g x i , y i ) , i ∈ [ n ] , g ∈ G } . If G acts unitarily on X 0 , then

In other words, the effect of using a linear steerable network for group-invariant binary classification is exactly the same as conducting data-augmentation for non-invariant models.

Remark 5.2. The equivalence between data augmentation and training with a linear steerable network is valid only in an asymptotic sense, yet the underlying training dynamics differ substantially. Specifically, β steer ( t ) is assured to maintain G -invariance throughout the training process, whereas β fc ( t ) achieves G -invariance solely in the limiting case as t →∞ . Moreover, the equivalence only holds for 'full-batch" data-augmentation over the entire group orbits.

Remark 5.3. For Corollary 5.1 to hold, it is crucial that ρ 0 ∈ GL ( X 0 ) is unitary, as otherwise the limit direction β ∞ fc of a linear fully-connected network trained on the augmented data set is generally not G -invariant (and hence cannot be equal to β ∞ steer ∈ R d 0 G ); see Example 5.4.

Example 5.4. Let G = Z 2 = { 0 , 1 } . Consider a (non-unitary) G -representation ρ 0 on X 0 = R 2 ,

Let S = { ( x , y ) } = { ((1 , 2) glyph[latticetop] , +1) } be a training set with only one point. By Theorem 4.1, the limit direction β ∞ steer of a linear-steerable-network-parameterized linear predictor satisfies

where γ ∗ = (0 , 1 3 ) glyph[latticetop] is the max-margin SVM solution for the transformed data set S = { ( x , y ) } = { ((0 , 3) glyph[latticetop] , +1) } . On the contrary, by Corollary 4.3, the limit direction β ∞ fc of a linear fully-connected network trained on the augmented data set S aug = { ((1 , 2) glyph[latticetop] , +1) , (( -1 , 4) glyph[latticetop] , +1) } aligns with the max-margin SVM solution γ ∗ aug for S aug ,

However, we demonstrate below that the equivalence between linear steerable networks and data augmentation can be re-established for non-unitary ρ 0 by defining a new inner product on R d 0 ,

When ρ 0 is unitary, 〈 v , w 〉 ρ 0 = 〈 v , w 〉 is the normal Euclidean inner product. With this new inner product, we modify the linear fully-connected network and its empirical loss on the augmented data set S aug = { ( g x i , y i ) : g ∈ G,i ∈ [ n ] } as

The following corollary shows that, for non-unitary ρ 0 on X 0 , the implicit bias of a linear steerable network is again the same as that of a modified linear fully-connected network f ρ 0 fc ( x ; W ) trained under data augmentation.

Corollary 5.5. Let β ∞ steer be the same as that in Corollary 5.1. Let β ρ 0 , ∞ fc = lim t →∞ β ρ 0 fc ( t ) ‖ β ρ 0 fc ( t ) ‖ be the limit direction of β ρ 0 fc ( t ) = P fc ( W ( t )) under the gradient flow of the modified empirical loss L ρ 0 P fc ( W ; S aug ) (34) for a linear fully-connected network on the augmented data set S aug . Then

Consequently, we have 〈 x , β ∞ steer 〉 ∝ 〈 x , β ρ 0 , ∞ fc 〉 ρ 0 for all x ∈ R d 0 .

6 Improved margin and generalization

We demonstrate in this section the improved margin and generalization of linear steerable networks over their non-invariant counterparts. In what follows, we assume ρ 0 to be unitary.

The following theorem shows that the margin of a linear-steerable-network-parameterized predictor β ∞ steer on the augmented data set S aug is always larger than that of a linear fully-connected network β ∞ fc , suggesting improved L 2 -robustness of the steerable-network-parameterized classifier.

Theorem 6.1. Let β ∞ steer be the directional limit of a linear-steerable-network-parameterized predictor trained on the original data set S = { ( x i , y i ) , i ∈ [ n ] } ; let β ∞ fc be that of a linear fully-connected network also trained on the same data set S . Let M steer and M fc , respectively, be the (signed) margin of β ∞ steer and β ∞ fc on the augmented data set S aug = { ( g x i , y i ) : i ∈ [ n ] , g ∈ G } , i.e.,

Then we always have M steer ≥ M fc .

Finally, we aim to quantify the improved generalization of linear steerable networks compared to fully-connected networks in binary classification of linearly separable group-invariant distributions defined below.

Definition 6.2. A distribution D on R d 0 ×{± 1 } is called linearly separable if there exists β ∈ R d 0 such that

It is easy to verify (by Lemma F.1) that if D is G -invariant and linearly separable, then D can be separated by a G -invariant linear classifier. The following theorem establishes the generalization bound of linear steerable networks in separable group-invariant binary classification.

Theorem 6.3. Let D be a G -invariant distribution over R d 0 ×{± 1 } that is linearly separable by an invariant classifier β 0 ∈ R d 0 G . Define

Let S = { ( x i , y i ) } n i =1 be i.i.d. samples from D , and let β ∞ steer be the limit direction of a steerablenetwork-parameterized linear predictor trained using gradient flow on S . Then, for any δ > 0 , we have with probability at least 1 -δ (over random samples S ∼ D n ) that glyph[negationslash]

Remark 6.4. In comparison, let β ∞ fc be the limit direction of a fully-connected-networkparameterized linear predictor trained on S . Then with probability at least 1 -δ , we have glyph[negationslash]

where R = inf { r > 0 : ‖ x ‖ ≤ r with probability 1 } . This is the classical generalization result for max-margin SVM (see, e.g., Shalev-Shwartz and Ben-David [2014].) Eq. (40) can also be viewed as a special case of Eq. (39) , as a fully-connected network is a G-CNN with G = { e } (cf. Remark 4.2), and therefore x = x and R = R .

By Proposition 3.6, the map x → x is an orthogonal projection, and thus we always have R ≤ R . Therefore the generalization bound for steerable networks in (39) is always smaller than that of the fully-connected network in (40) .

Remark 6.5. A comparison between Eq. (39) and Eq. (40) reveals that the improved generalization of linear steerable network does not necessarily depend on the group size | G | . Instead, it depends on how far the distribution D 's support is from the subspace R d 0 G , such that R could be much smaller than R . In fact, if the support of D is contained in R d 0 G , then R = R , and the steerable network does not achieve any generalization gain. This is consistent with Theorem 4.1, as in this case, the transformed data set S = { ( x i , y i ) : i ∈ [ n ] } is the same as the original data set S .

7 Conclusion and future work

In this work, we analyzed the implicit bias of gradient flow on general linear group-equivariant steerable networks in group-invariant binary classification. Our findings indicate that the parameterized predictor converges in a direction that aligns with the unique group-invariant classifier with a maximum margin that is dependent on the input representation. As a corollary of our main result,

we established the equivalence between data augmentation and learning with steerable networks in our setting. Finally, we demonstrated that linear steerable networks outperform their non-invariant counterparts in terms of improved margin and generalization bound.

Alimitation of our result is that the implicit bias of gradient flow studied herein holds in an asymptotic sense, and the convergence rate to the directional limit might be extremely slow. This is consistent with the findings in, e.g. , [Soudry et al., 2018, Yun et al., 2021]. Understanding the behavior of gradient flow in a non-asymptotic regime is an important direction for future work. Furthermore, in our current framework, we assume that the first-layer equivariant map is represented by the G -lifting map. This assumption ensures that the linear steerable network possesses sufficient capacity to parameterize all G -invariant linear classifiers. Exploring the implicit bias of steerable networks without this assumption would be a compelling next step. The removal of this constraint could facilitate the generalization of our findings from finite groups to compact groups.

References

Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning , pages 242-252. PMLR, 2019.

Cheng, Q. Qiu, R. Calderbank, and G. Sapiro. RotDCF: Decomposition of convolutional filters for rotation-equivariant deep networks. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=H1gTEj09FX .

Chizat and F. Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in neural information processing systems , 31, 2018.

Cohen and M. Welling. Group equivariant convolutional networks. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 2990-2999, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https://proceedings.mlr.press/v48/cohenc16.html .

S. Cohen and M. Welling. Steerable CNNs. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=rJQKYt5ll .

S. Cohen, M. Geiger, and M. Weiler. A general theory of equivariant CNNs on homogeneous spaces. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/ b9cfe8b6042cf759dc4c0cccb27a6737-Paper.pdf .

Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pages 1675-1685. PMLR, 2019.

Elesedy. Group symmetry in pac learning. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning , 2022.

Elesedy and S. Zaidi. Provably strict generalisation benefit for equivariant models. In International Conference on Machine Learning , pages 2959-2969. PMLR, 2021.

Gunasekar, J. Lee, D. Soudry, and N. Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning , pages 1832-1841. PMLR, 2018a.

Gunasekar, J. D. Lee, D. Soudry, and N. Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems , 31, 2018b.

Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Ji and M. Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations , 2019a. URL https://openreview.net/forum?id= HJflg30qKX .

Ji and M. Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference on Learning Theory , pages 1772-1798. PMLR, 2019b.

Ji and M. Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems , 33:17176-17186, 2020.

Lawrence, K. Georgiev, A. Dienes, and B. T. Kiani. Implicit bias of linear equivariant networks. arXiv preprint arXiv:2110.06084 , 2021.

Lyu and J. Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations , 2020. URL https://openreview.net/ forum?id=SJeLIgBKPS .

Mei, T. Misiakiewicz, and A. Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on Learning Theory , pages 2388-2464. PMLR, 2019.

Mohri and A. M. Medina. Learning theory and algorithms for revenue optimization in second price auctions with reserve. In International conference on machine learning , pages 262-270. PMLR, 2014.

S. Nacson, N. Srebro, and D. Soudry. Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3051-3059. PMLR, 2019.

Sannai, M. Imaizumi, and M. Kawano. Improved generalization bounds of group invariant/equivariant deep networks via quotient feature spaces. In Uncertainty in Artificial Intelligence , pages 771-780. PMLR, 2021.

Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Sokolic, R. Giryes, G. Sapiro, and M. Rodrigues. Generalization error of invariant classifiers. In Artificial Intelligence and Statistics , pages 1094-1103. PMLR, 2017.

Sosnovik, M. Szmaja, and A. Smeulders. Scale-equivariant steerable networks. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id= HJgpugrKPS .

Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1):2822-2878, 2018.

Vardi. On the implicit bias in deep-learning algorithms. arXiv preprint arXiv:2208.12591 , 2022.

Weiler and G. Cesa. General E(2)-equivariant steerable CNNs. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings. neurips.cc/paper/2019/file/45d6637b718d0f24a237069fe41b0db4-Paper.pdf .

Weiler, F. A. Hamprecht, and M. Storath. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 849-858, 2018.

E. Worrall, S. J. Garbin, D. Turmukhambetov, and G. J. Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5028-5037, 2017.

Yun, S. Krishnan, and H. Mobahi. A unifying view on implicit bias in training linear neural networks. In International Conference on Learning Representations , 2021. URL https:// openreview.net/forum?id=ZsZM-4iMQkH .

Zhu, Q. Qiu, R. Calderbank, G. Sapiro, and X. Cheng. Scaling-translation-equivariant networks with decomposed convolutional filters. Journal of machine learning research , 23(68):1-45, 2022.

A G-CNNs

Aspecial case of the steerable networks is the group convolutional neural network (G-CNN), wherein equivariance is achieved through group convolutions . More specifically, for each l ∈ [ L -1] , the hidden representation space ( X l , ρ l ) is set to

The representation ρ l ∈ GL ( X l ) in (41) is known as the regular representation of G . Intuitively, x l ∈ X l can be viewed as a matrix of size d l ×| G | , and ρ l ( g ) is a permutation of the columns of x l . With this choice of { ( X l , ρ l ) } L l =0 , a (linear) G-CNN is built as follows.

First layer: the first-layer equivariant map Ψ G-CNN 1 : X 0 →X 1 of a G-CNN is defined as

where w 1 = ( w j,k 1 ) j,k ∈ R d 0 × d 1 are the trainable parameters of the first layer. Eq. (42) is called a G -lifting map as it lifts a Euclidean signal x ∈ R d 0 to a function Ψ G-CNN 1 ( x ; w 1 ) on G .

Hidden layers: For l ∈ { 2 , · · · , L -1 } , define Ψ G-CNN l : X l -1 →X l as the group convolutions ,

where w l ( g ) ∈ R d l -1 × d l , ∀ g ∈ G . Equivalently, w l ∈ R d l -1 × d l ×| G | can be viewed as a 3D tensor. Last layer: Define Ψ G-CNN L : X L -1 →X L as the group-pooling followed by a fully-connected layer,

where w L ∈ R d L -1 is the weight of the last layer. An L -layer linear G-CNN is then the composition

Remark A.1. As a special case of linear steerable networks, a linear G-CNN f G-CNN ( x ; W ) (45) can be written as

where

B Proofs in Section 3

Proposition 3.3. Let f steer ( x ; W ) be the linear steerable network satisfying Assumption 3.1, where W = [ w l ] L l =1 ∈ W steer is the collection of all model parameters. There exists a multi-linear map M : ( w 2 , · · · , w L ) ↦→ M ( w 2 , · · · , w L ) ∈ R d 1 such that for all x ∈ R d 0 and W ∈ W steer ,

where x := 1 | G | ∑ g ∈ G g x is the average of all elements on the group orbit of x ∈ R d 0 .

Proof. Since the linear steerable network f steer ( x ; W ) is G -invariant, we have f steer ( x ; w ) = f steer ( g x ; w ) for all g ∈ G . Therefore,

where the second equality is due to the linearity of f steer ( x ; w ) in x . We thus have

where Φ( w 2 , · · · , w L ) ∈ X 1 = ( R d 1 ) G is multi-linear with respect to ( w 1 , · · · , w L ) . Under Assumption 3.1, we have

where M ( w 2 , · · · , w L ) := ∑ g ∈ G Φ( w 2 , · · · , w L )( g ) is also multi-linear in ( w 2 , · · · , w L ) .

Proposition 3.6. Let R d 0 G ⊂ R d 0 be the subspace of G -invariant linear predictors, i.e., R d 0 G = { β ∈ R d 0 : β glyph[latticetop] x = β glyph[latticetop] g x , ∀ x ∈ R d 0 , ∀ g ∈ G } . Then

(a) R d 0 G is characterized by

(b) Let A : R d 0 → R d 0 be the group-averaging map,

Then its adjoint A glyph[latticetop] : β ↦→ 1 | G | ∑ g ∈ G g glyph[latticetop] β is a projection operator from R d 0 to R d 0 G . In other words, Range ( A glyph[latticetop] ) = R d 0 G and A glyph[latticetop] ◦ A glyph[latticetop] = A glyph[latticetop] .

(c) If G acts unitarily on X 0 , i.e., ρ 0 ( g -1 ) = ρ 0 ( g ) glyph[latticetop] , then A = A glyph[latticetop] is self-adjoint. This implies that A : β ↦→ β is an orthogonal projection from R d 0 onto R d 0 G . In particular, we have

Proof. To prove (a), for the first equality, a vector β ∈ R d 0 G if and only if 0 = β glyph[latticetop] ( I -g ) x = 〈 ( I -g ) glyph[latticetop] β , x 〉 , ∀ x ∈ R d 0 , ∀ g ∈ G . This is equivalent to β ∈ ⋂ g ∈ G ker( I -g glyph[latticetop] ) , and therefore R d 0 G = ⋂ g ∈ G ker( I -g glyph[latticetop] ) .

For the second equality, if β ∈ ⋂ g ∈ G ker( I -g glyph[latticetop] ) , then ( I -g glyph[latticetop] ) β = 0 , ∀ g ∈ G . Hence

On the other hand, if β = 1 | G | ∑ h ∈ G h glyph[latticetop] y for some y ∈ R d 0 , then for all g ∈ G ,

Thus β ∈ ⋂ g ∈ G ker( I -g glyph[latticetop] ) .

Point (b) can be easily derived from (57) and (60).

To prove (c), notice that

Hence A = A glyph[latticetop] is self-adjoint. This combined with point (b) implies that A = A glyph[latticetop] : β ↦→ β is an orthogonal projection from R d 0 onto R d 0 G .

C Proof of Theorem 4.1

Theorem 4.1. Under Assumption 3.1 and Assumption 3.7, let β steer ( t ) = P steer ( W ( t )) be the timeevolution of the G -invariant linear predictors parameterized by a linear steerable network trained with gradient flow on the data set S = { ( x i , y i ) : i ∈ [ n ] } ; cf. Eq. (21) . Then

(a) The directional limit β ∞ steer = lim t →∞ β steer ( t ) ‖ β steer ( t ) ‖ exists and β ∞ steer ∝ 1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ , where γ ∗ is the maxL 2 -margin SVM solution for the transformed data S = { ( x i , y i ) : i ∈ [ n ] } :

Furthermore, if G acts unitarily on the input space X 0 , i.e., g -1 = g glyph[latticetop] , then β ∞ steer ∝ γ ∗ .

(b) Equivalently, β ∞ steer is proportional to the unique minimizer β ∗ of the problem

where Proj Range ( A ) is the projection from R d 0 to Range ( A ) = Range ( 1 | G | ∑ g ∈ G g ) . Moreover, if G acts unitarily on X 0 , then

Namely, β ∞ steer achieves the maximum L 2 -margin among all G -invariant linear predictors.

Before proving Theorem 4.1, we need the following lemma which holds for general L -homogeneous networks, of which linear steerable networks are a special case. Note that in what follows, ‖ · ‖ always denotes the Euclidean norm of a tensor viewed as a one-dimensional vector.

Lemma C.1 (paraphrased from Lyu and Li [2020] and Ji and Telgarsky [2020]) . Under Assumption 3.7, we have the following results of directional convergence and alignment.

(a) L P steer ( W ( t ); S ) → 0 , as t →∞ . Consequently, ‖ W ( t ) ‖ → ∞ and ‖P steer ( W ( t )) ‖ → ∞ .

(b) The directional convergence and alignment of the parameters W ( t ) and the gradients ∇ W L P steer ( W ( t )) ,

for some W ∞ ∈ W steer with ‖ W ∞ ‖ = 1 .

(c) The limit W ∞ is along the direction of a first-order stationary point of the constrained optimization problem

In other words, there exists a scaling factor τ > 0 such that τ W ∞ satisfies the Karush-KuhnTucker (KKT) conditions of (68) .

Proof of Theorem 4.1. The first part of the proof is inspired by Gunasekar et al. [2018b]. To prove (a) , let W ∞ = [ w ∞ l ] L l =1 = lim t →∞ W ( t ) ‖ W ( t ) ‖ be the limit direction of W ( t ) , and let ˜ W ∞ = [ ˜ w ∞ l ] L l =1 = τ W ∞ , τ > 0 , be the stationary point of problem (68) by Lemma C.1. The KKT condition for ˜ W ∞ implies the existence of dual variables α i ≥ 0 , i ∈ [ n ] , such that

Primal feasibility:

Stationarity:

Complementary slackness :

We claim that γ ∗ = P steer ( ˜ W ∞ ) is a stationary point of the following problem

That is, there exists ˜ α i ≥ 0 , i ∈ [ n ] , such that the following conditions are satisfied.

Primal feasibility:

Stationarity:

Complementary slackness :

Indeed, Eq. (73) holds due to Eq. (69) and 〈 P steer ( ˜ W ∞ ) , x i 〉 = 〈 P steer ( ˜ W ∞ ) , x i 〉 . Moreover, by the definition of P steer ( W ) and P steer ( W ) in (10), we have for any z ∈ R d 0 ,

Therefore Eq. (70) implies

Hence

glyph[negationslash]

where c = ‖ M ( ˜ w ∞ 2 , · · · , ˜ w ∞ L ) ‖ 2 ≥ 0 . From (73) we know that P steer ( ˜ W ∞ ) = 0 and hence c > 0 . Let ˜ α i = cα i , ∀ i ∈ [ n ] . Then ˜ α i ≥ 0 , and the stationarity (74) is satisfied due to Eq. (79), and the complementary slackness also holds due to ˜ α i being a positive scaling of α i for all i ∈ [ n ] . Therefore γ ∗ = P steer ( ˜ W ∞ ) is a stationary point of (72). Since problem (72) is strongly convex, γ ∗ = P steer ( ˜ W ∞ ) is in fact the unique minimizer of (72). Hence the limit direction of the predictor P steer ( W ( t )) is glyph[negationslash]

glyph[negationslash]

where the third equality is due to P steer being L -homogeneous (12), and the fourth equality comes from the continuity of P steer and that P steer ( ˜ W ∞ ) = ( 1 | G | ∑ g ∈ G g glyph[latticetop] ) P steer ( ˜ W ∞ ) = 0 (because otherwise (69) can not hold.)

Finally, if G acts unitarily on X 0 , then Eq. (74) combined with Proposition 3.6 implies that γ ∗ = P steer ( ˜ W ∞ ) ∈ R d 0 G . Hence

where the last equality is again due to Proposition 3.6. This concludes the proof of (a).

To prove (b) , we first show that problem (65) has a unique minimizer. To prove this, notice that Proj Range ( A ) : R d 0 G → Range ( A ) is injective; indeed, for any β ∈ R d 0 G ,

Thus 0 = A glyph[latticetop] β = β , where the second equality is due to A glyph[latticetop] being a projection onto Range ( A glyph[latticetop] ) = R d 0 G (Proposition 3.6) and β ∈ R d 0 G . Therefore the objective function in (65) is strongly convex on R d 0 G and there exists a unique minimizer.

To show β ∞ steer ∝ β ∗ , it suffices to verify that 1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ = β ∗ is the minimizer of (65). To this end, we notice that

1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ = A glyph[latticetop] γ ∗ ∈ R d 0 G by Proposition 3.6; · for all i ∈ [ n ]

Thus 1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ satisfies the constraints in (65). Assume for the sake of contradiction that 1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ = β ∗ , then by the uniqueness of the minimizer of (65) we have

where ˜ γ := Proj Range ( A ) β ∗ . We make the following two claims to be proved shortly:

These two claims combined with (86) imply that ˜ γ satisfies the constraint in (64) and has a smaller norm compared to γ ∗ , ‖ ˜ γ ‖ < ‖ γ ∗ ‖ . This contradicts the fact that γ ∗ is the minimizer of problem (64). Therefore we have 1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ = β ∗ .

If we assume in addition that ρ 0 is unitary on X 0 , then A = A glyph[latticetop] : β ↦→ β is an orthogonal projection from R d 0 onto R d 0 G = Range ( A ) . Therefore Proj Range ( A ) β = β for β ∈ R d 0 G , and hence problems (65) and (66) are equivalent.

Finally, we prove the above two claims.

Hence

where the second equality is due to β ∗ ∈ R d 0 G = Range ( A glyph[latticetop] ) and A glyph[latticetop] = A glyph[latticetop] ◦ A glyph[latticetop] is a projection; cf. Proposition 3.6. Therefore, for all i ∈ [ n ] ,

Proof of Claim 2: Proj Range ( A ) A glyph[latticetop] γ ∗ = γ ∗ .

We first note that both Proj Range ( A ) A glyph[latticetop] γ ∗ and γ ∗ are in the affine space A glyph[latticetop] γ ∗ +ker( A glyph[latticetop] ) . Indeed,

This also implies

Moreover, for any i ∈ [ n ] ,

Hence Proj Range ( A ) A glyph[latticetop] γ ∗ also satisfies the constraint in (64). Since the orthogonal projection Proj Range ( A ) A glyph[latticetop] γ ∗ achieves the minimal norm among all vectors in the affine space A glyph[latticetop] γ ∗ +ker( A glyph[latticetop] ) which includes γ ∗ , this can only happen when Proj Range ( A ) A glyph[latticetop] γ ∗ = γ ∗ is the unique minimizer of (64).

This concludes the proof of Theorem 4.1.

D Proofs in Section 5

Corollary 5.1. Let β ∞ steer = lim t →∞ β steer ( t ) ‖ β steer ( t ) ‖ be the directional limit of the linear predictor β steer ( t ) = P steer ( W ( t )) parameterized by a linear steerable network trained using gradient flow on the original data set S = { ( x i , y i ) , i ∈ [ n ] } . Correspondingly, let β ∞ fc = lim t →∞ β fc ( t ) ‖ β fc ( t ) ‖ , β fc ( t ) = P fc ( W ( t )) (13) , be that of a linear fully-connected network trained on the augmented data set S aug = { ( g x i , y i ) , i ∈ [ n ] , g ∈ G } . If G acts unitarily on X 0 , then

In other words, the effect of using a linear steerable network for group-invariant binary classification is exactly the same as conducting data-augmentation for non-invariant models.

Proof. By Theorem 4.1, a positive scaling γ ∗ = τ β ∞ steer of β ∞ steer satisfies the KKT condition of (22). That is, there exists α i ≥ 0 , i ∈ [ n ] , such that

Primal feasibility:

Stationarity:

Complementary slackness :

Using Corollary 4.3, we only need to show that γ ∗ is also the solution of

That is, there exists ˜ α i,g ≥ 0 , ∀ i ∈ [ n ] , ∀ g ∈ G , such that

Primal feasibility:

Stationarity:

Complementary slackness :

Indeed, we set ˜ α i,g = 1 | G | α i ≥ 0 . Since γ ∗ ∈ R d 0 G if ρ 0 is unitary (this can also be observed from Eq. (97)), we have y i 〈 γ ∗ , g x i 〉 = y i 〈 γ ∗ , x i 〉 , ∀ g ∈ G, ∀ i ∈ [ n ] . Hence we have primal feasibility (100) from (96),

Stationarity (101) holds since

where the last equality comes from (97). Finally, if y i 〈 γ ∗ , g x i 〉 > 1 , then (103) and (98) imply that

This proves the condition for complementary slackness (102).

Corollary 5.5. Let β ∞ steer be the same as that in Corollary 5.1. Let β ρ 0 , ∞ fc = lim t →∞ β ρ 0 fc ( t ) ‖ β ρ 0 fc ( t ) ‖ be the limit direction of β ρ 0 fc ( t ) = P fc ( W ( t )) under the gradient flow of the modified empirical loss L ρ 0 P fc ( W ; S aug ) (34) for a linear fully-connected network on the augmented data set S aug . Then

Consequently, we have 〈 x , β ∞ steer 〉 ∝ 〈 x , β ρ 0 , ∞ fc 〉 ρ 0 for all x ∈ R d 0 .

Proof. The proof is similar to that of Corollary 5.1. By Theorem 4.1, the limit direction β ∞ steer is proportional to 1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ , where γ ∗ satisfies the same KKT condition ((96), (97) and (98)) of problem (22) as in the proof of Corollary 5.1. On the other hand, note that the loss function L ρ 0 P fc ( W ; S aug ) (34) for the modified fully-connected network on the augmented data set S aug is equivalent to

where

Therefore, by Corollary 4.3, β ρ 0 , ∞ fc is proportional to the solution ˜ γ ∗ aug of

We claim that ˜ γ ∗ aug = A -1 ( 1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ ) . To see this, let ˜ α i,g = α i , ∀ i ∈ [ n ] , where α i ≥ 0 is the dual variable for γ ∗ of problem (22). We verify below that the KKT conditions for problem (111) are satisfied for the primal-dual pair A -1 ( 1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ ) and ( ˜ α i,g ) i ∈ [ n ] ,g ∈ G .

Primal feasibility: for any i ∈ [ n ] and h ∈ G ,

where the first equality is due to A glyph[latticetop] = A being symmetric, and the last inequality is due to (96).

Stationarity:

where the first equality is due to (97), and the last equality comes from the definition of the dual variable ˜ α i,g = 1 | G | α i , ∀ i ∈ [ n ] , ∀ g ∈ G .

Complementary slackness: if y i 〈 A -1 ( 1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ ) , Ag x i 〉 > 1 for some i ∈ [ n ] and g ∈ G , then (114) and (98) imply that ˜ α i,g = 1 | G | α i = 0 .

Hence ˜ γ ∗ aug = A -1 ( 1 | G | ∑ g ∈ G g glyph[latticetop] γ ∗ ) is indeed the solution of (111). Therefore,

This completes the proof.

E Proofs of Theorem 6.1

Theorem 6.1. Let β ∞ steer be the directional limit of a linear-steerable-network-parameterized predictor trained on the original data set S = { ( x i , y i ) , i ∈ [ n ] } ; let β ∞ fc be that of a linear fully-connected network also trained on the same data set S . Let M steer and M fc , respectively, be the (signed) margin of β ∞ steer and β ∞ fc on the augmented data set S aug = { ( g x i , y i ) : i ∈ [ n ] , g ∈ G } , i.e.,

Then we always have M steer ≥ M fc .

Proof. By Corollary 4.3 and Corollary 5.1, we have

Moreover, the margin M steer of the steerable network β ∞ steer on the augmented data set S aug is M steer = 1 ‖ γ ∗ steer ‖ . Consider the following three cases.

Case 1: min i ∈ [ n ] ,g ∈ G y i 〈 γ ∗ fc , g x i 〉 ≥ 1 . In this case, γ ∗ fc also satisfies the more restrictive constraint in (123). Therefore γ ∗ fc = γ ∗ steer , which implies

Case 2: min i ∈ [ n ] ,g ∈ G y i 〈 γ ∗ fc , g x i 〉 ≤ 0 . This implies that γ ∗ fc , and hence β ∞ fc has a nonpositive margin on D aug. Thus

Case 3: 0 < min i ∈ [ n ] ,g ∈ G y i 〈 γ ∗ fc , g x i 〉 < 1 . In this case, define

Then ˜ γ ∗ fc satisfies the constraint in (123), and hence

Therefore

This completes the proof.

F Proof of Theorem 6.3

To prove Theorem 6.3, we need first the following preliminaries.

Lemma F.1. If D is G -invariant and linearly separable, then D can be linearly separated by a G -invariant classifier. That is, there exists β ∈ R d 0 G such that

Proof. Since D is linearly separable, there exists β 0 ∈ R d 0 such that P ( x ,y ) ∼D [ y 〈 x , β 0 〉 ≥ 1] = 1 . Let β = 1 | G | ∑ g ∈ G g glyph[latticetop] β 0 ∈ R d 0 G , and we aim to show that D can also be linearly separated by β . Indeed, for all ( x , y ) ∈ R d 0 ×{± 1 } ,

Let E = { ( x , y ) : y 〈 x , β 0 〉 < 1 } ⊂ R d 0 ×{± 1 } , then

where the second equality in (136) is due to D being G -invariant. Therefore D can be separated by β ∈ R d 0 G .

Definition F.2 (Empirical Rademacher complexity) . Let F ⊂ R Z be a class of real-valued functions on Z , and let S = ( z 1 , · · · , z n ) be a set of n samples from Z . The (empirical) Rademacher complexity ̂ R n ( F , S ) of F over S is defined as

where σ = ( σ 1 , · · · , σ n ) are the i.i.d. Rademacher variables satisfying P [ σ i = 1] = P [ σ i = -1] = 1 / 2 .

Lemma F.3 (Talagrand's Contraction Lemma, Lemma 8 in Mohri and Medina [2014]) . Let Φ i : R → R , i ∈ [ n ] , be 1 -Lipschitz functions, F ⊂ R Z be a function class, and S = ( z 1 , . . . , z n ) ∈ Z n be a set of n samples from Z . Then

Lemma F.4. Let H = { x ↦→〈 β , x 〉 : β ∈ R d 0 G , ‖ β ‖ ≤ B } be the function space of G -invariant linear functions of bounded norm, and let S x = { x 1 , · · · , x n } ⊂ R d 0 be a set of n samples from R d 0 . Then

Proof. By the definition of empirical Rademacher complexity (138), we have

where (142) is due to β ∈ R d 0 G . Since σ 1 , · · · , σ n are i.i.d., we have

glyph[negationslash]

Eq. (145) combined with (148) completes the proof.

We also need the following standard result on the generalization bound based on the Rademacher complexity [Shalev-Shwartz and Ben-David, 2014].

Lemma F.5. Let H be a set of hypotheses, Z = R d 0 ×{± 1 } be the set of labeled samples, and

be a loss function satisfying 0 ≤ glyph[lscript] ( h, z ) ≤ c for all z ∈ Z and h ∈ H . Define the function class F = glyph[lscript] ( H , · ) := { z ↦→ glyph[lscript] ( h, z ) : h ∈ H} . Then, for any δ > 0 and any distribution D on Z , we have with probability at least 1 -δ over i.i.d. samples S = ( z 1 , · · · , z n ) ∼ D n that

where L D ( h ) := E z ∼D glyph[lscript] ( h, z ) and L S ( h ) := 1 n ∑ n i =1 glyph[lscript] ( h, z i ) are, respectively, the population and empirical loss of a hypothesis h ∈ H , and R n ( F ) := E S ∼D n ̂ R n ( F , S ) is the Rademacher complexity of F .

Finally, we can prove Theorem 6.3 restated below

Theorem 6.3. Let D be a G -invariant distribution over R d 0 ×{± 1 } that is linearly separable by an invariant classifier β 0 ∈ R d 0 G . Define

Let S = { ( x i , y i ) } n i =1 be i.i.d. samples from D , and let β ∞ steer be the limit direction of a steerablenetwork-parameterized linear predictor trained using gradient flow on S . Then, for any δ > 0 , we have with probability at least 1 -δ (over random samples S ∼ D n ) that glyph[negationslash]

Proof of Theorem 6.3. Let H = { β ∈ R d 0 G : ‖ β ‖ ≤ ‖ β 0 ‖} and with slight abuse of notation we identify β ∈ H with the invariant linear map x ↦→ 〈 x , β 〉 . According to Theorem 4.1, let β ∗ = τ β ∞ steer , τ > 0 , be the minimizer of (24). Since by assumption β 0 also satisfies the constraint in (24), we have ‖ β ∗ ‖ ≤ ‖ β 0 ‖ , and hence β ∗ ∈ H .

Consider the ramp loss glyph[lscript] defined as

It is easy to verify that glyph[lscript] upper bounds the 0-1 loss: for all β ∈ H and ( x , y ) ∈ R d 0 ×{± 1 } , glyph[negationslash]

This implies that

glyph[negationslash]

Since L S ( β ∗ ) is always 0 by definition (24) and β ∗ ∈ H , we have by Lemma F.5 that glyph[negationslash]

glyph[negationslash]

where F = glyph[lscript] ( H , · ) = { ( x , y ) ↦→ glyph[lscript] ( β , ( x , y )) : β ∈ H} . Therefore, to prove Theorem 6.3 it suffices to show that R n ( F ) ≤ R ‖ β 0 ‖ √ n . To this end, let S = { ( x i , y i ) : i ∈ [ n ] } be a set of n labeled samples, and let S x = { x i : i ∈ [ n ] } be the corresponding set of unlabeled inputs, then

where Φ i : R → R , a ↦→ min { 1 , max { 0 , 1 -y i a }} , is 1 -Lipschitz. Lemma F.3 thus implies that

glyph[negationslash]

Using Lemma F.4 and the fact that x ≤ R with probability 1 over ( x , y ) ∼ D , we arrive at

This completes the proof of Theorem 6.3.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies an implicit bias of gradient descent (GD) on linear equivariant neural networks. The authors show that the asymptotic GD solution of a linear steerable neural network, an instance of the equivariant nets, agrees with that of an FC net trained with data augmentation. They also show that the former achieves better margin and generalization bound than the latter.\n\nSTRENGTHS:\n1. This study tackles a critical problem --- analysis of the solution of equivariant neural networks. \n2. The paper is well-written and easy to follow.\n3. The theory is presented cleanly and solidly. I found a few typos, but The mostly I could read the theory part without confusion.\n\nWEAKNESSES:\n1. The paper does not contain empirical evaluations. The authors admit that the analysis relies on the asymptotic approximation, which might have a gap toward a real solution. I feel the experiments make the paper significantly more convincing. \n2. The paper limits the analysis to linear cases, but nonlinear networks are practically essential. Extending the theory toward nonlinearity might not be trivial, but at least we can evaluate them through experiments.\n\nQUESTIONS:\n1. The implications of Theorem 4.1 and Corollary 5.1 are interesting. Theorem 4.1 says that the GD solution of a steerable net converges to the max margin solution trained on the group-averaged data $\\bar{x}$. In contrast, Corollary 5.1 shows an FC net trained on the group-augmented data converges to the steerable net, which suggests that group averaging in the data space and data augmentation have roughly the same effect. This is not trivial because group averaging takes all possible augmentations for each data instance, but data augmentation puts a single transformation. In the sense of loss function, the former is written as $\\mathrm{loss}(E_{g}[gx])$ while the latter is $E_{g}[\\mathrm{loss}(gx)]$. Can you explain why these two are possibly equivalent?\n2. Related to the above question, I feel that group averaging is somehow similar to mixup, which augments data as $\\lambda x + (1-\\lambda) x'$ for $\\lambda \\in [0, 1]$. We may write group averaging as a special case of mixup, where we take an average in terms of groups instead of data instances. Do you have any insight in this direction?\n3. In line 186--187 $\\beta$ should be in $R^{d_0}_G$ instead of $R^{d_0}$\n4. Corollary 5.1: We may need some conditions for gs used in $S_aug$, such as g is drawn uniformly randomly from G.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper present several results on linear equivariant steerable networks. 1) linear equivariant NN converge in the same direction as linear networks trained with data-augmentation, 2) equivariant networks always attain a larger margin then group-augmented fully connected networks, 3) also the generalization bounds are better for equivariant networks when the data distributions is group invariant.\n\n* score has been updated after the response to my assessment.\n\nSTRENGTHS:\n* The paper presents a collection of statements about equivariant linear networks which could be useful as a general reference\n\n* Although the paper is extremely dense in information, it is readable and self-contained. The supplementary materials provide all the proofs.\n\nWEAKNESSES:\n1. The paper is essentially a mathematics paper and is the question to what extend it fits the scope of NeurIPS. As a non-mathematician, but someone interested in these topics, I did verify some of the proofs and the overall presentation is clear, however, I find it hard to assess the significance of the work.\n\n2. In particular, the limited scope on linear layers almost make the result seem trivial (though not from the mathematical perspective). To what extend do these results give insights on non-linear, multi-layer equivariant neural networks?\n\nQUESTIONS:\n1. Why is the emphasis on equivariant *steerable* networks? (instead of just equivariant networks) Steerability is I suppose a matter of definition, but classically the difference lies in whether you use the regular representation (G-convs) or irreducible representations (steerable G-convs) to generate the G-feature maps. Since the paper primarily focusses on the regular representation, I do not see why it is important to emphasize steerability. Moreover, line 79 says “more generally, steerable”, but I don’t think steerable G-CNNs are more general per se. That is, any linear layer is equivariant iff it is a group convolution [bekkers2018, thm1] regardless of the used feature type (one can always convert between regular representations and irreps via a generalized Fourier transform). See e.g. WeilerAndCesa 2019 or Brandstetter et al 2022 where this is recently discussed. \n\nBekkers, E. J. (2019, September). B-Spline CNNs on Lie groups. In International Conference on Learning Representations.\n\nWeiler, M., & Cesa, G. (2019). General e (2)-equivariant steerable cnns. Advances in neural information processing systems, 32.\n\nBrandstetter, J., Hesselink, R., van der Pol, E., Bekkers, E. J., & Welling, M. (2021, October). Geometric and Physical Quantities improve E (3) Equivariant Message Passing. In International Conference on Learning Representations.\n\n2. How does the following reference relate to the current work? It is also about generalization bounds of equivariant networks (through PAC learning) and thus seems related.\n\nElesedy, B. (2022, March). Group symmetry in PAC learning. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning.\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper presents a collection of theoretical findings that highlight the equivalence between using a linear steerable network for group-invariant binary classification and employing data augmentation for non-invariant models within the specified context. Additionally, the paper demonstrates that linear steerable networks outperform their non-invariant counterparts in terms of generalization bounds.

STRENGTHS:
* The paper builds upon previous studies that have explored the implicit regularization properties of different network architectures. It references works that have examined the convergence of linear predictors trained under gradient descent (GD) towards max-margin support vector machines (SVMs) and the implicit regularization of linear convolutional and tensor networks.

* The paper discusses the design of group-equivariant DNNs that preserve symmetry transformations in both input and output spaces. It highlights the empirical evidence suggesting that group-equivariant steerable networks improve generalization performance, especially in tasks with group symmetry and limited data.

* The paper presents its key result, which demonstrates that linear equivariant steerable networks trained under gradient flow (GF) converge towards the unique group-invariant classifier that maximizes the margin with respect to a norm defined by the input group representation.

WEAKNESSES:
This paper is theoretical in nature. It does not present any new empirical results or experiments to validate its claims. Unfortunately, the reviewer has little knowledge of the topic.

QUESTIONS:
Please improve the presentation of implications of the research findings and future research directions.

LIMITATIONS:
The limitations of the theoretical study are adequately discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the implicit bias on the use gradient flow for the purpose of binary (group-invariant) classification while using linear equivariant steerable networks. There is now a fair amount of work on implicit bias in similar settings in the non-equivariant case. There is also some work for linear equivariant networks. However, as the authors note, it is unclear whether a steerable network, trained by gradient descent, can achieve a minimizer with some parameter norm which is comparable to non-equivariant cases. The main results essentially show that linear steerable networks trained on some data converge to the same direction as a fully connected NN trained on augmented data. Moreover, such a network trained on the original data always attains a wider margin than a fully connected network trained on augmented data. An additional result is shown that a linear steerable network trained using gradient flow has better generalization than its non-equivariant counterpart when the data has an group invariant distribution. Section 3 describes the background and results formalizing and proving some of the above points are shown in later sections. \n\nMinor comments:\n- On page 3, line 122: \"Equivalently, we want the following diagram to be commutative..\" --> reword this since that diagram is already commutative, but that you want your network to adhere to it.\n\nSTRENGTHS:\n- The results, to the best of my knowledge, are solid and represent a good contribution complementing a host of body of work for non-equivariant models. \n- Generally speaking the paper is well written (although there is a flip side to this, which I will write in the weakness) and has a non-trivial contribution.\n\nWEAKNESSES:\n- One issue with the paper is its density. A lot of time is spent on background that doesn't need to be in the main paper. I understand the authors want to be thorough and that this is difficult. However, the essential idea and the proof techniques that the authors use, and the literature that it builds upon, are well-known, hence so much background is not necessary. I would suggest putting some of this material in the appendix and expanding on the results presented in the paper better and also comparing them to existing results. \n- Related to the above. I think the writing can be made more easy going (even if I had no problem in following it). I would understand for certain problems such heavy formalization is necessary. But I think this particular problem doesn't need it. It could be written simply. But in any case I would like to see more discussion about the actual results than the background. If the authors think so much of the background is necessary, I would be curious to hear their arguments. I think we can drop describing what a group or representation is, for example. It doesn't have to be done in every paper.\n\nQUESTIONS:\n- Please improve the discussion around the work of Lawrence (2021) at the end of page 2 and beginning of page 3. I am unable to parse the comments here. How are they \"not actually invariant or equivariant\" just because \"input is on a group\"? The inputs can be on a group in general in equivariant networks, or on spaces on which a group acts. I am also confused with the random claim that the paper works with Euclidean inputs with non-transitive actions. I don't think the formalism here is more general than operating on homogeneous spaces. Do the authors mean something else? I would appreciate a clarification.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the implicit bias of gradient flow optimization for linear equivariant steerable networks. The paper consists of the following steps and contributions: \n* The equivariant network is assumed to have regular representation at the input (Assumption 3.1). The architecture is characterized in Proposition 3.2, and is based on an overparameterized linear function applied to group averaged input.\n* The space of G-invariant linear classifiers is characterized in Proposition 3.6, which can be seen as the range of the adjoint of group projection operator. (same as the group projection for unitary representations)\n* The analysis of gradient descent is done based on using gradient flow (eq 25). Assuming the existence of G-invariant separating function (Assumption 3.7), Theorem 4.1 shows that the gradient flow converges to a maximum margin G-invariant classifier.\n* A fully connected linear network with data augmentation converges to the solution \\textbf{direction} for linear steerable network for unitary input representation (Corollary 5.1), and with modified loss for non-unitary input representation (Corollary 5.4). \n* Despite sharing the direction, the margin is still larger for the steerable model (Theorem 6.1).\n* Finally, the generalization bound for steerable and non-steerable models are obtained using classical results for max-margin SVM from the book of S. Shalev-Shwartz and S. Ben-David. (Theorem 6.3)\n\nAs I mention below, the paper has a collection of interesting result in a previously unexplored topic. However, I have some technical concerns about the definition of the model, and would like to hear the authors’ answers.\n\nSTRENGTHS:\nThe paper considers the problem of implicit bias for equivariant networks using gradient flow analysis, which was not previously studied in the literature. Although the results seem to be mostly based on s. techniques of Lyu and Li [2020] and Ji and Telgarsky [2020], the conclusions are interesting, in particular, the equivalence of data augmentation and model equivariance in terms of converged direction.\n\nWEAKNESSES:\n* I have some concerns about the choice of linear model for equivariant networks. When considering linear group invariant classifiers, the steerable network, based on Schur’s lemma, utilizes only the trivial representation (See M. Weiler and G. Cesa 2019). The authors noticed a similar issue with Lawrence et al. 2021 paper, as they mentioned “In their case, the only group-invariant linear classifier is the group average of the input up to a scalar multiplication, due to the transitivity of the regular representation”. To summarize, a linear invariant network with equivariant kernels works only with trivial representations. In other words, only a single basis in equation (4), the one corresponding to the trivial irrep, matters for the optimization and the output. It seems to me that the current definition in (4) can be misleading in this sense. Note that Proposition 3.2 indicates as well that the network is only dependent on a group averaged version of $\\mathbf{x}$. \n* It seems that the equation (7) does not define an equivariant map. If we assume a regular representation,  then the map is equivariant only if $\\mathbf{w}_1$ has a group circulant structure. With this additional required structure, Lemma B.1 in the supplementary material cannot be directly applied, because of entry repetition in $\\mathbf{w}_1$. Note that according to assumption 3.1, the regular representation and $G$-lifting map is assumed for the first later.\n* The independence of the generalization error from group size in Remark 6.5 should be considered with care. Note that there is an experimental evidence, although light, for the impact of group size on the generalization (see Sokolic et al). On the other hand, the group size can shrink the size of the space $\\mathbb{R}^{d_0}_{G}$, and can potentially reduce $\\overline{R}$ proportional to the group size. I feel that Remark 6.5 should be a bit more nuanced.\n\nQUESTIONS:\n* Does not the choice of regular representation for the input impose that $d_0$ is a function of group size? The steerable network of Cohen and Welling 2017 is more general and includes translation operations and features fields. \n* How does the result generalize to compact but not finite groups? For example, the data augmentation for compact inifinite groups seems to be non-trivial. Some discussions on this point can be useful for the paper.\n* There are some follow-up works on the generalization bounds for equivariant networks with implicit and explicit bounds on the error. It is good to discuss the connection and compare the bound. It seems that all these works relate the generalization gain to the group size, which is different  from the current paper.\n   * Sicheng Zhu, Bang An, and Furong Huang. Understanding the Generalization Benefit of Model Invariance from a Data Perspective. In Advances in Neural Information Processing Systems, 2021\n  * Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy. On the benefits of invariance in neural networks. arXiv preprint arXiv:2005.00178, 2020\n  * Arash Behboodi, Gabriele Cesa, Taco Cohen, A PAC-Bayesian Generalization Bound for Equivariant Networks, In Advances in Neural Information Processing Systems, 2022\n\nLIMITATIONS:\nSee above comments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

comments: SUMMARY:
This paper analyzes the implicit bias of gradient flow on linear group-equivariant steerable networks for group-invariant binary classification. It shows that the parameterized predictor converges in a direction that aligns with the unique group-invariant classifier with a maximum margin that is defined by the input representation. Under a unitary assumption, this paper also suggests the equivalence between data augmentation used in non-invariant models and learning with steerable networks. When the underlying distribution is group-invariant, linear steerable networks outperform their non-invariant counterparts in terms of improved margin and generalization bound.

STRENGTHS:
1. The paper is organized in a way easy to read and follow. The authors provide sufficient background knowledge for readers to understand Equivariant Steerable Networks. And all the assumptions are stated clearly in the problem setup section.
2. The contributions of the paper have been listed in a rather detailed and clear way in the paper. The authors point out their main findings in theorems, followed by explanations in remarks and close each section with corollaries. 
3. The findings presented in this paper explain the success of equivariant steerable networks to a certain extent.

WEAKNESSES:
1. The authors claim to fully characterize the implicit bias of the training algorithm. However, there is no experiment on empirical dataset other than theoretical results throughout the paper. It would be more promising if the conclusion can be supported by empirical results.

QUESTIONS:
Are these findings limited to binary classification?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-23