SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On the other hand, since  $\mathcal{P}_{\mathrm{steer}}(\mathbf{W}) = \beta \in \mathbb{R}^{d_0}$  always corresponds to a  $G$ -invariant linear predictor—we have slightly abused the notation by identifying  $\beta \in \mathbb{R}^{d_0}$  with the map  $\mathbf{x} \mapsto \langle \mathbf{x}, \beta \rangle$ —one can alternatively consider the empirical risk minimization directly over the invariant linear predictors  $\beta$ :
$$
\min  _ {\boldsymbol {\beta} \in \mathbb {R} _ {G} ^ {d _ {0}}} \mathcal {L} (\boldsymbol {\beta}; S) := \sum_ {i = 1} ^ {n} \ell_ {\exp} \left(\left\langle \mathbf {x} _ {i}, \boldsymbol {\beta} \right\rangle , y _ {i}\right), \tag {17}
$$
where  $\mathbb{R}_G^{d_0} \subset \mathbb{R}^{d_0}$  is the subspace of all  $G$ -invariant linear predictors, which is characterized by the following proposition.
Proposition 3.6. Let  $\mathbb{R}_G^{d_0} \subset \mathbb{R}^{d_0}$  be the subspace of  $G$ -invariant linear predictors, i.e.,  $\mathbb{R}_G^{d_0} = \{\beta \in \mathbb{R}^{d_0} : \beta^\top \mathbf{x} = \beta^\top g\mathbf{x}, \forall \mathbf{x} \in \mathbb{R}^{d_0}, \forall g \in G\}$ . Then
(a)  $\mathbb{R}_G^{d_0}$  is characterized by
$$
\mathbb {R} _ {G} ^ {d _ {0}} = \bigcap_ {g \in G} \ker (I - g ^ {\top}) = \operatorname {R a n g e} \left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top}\right). \tag {18}
$$
(b) Let  $\mathcal{A}:\mathbb{R}^{d_0}\to \mathbb{R}^{d_0}$  be the group-averaging map,
$$
\mathcal {A} (\beta) := \bar {\beta} = \frac {1}{| G |} \sum_ {g \in G} g \beta . \tag {19}
$$
Then its adjoint  $\mathcal{A}^{\top}:\beta \mapsto \frac{1}{|G|}\sum_{g\in G}g^{\top}\beta$  is a projection operator from  $\mathbb{R}^{d_0}$  to  $\mathbb{R}_G^{d_0}$ . In other words,  $\mathrm{Range}(\mathcal{A}^{\top}) = \mathbb{R}_{G}^{d_{0}}$  and  $\mathcal{A}^{\top}\circ \mathcal{A}^{\top} = \mathcal{A}^{\top}$ .
(c) If  $G$  acts unitarily on  $\mathcal{X}_0$ , i.e.,  $\rho_0(g^{-1}) = \rho_0(g)^\top$ , then  $\mathcal{A} = \mathcal{A}^\top$  is self-adjoint. This implies that  $\mathcal{A}:\beta \mapsto \overline{\beta}$  is an orthogonal projection from  $\mathbb{R}^{d_0}$  onto  $\mathbb{R}_G^{d_0}$ . In particular, we have
$$
\bar {\beta} = \beta \Longleftrightarrow \beta \in \mathbb {R} _ {G} ^ {d _ {0}}, \quad a n d \| \bar {\beta} \| \leq \| \beta \|, \forall \beta \in \mathbb {R} ^ {d _ {0}}. \tag {20}
$$
Proposition 3.6 combined with (10) demonstrates that a linear steerable network  $f_{\mathrm{steer}}(\cdot; \mathbf{W}) = \langle \cdot, \mathcal{P}_{\mathrm{steer}}(\mathbf{W}) \rangle$  can realize any  $G$ -invariant linear predictor  $\beta \in \mathbb{R}_G^{d_0}$ ; that is,  $\{\mathcal{P}_{\mathrm{steer}}(\mathbf{W}) : \mathbf{W} \in \mathcal{W}_{\mathrm{steer}}\} = \mathbb{R}_G^{d_0}$ . Therefore (16) and (17) are equivalent optimization problems parameterized in different ways. However, minimizing (16) using gradient-based methods may potentially lead to different classifiers compared to those obtained from optimizing (17) directly.
Gradient flow. Given an initialization  $\mathbf{W}(0) \in \mathcal{W}_{\mathrm{steer}}$ , the gradient flow  $\{\mathbf{W}(t)\}_{t \geq 0}$  for (16) is the solution of the following ordinary differential equation (ODE),
$$
\frac {\mathrm {d} \mathbf {W}}{\mathrm {d} t} = - \nabla_ {\mathbf {W}} \mathcal {L} _ {\mathcal {P} _ {\text {s t e e r}}} (\mathbf {W}; S) = - \nabla_ {\mathbf {W}} \left[ \sum_ {i = 1} ^ {n} \ell_ {\exp} \left(\left\langle \mathbf {x} _ {i}, \mathcal {P} _ {\text {s t e e r}} (\mathbf {W}) \right\rangle , y _ {i}\right) \right]. \tag {21}
$$
The purpose of this work is to inspect the asymptotic behavior of the  $G$ -invariant linear predictors  $\beta_{\mathrm{steer}}(t) = \mathcal{P}_{\mathrm{steer}}(\mathbf{W}(t))$  parameterized by the linear steerable network trained under gradient flow (21). In particular, we aim to analyze the directional limit of  $\beta_{\mathrm{steer}}(t)$  as  $t \to \infty$ , i.e.,  $\lim_{t \to \infty} \frac{\beta_{\mathrm{steer}}(t)}{\|\beta_{\mathrm{steer}}(t)\|}$ . Before ending this section, we make the following assumption on the gradient flow  $\mathbf{W}(t)$  that also appears in the prior works Ji and Telgarsky [2020], Lyu and Li [2020], Yun et al. [2021].
Assumption 3.7. The gradient flow  $\mathbf{W}(t)$  satisfies  $\mathcal{L}_{\mathcal{P}_{\mathrm{steer}}}\left(\mathbf{W}(t_0);S\right) < 1$  for some  $t_0 > 0$ .
This assumption implies that the data set  $S = \{(\mathbf{x}_i, y_i) : i \in [n]\}$  can be separated by a  $G$ -invariant linear predictor  $\mathcal{P}_{\mathrm{steer}}(\mathbf{W}(t_0))$ , and our analysis is focused on the "late phase" of the gradient flow training as  $t \to \infty$ .
4 Implicit bias of linear steerable networks
Our main result on the implicit bias of gradient flow on linear steerable networks in binary classification is summarized in the following theorem.
Theorem 4.1. Under Assumption 3.1 and Assumption 3.7, let  $\beta_{\mathrm{steer}}(t) = \mathcal{P}_{\mathrm{steer}}(\mathbf{W}(t))$  be the time-evolution of the  $G$ -invariant linear predictors parameterized by a linear steerable network trained with gradient flow on the data set  $S = \{(\mathbf{x}_i,y_i):i\in [n]\}$ ; cf. Eq. (21). Then
(a) The directional limit  $\beta_{\mathrm{steer}}^{\infty} = \lim_{t\to \infty}\frac{\beta_{\mathrm{steer}}(t)}{\|\beta_{\mathrm{steer}}(t)\|}$  exists and  $\beta_{\mathrm{steer}}^{\infty}\propto \frac{1}{|G|}\sum_{g\in G}g^{\top}\gamma^{*}$ , where  $\gamma^{*}$  is the max- $L^2$ -margin SVM solution for the transformed data  $\overline{S} = \{(\overline{\mathbf{x}}_i,y_i):i\in [n]\}$ :
$$
\boldsymbol {\gamma} ^ {*} = \arg \min  _ {\boldsymbol {\gamma} \in \mathbb {R} ^ {d _ {0}}} \| \boldsymbol {\gamma} \| ^ {2}, \quad \text {s . t .} y _ {i} \left\langle \overline {{\mathbf {x}}} _ {i}, \boldsymbol {\gamma} \right\rangle \geq 1, \forall i \in [ n ]. \tag {22}
$$
Furthermore, if  $G$  acts unitarily on the input space  $\mathcal{X}_0$ , i.e.,  $g^{-1} = g^{\top}$ , then  $\beta_{\mathrm{steer}}^{\infty} \propto \gamma^{*}$ .
(b) Equivalently,  $\beta_{\mathrm{steer}}^{\infty}$  is proportional to the unique minimizer  $\beta^{*}$  of the problem
$$
\boldsymbol {\beta} ^ {*} = \arg \min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {d _ {0}}} \| \operatorname {P r o j} _ {\text {R a n g e} (\mathcal {A})} \boldsymbol {\beta} \| ^ {2}, \quad \text {s . t .} \boldsymbol {\beta} \in \mathbb {R} _ {G} ^ {d _ {0}}, \text {a n d} y _ {i} \left\langle \mathbf {x} _ {i}, \boldsymbol {\beta} \right\rangle \geq 1, \forall i \in [ n ], \tag {23}
$$
where  $\operatorname{Proj}_{\operatorname{Range}(\mathcal{A})}$  is the projection from  $\mathbb{R}^{d_0}$  to  $\operatorname{Range}(\mathcal{A}) = \operatorname{Range}\left(\frac{1}{|G|}\sum_{g\in G}g\right)$ . Moreover, if  $G$  acts unitarily on  $\mathcal{X}_0$ , then
$$
\boldsymbol {\beta} _ {\text {s t e e r}} ^ {\infty} \propto \boldsymbol {\beta} ^ {*} = \arg \min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {d _ {0}}} \| \boldsymbol {\beta} \| ^ {2}, \quad \text {s . t .} \boldsymbol {\beta} \in \mathbb {R} _ {G} ^ {d _ {0}}, a n d y _ {i} \langle \mathbf {x} _ {i}, \boldsymbol {\beta} \rangle \geq 1, \forall i \in [ n ]. \tag {24}
$$
Namely,  $\beta_{\mathrm{steer}}^{\infty}$  achieves the maximum  $L^2$ -margin among all  $G$ -invariant linear predictors.
Theorem 4.1 suggests that gradient flow implicitly guides a linear steerable network toward the unique  $G$ -invariant classifier with a maximum margin defined by the input representation  $\rho_0$ .
Remark 4.2. According to Remark 3.4, when  $G = \{e\}$  is a trivial group, then a linear  $G$ -CNN (which is a special case of linear steerable networks) reduces to a fully-connected network  $f_{\mathrm{fc}}(\mathbf{x};\mathbf{W})$ . Since the representation of a trivial group is always unitary, we have the following corollary which also appeared in Ji and Telgarsky [2020] and Yun et al. [2021].
Corollary 4.3. Let  $\{\mathbf{W}(t)\}_{t\geq 0}\subset \mathcal{W}_{\mathrm{fc}}$  be the gradient flow of the parameters when training a linear fully-connected network on the data set  $S = \{(\mathbf{x}_i,y_i),i\in [n]\}$  , i.e.,
$$
\frac {\mathrm {d} \mathbf {W}}{\mathrm {d} t} = - \nabla_ {\mathbf {W}} \mathcal {L} _ {\mathcal {P} _ {\mathrm {f c}}} (\mathbf {W}; S) := - \nabla_ {\mathbf {W}} \left[ \sum_ {i = 1} ^ {n} \ell_ {\exp} \left(\left\langle \mathbf {x} _ {i}, \mathcal {P} _ {\mathrm {f c}} (\mathbf {W}) \right\rangle , y _ {i}\right) \right]. \tag {25}
$$
Then the classifier  $\beta_{\mathrm{fc}}(t) = \mathcal{P}_{\mathrm{fc}}(\mathbf{W}(t))$  converges in a direction that aligns with the max- $L^2$ -margin SVM solution  $\gamma^{*}$  for the original data set  $S = \{(\mathbf{x}_i, y_i) : i \in [n]\}$ ,
$$
\boldsymbol {\gamma} ^ {*} = \arg \min  _ {\boldsymbol {\gamma} \in \mathbb {R} ^ {d _ {0}}} \| \boldsymbol {\gamma} \| ^ {2}, \quad \text {s . t .} y _ {i} \left\langle \mathbf {x} _ {i}, \boldsymbol {\gamma} \right\rangle \geq 1, \forall i \in [ n ]. \tag {26}
$$
Remark 4.4. While Theorem 4.1 provides a complete characterization of the implicit bias exhibited by gradient flow in linear steerable networks, it is imperative to note that the convergence rate to the directional limit is, in fact, exponentially slow. This is consistent with the findings in, e.g., [Soudry et al., 2018, Yun et al., 2021]. A comprehensive analysis of gradient flow behavior in a non-asymptotic regime falls outside the scope of this current study.
5 The equivalence between steerable networks and data augmentation
Compared to hard-wiring symmetry priors into model architectures through equivariant steerable networks, an alternative approach to incorporate symmetry into the learning process is by training a non-equivariant model with the aid of data augmentation. In this section, we demonstrate that these two approaches are equivalent for binary classification under a unitary assumption for  $\rho_0$ .
Corollary 5.1. Let  $\beta_{\mathrm{steer}}^{\infty} = \lim_{t\to \infty}\frac{\beta_{\mathrm{steer}}(t)}{\|\beta_{\mathrm{steer}}(t)\|}$  be the directional limit of the linear predictor  $\beta_{\mathrm{steer}}(t) = \mathcal{P}_{\mathrm{steer}}(\mathbf{W}(t))$  parameterized by a linear steerable network trained using gradient flow on the original data set  $S = \{(\mathbf{x}_i,y_i),i\in [n]\}$ . Correspondingly, let  $\beta_{\mathrm{fc}}^{\infty} = \lim_{t\to \infty}\frac{\beta_{\mathrm{fc}}(t)}{\|\beta_{\mathrm{fc}}(t)\|}$ ,  $\beta_{\mathrm{fc}}(t) = \mathcal{P}_{\mathrm{fc}}(\mathbf{W}(t))$  (13), be that of a linear fully-connected network trained on the augmented data set  $S_{\mathrm{aug}} = \{(g\mathbf{x}_i,y_i),i\in [n],g\in G\}$ . If  $G$  acts unitarily on  $\mathcal{X}_0$ , then
$$
\beta_ {\text {s t e e r}} ^ {\infty} = \beta_ {\mathrm {f c}} ^ {\infty}. \tag {27}
$$
In other words, the effect of using a linear steerable network for group-invariant binary classification is exactly the same as conducting data-augmentation for non-invariant models.
Remark 5.2. The equivalence between data augmentation and training with a linear steerable network is valid only in an asymptotic sense, yet the underlying training dynamics differ substantially. Specifically,  $\beta_{\mathrm{steer}}(t)$  is assured to maintain  $G$ -invariance throughout the training process, whereas  $\beta_{\mathrm{fc}}(t)$  achieves  $G$ -invariance solely in the limiting case as  $t \to \infty$ . Moreover, the equivalence only holds for "full-batch" data-augmentation over the entire group orbits.
Remark 5.3. For Corollary 5.1 to hold, it is crucial that  $\rho_0\in \mathrm{GL}(\mathcal{X}_0)$  is unitary, as otherwise the limit direction  $\beta_{\mathrm{fc}}^{\infty}$  of a linear fully-connected network trained on the augmented data set is generally not  $G$  -invariant (and hence cannot be equal to  $\beta_{\mathrm{steer}}^{\infty}\in \mathbb{R}_{G}^{d_{0}}$ ); see Example 5.4.
Example 5.4. Let  $G = \mathbb{Z}_2 = \{\overline{0},\overline{1}\}$ . Consider a (non-unitary)  $G$ -representation  $\rho_0$  on  $\mathcal{X}_0 = \mathbb{R}^2$ ,
$$
\rho_ {0} (\overline {{0}}) = \left[ \begin{array}{l l} 1 & 0 \\ 0 & 1 \end{array} \right], \quad \rho_ {0} (\overline {{1}}) = \left[ \begin{array}{l l} 1 & 0 \\ - 1 & 1 \end{array} \right] \left[ \begin{array}{l l} - 1 & 0 \\ 0 & 1 \end{array} \right] \left[ \begin{array}{l l} 1 & 0 \\ 1 & 1 \end{array} \right] = \left[ \begin{array}{l l} - 1 & 0 \\ 2 & 1 \end{array} \right]. \tag {28}
$$
Let  $S = \{(\mathbf{x},y)\} = \{((1,2)^\top , + 1)\}$  be a training set with only one point. By Theorem 4.1, the limit direction  $\beta_{\mathrm{steer}}^{\infty}$  of a linear-steerable-network-parameterized linear predictor satisfies
$$
\beta_ {\text {s t e e r}} ^ {\infty} \propto \frac {1}{2} \sum_ {g \in G} g ^ {\top} \boldsymbol {\gamma} ^ {*} = \left[ \begin{array}{l l} 0 & 1 \\ 0 & 1 \end{array} \right] \boldsymbol {\gamma} ^ {*} = \left[ \begin{array}{l l} 0 & 1 \\ 0 & 1 \end{array} \right] \left[ \begin{array}{l} 0 \\ \frac {1}{3} \end{array} \right] = \left[ \begin{array}{l} \frac {1}{3} \\ \frac {1}{3} \end{array} \right], \tag {29}
$$
where  $\gamma^{*} = (0,\frac{1}{3})^{\top}$  is the max-margin SVM solution for the transformed data set  $\overline{S} = \{(\overline{\mathbf{x}},y)\} = \{((0,3)^{\top}, + 1)\}$ . On the contrary, by Corollary 4.3, the limit direction  $\beta_{\mathrm{fc}}^{\infty}$  of a linear fully-connected network trained on the augmented data set  $S_{\mathrm{aug}} = \{((1,2)^{\top}, + 1),((-1,4)^{\top}, + 1)\}$  aligns with the max-margin SVM solution  $\gamma_{\mathrm{aug}}^{*}$  for  $S_{\mathrm{aug}}$ ,
$$
\begin{array}{l} \beta_ {\mathrm {f c}} ^ {\infty} \propto \gamma_ {\mathrm {a u g}} ^ {*} = \arg \min  _ {\gamma \in \mathbb {R} ^ {2}} \| \gamma \| ^ {2}, \quad \text {s . t .} y \langle \mathbf {x}, \gamma \rangle \geq 1, \forall (\mathbf {x}, y) \in S _ {\mathrm {a u g}} (30) \\ = (0. 2, 0. 4) ^ {\top} \not \propto \beta_ {\text {s t e e r}} ^ {\infty}. (31) \\ \end{array}
$$
However, we demonstrate below that the equivalence between linear steerable networks and data augmentation can be re-established for non-unitary  $\rho_0$  by defining a new inner product on  $\mathbb{R}^{d_0}$ ,
$$
\left\langle \mathbf {v}, \mathbf {w} \right\rangle_ {\rho_ {0}} := \mathbf {v} ^ {\top} \left(\frac {1}{| G |} \sum_ {g \in G} \rho_ {0} (g) ^ {\top} \rho_ {0} (g)\right) ^ {1 / 2} \mathbf {w}, \quad \forall \mathbf {v}, \mathbf {w} \in \mathcal {X} _ {0} = \mathbb {R} ^ {d _ {0}}. \tag {32}
$$
When  $\rho_0$  is unitary,  $\langle \mathbf{v},\mathbf{w}\rangle_{\rho_0} = \langle \mathbf{v},\mathbf{w}\rangle$  is the normal Euclidean inner product. With this new inner product, we modify the linear fully-connected network and its empirical loss on the augmented data set  $S_{\mathrm{aug}} = \{(g\mathbf{x}_i,y_i):g\in G,i\in [n]\}$  as
$$
f _ {\mathrm {f c}} ^ {\rho_ {0}} (\mathbf {x}; \mathbf {W}) := \left\langle \mathbf {x}, \mathcal {P} _ {\mathrm {f c}} (\mathbf {W}) \right\rangle_ {\rho_ {0}}, \tag {33}
$$
$$
\mathcal {L} _ {\mathcal {P} _ {\mathrm {f c}}} ^ {\rho_ {0}} (\mathbf {W}; S _ {\mathrm {a u g}}) := \sum_ {g \in G} \sum_ {i = 1} ^ {n} \ell_ {\exp} \left(f _ {\mathrm {f c}} ^ {\rho_ {0}} \left(g \mathbf {x} _ {i}; \mathbf {W}\right), y _ {i}\right) = \sum_ {g \in G} \sum_ {i = 1} ^ {n} \ell_ {\exp} \left(\langle g \mathbf {x} _ {i}, \mathcal {P} _ {\mathrm {f c}} (\mathbf {W}) \rangle_ {\rho_ {0}}, y _ {i}\right). \tag {34}
$$
The following corollary shows that, for non-unitary  $\rho_0$  on  $\mathcal{X}_0$ , the implicit bias of a linear steerable network is again the same as that of a modified linear fully-connected network  $f_{\mathrm{fc}}^{\rho_0}(\mathbf{x};\mathbf{W})$  trained under data augmentation.
Corollary 5.5. Let  $\beta_{\mathrm{steer}}^{\infty}$  be the same as that in Corollary 5.1. Let  $\beta_{\mathrm{fc}}^{\rho_0,\infty} = \lim_{t\to \infty}\frac{\beta_{\mathrm{fc}}^{\rho_0}(t)}{\|\beta_{\mathrm{fc}}^{\rho_0}(t)\|}$  be the limit direction of  $\beta_{\mathrm{fc}}^{\rho_0}(t) = \mathcal{P}_{\mathrm{fc}}(\mathbf{W}(t))$  under the gradient flow of the modified empirical loss  $\mathcal{L}_{\mathcal{P}_{\mathrm{fc}}}^{\rho_0}(\mathbf{W};S_{\mathrm{aug}})$  (34) for a linear fully-connected network on the augmented data set  $S_{\mathrm{aug}}$ . Then
$$
\beta_ {\text {s t e e r}} ^ {\infty} \propto \left(\frac {1}{| G |} \sum_ {g \in G} \rho_ {0} (g) ^ {\top} \rho_ {0} (g)\right) ^ {1 / 2} \beta_ {\mathrm {f c}} ^ {\rho_ {0}, \infty}. \tag {35}
$$
Consequently, we have  $\langle \mathbf{x},\beta_{\mathrm{steer}}^{\infty}\rangle \propto \langle \mathbf{x},\beta_{\mathrm{fc}}^{\rho_0,\infty}\rangle_{\rho_0}$  for all  $\mathbf{x}\in \mathbb{R}^{d_0}$ .
6 Improved margin and generalization
We demonstrate in this section the improved margin and generalization of linear steerable networks over their non-invariant counterparts. In what follows, we assume  $\rho_0$  to be unitary.
The following theorem shows that the margin of a linear-steerable-network-parameterized predictor  $\beta_{\mathrm{steer}}^{\infty}$  on the augmented data set  $S_{\mathrm{aug}}$  is always larger than that of a linear fully-connected network  $\beta_{\mathrm{fc}}^{\infty}$ , suggesting improved  $L^2$ -robustness of the steerable-network-parameterized classifier.
Theorem 6.1. Let  $\beta_{\mathrm{steer}}^{\infty}$  be the directional limit of a linear-steerable-network-parameterized predictor trained on the original data set  $S = \{(\mathbf{x}_i, y_i), i \in [n]\}$ ; let  $\beta_{\mathrm{fc}}^{\infty}$  be that of a linear fully-connected network also trained on the same data set  $S$ . Let  $M_{\mathrm{steer}}$  and  $M_{\mathrm{fc}}$ , respectively, be the (signed) margin of  $\beta_{\mathrm{steer}}^{\infty}$  and  $\beta_{\mathrm{fc}}^{\infty}$  on the augmented data set  $S_{\mathrm{aug}} = \{(g\mathbf{x}_i, y_i) : i \in [n], g \in G\}$ , i.e.,
$$
M _ {\text {s t e e r}} := \min  _ {i \in [ n ], g \in G} y _ {i} \left\langle \boldsymbol {\beta} _ {\text {s t e e r}} ^ {\infty}, g \mathbf {x} _ {i} \right\rangle , \quad M _ {\mathrm {f c}} := \min  _ {i \in [ n ], g \in G} y _ {i} \left\langle \boldsymbol {\beta} _ {\mathrm {f c}} ^ {\infty}, g \mathbf {x} _ {i} \right\rangle . \tag {36}
$$
Then we always have  $M_{\mathrm{steer}} \geq M_{\mathrm{fc}}$ .
Finally, we aim to quantify the improved generalization of linear steerable networks compared to fully-connected networks in binary classification of linearly separable group-invariant distributions defined below.
Definition 6.2. A distribution  $\mathcal{D}$  on  $\mathbb{R}^{d_0} \times \{\pm 1\}$  is called linearly separable if there exists  $\beta \in \mathbb{R}^{d_0}$  such that
$$
\mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ y \langle \mathbf {x}, \boldsymbol {\beta} \rangle \geq 1 ] = 1. \tag {37}
$$
It is easy to verify (by Lemma F.1) that if  $\mathcal{D}$  is  $G$ -invariant and linearly separable, then  $\mathcal{D}$  can be separated by a  $G$ -invariant linear classifier. The following theorem establishes the generalization bound of linear steerable networks in separable group-invariant binary classification.
Theorem 6.3. Let  $\mathcal{D}$  be a  $G$ -invariant distribution over  $\mathbb{R}^{d_0} \times \{\pm 1\}$  that is linearly separable by an invariant classifier  $\beta_0 \in \mathbb{R}_G^{d_0}$ . Define
$$
\bar {R} = \inf  \left\{r > 0: \| \bar {\mathbf {x}} \| \leq r \text {w i t h p r o b a b i l i t y} 1 \right\}. \tag {38}
$$
Let  $S = \{(\mathbf{x}_i, y_i)\}_{i=1}^n$  be i.i.d. samples from  $\mathcal{D}$ , and let  $\beta_{\mathrm{steer}}^\infty$  be the limit direction of a steerable-network-parameterized linear predictor trained using gradient flow on  $S$ . Then, for any  $\delta > 0$ , we have with probability at least  $1 - \delta$  (over random samples  $S \sim \mathcal{D}^n$ ) that
$$
\mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ y \neq \operatorname {s i g n} (\langle \mathbf {x}, \boldsymbol {\beta} _ {\text {s t e e r}} ^ {\infty} \rangle) ] \leq \frac {2 \bar {R} \| \boldsymbol {\beta} _ {0} \|}{\sqrt {n}} + \sqrt {\frac {\log (1 / \delta)}{2 n}}. \tag {39}
$$
Remark 6.4. In comparison, let  $\beta_{\mathrm{fc}}^{\infty}$  be the limit direction of a fully-connected-network-parameterized linear predictor trained on  $S$ . Then with probability at least  $1 - \delta$ , we have
$$
\mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ y \neq \operatorname {s i g n} (\langle \mathbf {x}, \beta_ {\mathrm {f c}} ^ {\infty} \rangle) ] \leq \frac {2 R \| \beta_ {0} \|}{\sqrt {n}} + \sqrt {\frac {\log (1 / \delta)}{2 n}}, \tag {40}
$$
where  $R = \inf \left\{ r > 0 : \| \mathbf{x} \| \leq r \text{ with probability } 1 \right\}$ . This is the classical generalization result for max-margin SVM (see, e.g., Shalev-Shwartz and Ben-David [2014].) Eq. (40) can also be viewed as a special case of Eq. (39), as a fully-connected network is a  $G$ -CNN with  $G = \{e\}$  (cf. Remark 4.2), and therefore  $\overline{\mathbf{x}} = \mathbf{x}$  and  $R = \overline{R}$ .
By Proposition 3.6, the map  $\mathbf{x} \to \overline{\mathbf{x}}$  is an orthogonal projection, and thus we always have  $\overline{R} \leq R$ . Therefore the generalization bound for steerable networks in (39) is always smaller than that of the fully-connected network in (40).
Remark 6.5. A comparison between Eq. (39) and Eq. (40) reveals that the improved generalization of linear steerable network does not necessarily depend on the group size  $|G|$ . Instead, it depends on how far the distribution  $\mathcal{D}$ 's support is from the subspace  $\mathbb{R}_G^{d_0}$ , such that  $\overline{R}$  could be much smaller than  $R$ . In fact, if the support of  $\mathcal{D}$  is contained in  $R_G^{d_0}$ , then  $\overline{R} = R$ , and the steerable network does not achieve any generalization gain. This is consistent with Theorem 4.1, as in this case, the transformed data set  $\bar{S} = \{(\overline{\mathbf{x}}_i, y_i) : i \in [n]\}$  is the same as the original data set  $S$ .
In this work, we analyzed the implicit bias of gradient flow on general linear group-equivariant steerable networks in group-invariant binary classification. Our findings indicate that the parameterized predictor converges in a direction that aligns with the unique group-invariant classifier with a maximum margin that is dependent on the input representation. As a corollary of our main result,
we established the equivalence between data augmentation and learning with steerable networks in our setting. Finally, we demonstrated that linear steerable networks outperform their non-invariant counterparts in terms of improved margin and generalization bound.
A limitation of our result is that the implicit bias of gradient flow studied herein holds in an asymptotic sense, and the convergence rate to the directional limit might be extremely slow. This is consistent with the findings in, e.g., [Soudry et al., 2018, Yun et al., 2021]. Understanding the behavior of gradient flow in a non-asymptotic regime is an important direction for future work. Furthermore, in our current framework, we assume that the first-layer equivariant map is represented by the  $G$ -lifting map. This assumption ensures that the linear steerable network possesses sufficient capacity to parameterize all  $G$ -invariant linear classifiers. Exploring the implicit bias of steerable networks without this assumption would be a compelling next step. The removal of this constraint could facilitate the generalization of our findings from finite groups to compact groups.
The research was partially supported by NSF under DMS-2052525, DMS-2140982, and DMS-2244976.