SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ziyu Chen
Department of Mathematics and Statistics  
University of Massachusetts Amherst
ziyuchen@umass.edu
Wei Zhu
weizhu@umass.edu
We study the implicit bias of gradient flow on linear equivariant steerable networks in group-invariant binary classification. Our findings reveal that the parameterized predictor converges in direction to the unique group-invariant classifier with a maximum margin defined by the input group action. Under a unitary assumption on the input representation, we establish the equivalence between steerable networks and data augmentation. Furthermore, we demonstrate the improved margin and generalization bound of steerable networks over their non-invariant counterparts.
Despite recent theoretical breakthroughs in deep learning, it is still largely unknown why overparameterized deep neural networks (DNNs) with infinitely many solutions achieving near-zero training error can effectively generalize on new data. However, the consistently impressive results of DNNs trained with first-order optimization methods, e.g., gradient descent (GD), suggest that the training algorithm is implicitly guiding the model towards a solution with strong generalization performance.
Indeed, recent studies have shown that gradient-based training methods effectively regularize the solution by implicitly minimizing a certain complexity measure of the model [Vardi, 2022]. For example, Gunasekar et al. [2018b] showed that in separable binary classification, the linear predictor parameterized by a linear fully-connected network trained under GD converges in the direction of a max-margin support vector machine (SVM), while linear convolutional networks are implicitly regularized by a depth-related bridge penalty in the Fourier domain. Yun et al. [2021] extended this finding to linear tensor networks. Lyu and Li [2020] and Ji and Telgarsky [2020] established the implicit max-margin regularization for (nonlinear) homogeneous DNNs in the parameter space.
On the other hand, another line of research aims to explicitly regularize DNNs through architectural design to exploit the inherent structure of the learning problem. In recent years, there has been a growing interest in leveraging group symmetry for this purpose, given its prevalence in both scientific and engineering domains. A significant body of literature has been devoted to designing group-equivariant DNNs that ensure outputs transform covariantly to input symmetry transformations. Group-equivariant steerable networks represent a general class of symmetry-preserving models that achieve equivariance with respect to any pair of input-output group actions [Cohen et al., 2019, Weiler and Cesa, 2019, Cohen and Welling, 2017]. Empirical evidence suggests that equivariant steerable networks yield substantial improvements in generalization performance for learning tasks with group symmetry, especially when working with limited amounts of data.
There have been several recent attempts to account for the empirical success of equivariant steerable networks through establishing a tighter upper bound on the test risk for these models. This is typically accomplished by evaluating the complexity measures of equivariant and non-equivariant models under the same norm constraint on the network parameters [Sokolic et al., 2017, Sannai
et al., 2021, Elesedy, 2022]. Nevertheless, it remains unclear whether or why a GD-trained steerable network can achieve a minimizer with a parameter norm comparable to that of its non-equivariant counterpart. Consequently, the effectiveness of such complexity-measure-based arguments to explain the generalization enhancement of steerable networks in group symmetric learning tasks may not be directly applicable.
In light of the above issues, in this work, we aim to fully characterize the implicit bias of the training algorithm on linear equivariant steerable networks in group-invariant binary classification. Our result shows that when trained under gradient flow (GF), i.e., GD with an infinitesimal step size, the steerable-network-parameterized predictor converges in direction to the unique group-invariant classifier attaining a maximum margin with respect to a norm defined by the input group representation. This result has three important implications: under a unitary input group action,
- a linear steerable network trained on the original data set converge in the same direction as a linear fully-connected network trained on the group-augmented data set. This suggests the equivalence between training with linear steerable networks and data augmentation;
- when trained on the same original data set, a linear steerable network always attains a wider margin on the group-augmented data set compared to a fully-connected network;
- when the underlying distribution is group-invariant, a GF-trained linear steerable network achieves a tighter generalization bound compared to its non-equivariant counterpart. This improvement in generalization is not necessarily dependent on the group size, but rather it depends on the support of the invariant distribution.
Before we end this section, we note that a similar topic has recently been explored by Lawrence et al. [2021] in the context of linear Group Convolutional Neural Networks (G-CNNs), a special case of the equivariant steerable networks considered in this work. However, we point out that the models they studied were not truly group-invariant, and thus their implicit bias result does not explain the improved generalization of G-CNNs. We will further elaborate on the comparison between our work and [Lawrence et al., 2021] in Section 2.
Implicit biases: Recent studies have shown that for linear regression with the logistic or exponential loss on linearly separable data, the linear predictor under GD/SGD converges in direction to the max- $L^2$ -margin SVM [Soudry et al., 2018, Nacson et al., 2019, Gunasekar et al., 2018a]. These results are extended to linear fully-connected networks and linear Convolutional Neural Networks (CNNs) by Gunasekar et al. [2018b] under the assumption of directional convergence and alignment of the network parameters, which are later proved by Ji and Telgarsky [2019a,b], Lyu and Li [2020], Ji and Telgarsky [2020]. The implicit regularization of gradient flow (GF) is further generalized to linear tensor networks by Yun et al. [2021]. For overparameterized nonlinear networks in the infinite-width regime, rigorous analysis on the optimization of DNNs has also been studied from the neural tangent kernel [Jacot et al., 2018, Du et al., 2019, Allen-Zhu et al., 2019] and the mean-field perspectives [Mei et al., 2019, Chizat and Bach, 2018]. However, these models are not explicitly designed to be group-invariant/equivariant, and the implicit bias of gradient-based methods might guide them to converge to sub-optimal solutions in learning tasks with intrinsic group symmetry.
Equivariant neural networks. Since their introduction by Cohen and Welling [2016], group equivariant network design has become a burgeoning field with numerous applications from computer vision to scientific computing. Unlike the implicit bias induced by training algorithms, equivariant networks explicitly incorporate symmetry priors into model design through either group convolutions [Cheng et al., 2019, Weiler et al., 2018, Sosnovik et al., 2020, Zhu et al., 2022] or, more generally, steerable convolutions [Weiler and Cesa, 2019, Cohen et al., 2019, Worrall et al., 2017]. Despite their empirical success, it remains largely unknown why and whether equivariant networks trained under gradient-based methods actually converge to solutions with smaller test risk in group-symmetric learning tasks [Sokolic et al., 2017, Sannai et al., 2021]. In a recent study, Elesedy and Zaidi [2021] demonstrated a provably strict generalisation benefit for equivariant networks in linear regression. However, the network studied therein is non-equivariant and only symmetrized after training; such test-time data augmentation is different from practical usage of equivariant networks.
Comparison to Lawrence et al. [2021]. A recent study by Lawrence et al. [2021] also analyzes the implicit bias of linear equivariant G-CNNs, which are a special case of the steerable networks considered in this work. However, the networks studied therein are not truly equivariant/invariant. More specifically, the input space  $\mathcal{X}_0$  they considered is the set of functions on the group  $G$ , i.e.,  $\mathcal{X}_0 = \{f: G \to \mathbb{R}\}$ , and the input  $G$ -action is given by the regular representation
$$
\left[ \rho_ {0} (g) f \right] \left(g ^ {\prime}\right) = f \left(g ^ {- 1} g ^ {\prime}\right), \quad \forall g \in G, \forall f \in \mathcal {X} _ {0}. \tag {1}
$$
In their case, due to the transitivity of the regular representation (1),  $G$ -invariant linear functions  $\Phi : \mathcal{X}_0 \to \mathbb{R}$  are constrained to the (trivial) form:
$$
\Phi (f) = C \sum_ {g \in G} f (g),
$$
where  $C$  is a multiplicative constant. To avoid learning this trivial function, Lawrence et al. [2021] chose to parameterize  $\Phi$  using a linear G-CNN, in which the final layer is replaced by a fully-connected layer. While this fully-connected layer provides the capability to learn more complex and nontrivial functions, it simultaneously undermines the property of group invariance. Therefore, their implicit bias result does not explain the improved generalization of G-CNNs. In contrast, we assume Euclidean inputs with non-transitive group actions, allowing linear steerable networks to learn non-trivial group-invariant models.
3 Background and problem setup
We provide a brief background in group theory and group-equivariant steerable networks. We also explain the setup of our learning problem for group-invariant binary classification.
3.1 Group and group equivariance
A group is a set  $G$  equipped with a binary operator, the group product, satisfying the axioms of associativity, identity, and invertibility. We always assume in this work that  $G$  is finite, i.e.,  $|G| < \infty$ , where  $|G|$  denotes its cardinality.
Given a vector space  $\mathcal{X}$ , let  $\mathrm{GL}(\mathcal{X})$  be the general linear group of  $\mathcal{X}$  consisting of all invertible linear transformations on  $\mathcal{X}$ . A map  $\rho: G \to \mathrm{GL}(\mathcal{X})$  is called a group representation (or linear group action) of  $G$  on  $\mathcal{X}$  if  $\rho$  is a group homomorphism from  $G$  to  $\mathrm{GL}(\mathcal{X})$ , namely
$$
\rho (g h) = \rho (g) \rho (h) \in \operatorname {G L} (\mathcal {X}), \quad \forall g, h \in G. \tag {2}
$$
When the representation  $\rho$  is clear from the context, we also abbreviate the group action  $\rho(g)\mathbf{x}$  as  $g\mathbf{x}$ .
Given a pair of vector spaces  $\mathcal{X},\mathcal{Y}$  and their respective  $G$  -representations  $\rho_{\mathcal{X}}$  and  $\rho_{\mathcal{Y}}$  , a linear map  $\Psi :\mathcal{X}\to \mathcal{Y}$  is said to be  $G$  -equivariant if it commutes with the  $G$  -representations  $\rho_{\mathcal{X}}$  and  $\rho_{\mathcal{Y}}$  , i.e.,
$$
\Psi \circ \rho_ {\mathcal {X}} (g) = \rho_ {\mathcal {Y}} (g) \circ \Psi , \quad \forall g \in G. \tag {3}
$$
Linear equivariant maps are also called *intertwiners*, and we denote by  $\mathrm{Hom}_G(\rho_{\mathcal{X}}, \rho_{\mathcal{Y}})$  the space of all *intertwiners* satisfying (3). When  $\rho_{\mathcal{Y}} \equiv \mathrm{Id}$  is the trivial representation, then  $\Psi[\rho_{\mathcal{X}}(g)(\mathbf{x})] = \Psi[\mathbf{x}]$  for all  $\mathbf{x} \in \mathcal{X}$ ; namely,  $\Psi$  becomes a  $G$ -invariant linear map.
3.2 Equivariant steerable networks and G-CNNs
Let  $\mathcal{X}_0 = \mathbb{R}^{d_0}$  be a  $d_0$ -dimensional input Euclidean space, equipped with the usual inner product. Let  $\rho_0: G \to \mathrm{GL}(\mathcal{X}_0)$  be a  $G$ -representation on  $\mathcal{X}_0$ . Suppose we have an unknown target function  $f^*: \mathcal{X}_0 = \mathbb{R}^{d_0} \to \mathbb{R}$  that is  $G$ -invariant under  $\rho_0$ , i.e.,  $f^*(g\mathbf{x}) \coloneqq f^*(\rho_0(g)\mathbf{x}) = f^*(\mathbf{x})$  for all  $g \in G$  and  $\mathbf{x} \in \mathcal{X}_0$ . The goal of equivariant steerable networks is to approximate  $f^*$  using an  $L$ -layer neural network  $f = \Psi_L \circ \Psi_{L-1} \circ \dots \circ \Psi_1$  that is guaranteed to be also  $G$ -invariant.
Since the composition of equivariant maps is also equivariant, it suffices to specify a collection of  $G$ -representation spaces  $\{(\mathcal{X}_l,\rho_l)\}_{l = 1}^L$ , with  $(\mathcal{X}_L,\rho_L) = (\mathbb{R},\mathrm{Id})$  being the trivial representation, such that each layer  $\Psi_{l}\in \operatorname{Hom}_{G}(\rho_{l - 1},\rho_{L}):\mathcal{X}_{l - 1}\to \mathcal{X}_{l}$  is  $G$ -equivariant. Equivalently, we want the
following diagram to be commutative,
$$
\begin{array}{l} \mathcal {X} _ {0} \xrightarrow {\Psi_ {1}} \mathcal {X} _ {1} \xrightarrow {\Psi_ {2}} \mathcal {X} _ {2} \xrightarrow {\Psi_ {3}} \dots \xrightarrow {\Psi_ {L - 1}} \mathcal {X} _ {L - 1} \xrightarrow {\Psi_ {L}} \mathcal {X} _ {L} \\ \rho_ {0} (g) \Bigg \downarrow \quad \rho_ {1} (g) \Bigg \downarrow \quad \rho_ {2} (g) \Bigg \downarrow \quad \rho_ {L - 1} (g) \Bigg \downarrow \quad \rho_ {L} (g) \Bigg \downarrow , \quad \forall g \in G. \tag {4} \\ \mathcal {X} _ {0} \xrightarrow {\Psi_ {1}} \mathcal {X} _ {1} \xrightarrow {\Psi_ {2}} \mathcal {X} _ {2} \xrightarrow {\Psi_ {3}} \dots \xrightarrow {\Psi_ {L - 1}} \mathcal {X} _ {L - 1} \xrightarrow {\Psi_ {L}} \mathcal {X} _ {L} \\ \end{array}
$$
Equivariant steerable networks. Given the representations  $\{(\mathcal{X}_l,\rho_l)\}_{l = 0}^L$ , a (linear) steerable network is constructed as follows. For each  $l\in [L]\coloneqq \{1,\dots ,L\}$ , we choose a finite collection of  $N_{l}$  (pre-computed) intertwiners  $\{\psi_l^j\}_{j = 1}^{N_l}\subset \mathrm{Hom}_G(\rho_{l - 1},\rho_l)$ . Typically,  $\{\psi_l^j\}_{j = 1}^{N_l}$  is a basis of  $\mathrm{Hom}_G(\rho_{l - 1},\rho_l)$ , but it is not necessary in our setting. The  $l$ -th layer equivariant map  $\Psi_l^{\mathrm{steer}}:\mathcal{X}_{l - 1}\to$ $\mathcal{X}_l$  of a steerable network is then parameterized by
$$
\Psi_ {l} ^ {\text {s t e e r}} (\mathbf {x}; \mathbf {w} _ {l}) = \sum_ {j \in [ N _ {l} ]} w _ {l} ^ {j} \psi_ {l} ^ {j} (\mathbf {x}), \quad \forall \mathbf {x} \in \mathcal {X} _ {l - 1}, \tag {5}
$$
where the coefficients  $\mathbf{w}_l = [w_l^j]_{j\in [N_l]}^\top \in \mathbb{R}^{N_l}$  are the trainable parameters of the  $l$ -th layer. An  $L$ -layer linear steerable network  $f_{\mathrm{steer}}(\mathbf{x};\mathbf{W})$  is then defined as the composition
$$
f _ {\text {s t e e r}} (\mathbf {x}; \mathbf {W}) = \Psi_ {L} ^ {\text {s t e e r}} \left(\dots \Psi_ {2} ^ {\text {s t e e r}} \left(\Psi_ {1} ^ {\text {s t e e r}} (\mathbf {x}; \mathbf {w} _ {1}); \mathbf {w} _ {2}\right) \dots ; \mathbf {w} _ {L}\right), \tag {6}
$$
where  $\mathbf{W} = [\mathbf{w}_l]_{l=1}^L \in \prod_{l=1}^L \mathbb{R}^{N_l} \eqqcolon \mathcal{W}_{\mathrm{steer}}$  is the collection of all trainable parameters. The network  $f_{\mathrm{steer}}(\mathbf{x}; \mathbf{W})$  defined in (6) is referred to as a steerable network because it steers the layer-wise output to transform according to any specified representation.
G-CNNs. A special case of the steerable networks is the group convolutional neural network (G-CNN), wherein the hidden representation space  $(\mathcal{X}_l,\rho_l)$  for each  $l\in [L - 1]$  is set to
$$
\mathcal {X} _ {l} = \left(\mathbb {R} ^ {d _ {l}}\right) ^ {G} = \left\{\mathbf {x} _ {l}: G \rightarrow \mathbb {R} ^ {d _ {l}} \right\}, \quad \rho_ {l} (g) \mathbf {x} _ {l} (h) := \mathbf {x} _ {l} \left(g ^ {- 1} h\right) \in \mathbb {R} ^ {d _ {1}}, \forall g, h \in G. \tag {7}
$$
The representation  $\rho_{l}\in \mathrm{GL}(\mathcal{X}_{l})$  in (7) is known as the regular representation of  $G$ . Intuitively,  $\mathbf{x}_l\in \mathcal{X}_l$  can be viewed as a matrix of size  $d_{l}\times |G|$ , and  $\rho_l(g)$  is a permutation of the columns of  $\mathbf{x}_l$ .
With this choice of  $\{(\mathcal{X}_l,\rho_l)\}_{l = 0}^L$ , the first-layer equivariant map of a G-CNN is defined as
$$
\Psi_ {1} ^ {\mathrm {G - C N N}} (\mathbf {x}; \mathbf {w} _ {1}) (g) = \mathbf {w} _ {1} ^ {\top} g ^ {- 1} \mathbf {x} \in \mathbb {R} ^ {d _ {1}}, \quad \forall \mathbf {x} \in \mathbb {R} ^ {d _ {0}}, \tag {8}
$$
where  $\mathbf{w}_1 = (w_1^{j,k})_{j,k} \in \mathbb{R}^{d_0 \times d_1}$  are the trainable parameters of the first layer. Eq. (8) is called a  $G$ -lifting map as it lifts a Euclidean signal  $\mathbf{x} \in \mathbb{R}^{d_0}$  to a function  $\Psi_1^{\mathrm{G - CNN}}(\mathbf{x}; \mathbf{w}_1)$  on  $G$ . For the subsequent layers, equivariance is achieved through group convolutions, and the readers are referred to the appendix for a detailed explanation.
Assumptions on the representations. The input representation  $\rho_0\in \mathrm{GL}(\mathcal{X}_0 = \mathbb{R}^{d_0})$  is assumed to be given, and  $\rho_{L}\equiv 1\in \mathrm{GL}(\mathcal{X}_{L} = \mathbb{R})$  is set to the trivial representation for group-invariant outputs. In this work, we make the following special choices for the first-layer representation  $(\rho_{1},\mathcal{X}_{1})$  as well as the equivariant map  $\Psi_1^{\mathrm{steer}}(\mathbf{x},\mathbf{w}_1)$  in a general steerable network.
Assumption 3.1. We adopt the regular representation (7) for the first layer, and set the first-layer equivariant map  $\Psi_1^{\mathrm{steer}}(\mathbf{x};\mathbf{w}_1)$  to the  $G$  -lifting map (8).
Remark 3.2. The rationale of Assumption 3.1 is for the network to have enough capacity to parameterize any  $G$ -invariant linear classifier; this will be further explained in Proposition 3.6. We note that the representations  $(\rho_l, \mathcal{X}_l)$  and steerable maps  $\Psi_l^{\mathrm{steer}}(\mathbf{x}, \mathbf{w}_l)$  for all subsequent layers,  $l \in \{2, \dots, L-1\}$ , can be arbitrary.
Under Assumption 3.1, we can characterize the linear steerable networks  $f_{\mathrm{steer}}(\cdot ;\mathbf{W})$  in the following proposition.
Proposition 3.3. Let  $f_{\mathrm{steer}}(\mathbf{x}; \mathbf{W})$  be the linear steerable network satisfying Assumption 3.1, where  $\mathbf{W} = [\mathbf{w}_l]_{l=1}^L \in \mathcal{W}_{\mathrm{steer}}$  is the collection of all model parameters. There exists a multi-linear map  $M: (\mathbf{w}_2, \dots, \mathbf{w}_L) \mapsto M(\mathbf{w}_2, \dots, \mathbf{w}_L) \in \mathbb{R}^{d_1}$  such that for all  $\mathbf{x} \in \mathbb{R}^{d_0}$  and  $\mathbf{W} \in \mathcal{W}_{\mathrm{steer}}$ ,
$$
f _ {\text {s t e e r}} (\mathbf {x}; \mathbf {W}) = f _ {\text {s t e e r}} (\overline {{\mathbf {x}}}; \mathbf {W}) = \left\langle \overline {{\mathbf {x}}}, \mathbf {w} _ {1} M \left(\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}\right) \right\rangle , \tag {9}
$$
where  $\overline{\mathbf{x}} := \frac{1}{|G|} \sum_{g \in G} g\mathbf{x}$  is the average of all elements on the group orbit of  $\mathbf{x} \in \mathbb{R}^{d_0}$ .
Although a straightforward proof of Proposition 3.3 can be readily derived using Schur's Lemma, we have opted to include an elementary proof of Proposition 3.3 in the appendix for the sake of completeness. If we define
$$
\overline {{\mathcal {P}}} _ {\text {s t e e r}} (\mathbf {W}) := \mathbf {w} _ {1} M \left(\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}\right), \quad \mathcal {P} _ {\text {s t e e r}} (\mathbf {W}) := \left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top}\right) \overline {{\mathcal {P}}} _ {\text {s t e e r}} (\mathbf {W}), \tag {10}
$$
where  $g^{\top} \coloneqq \rho_0(g)^{\top}$ , then
$$
f _ {\text {s t e e r}} (\mathbf {x}; \mathbf {W}) = \left\langle \mathbf {x}, \mathcal {P} _ {\text {s t e e r}} (\mathbf {W}) \right\rangle = \left\langle \overline {{\mathbf {x}}}, \overline {{\mathcal {P}}} _ {\text {s t e e r}} (\mathbf {W}) \right\rangle . \tag {11}
$$
By the multi-linearity of  $M(\mathbf{w}_2, \dots, \mathbf{w}_L)$ , one can verify that  $f_{\mathrm{steer}}(\mathbf{x}; \mathbf{W})$ ,  $\mathcal{P}_{\mathrm{steer}}(\mathbf{W})$ , and  $\overline{\mathcal{P}}_{\mathrm{steer}}(\mathbf{W})$  are all  $L$ -homogeneous in  $\mathbf{W}$ ; that is, for all  $\nu > 0$  and  $\mathbf{W} \in \mathcal{W}_{\mathrm{steer}}$ ,
$$
f _ {\text {s t e e r}} (\mathbf {x}; \nu \mathbf {W}) = \nu^ {L} f _ {\text {s t e e r}} (\mathbf {x}; \mathbf {W}), \mathcal {P} _ {\text {s t e e r}} (\nu \mathbf {W}) = \nu^ {L} \mathcal {P} _ {\text {s t e e r}} (\mathbf {W}), \overline {{\mathcal {P}}} _ {\text {s t e e r}} (\nu \mathbf {W}) = \nu^ {L} \overline {{\mathcal {P}}} _ {\text {s t e e r}} (\mathbf {W}). \tag {12}
$$
Remark 3.4. In comparison, an  $L$ -layer linear fully-connected network  $f_{\mathrm{fc}}(\mathbf{x}; \mathbf{W})$  is given by
$$
f _ {\mathrm {f c}} (\mathbf {x}; \mathbf {W}) = \mathbf {w} _ {L} ^ {\top} \mathbf {w} _ {L - 1} ^ {\top} \dots \mathbf {w} _ {1} ^ {\top} \mathbf {x} = \left\langle \mathbf {x}, \mathcal {P} _ {\mathrm {f c}} (\mathbf {W}) \right\rangle , \quad \mathcal {P} _ {\mathrm {f c}} (\mathbf {W}) := \mathbf {w} _ {1} \dots \mathbf {w} _ {L}. \tag {13}
$$
where  $\mathbf{W} = [\mathbf{w}_l]_{l=1}^L \in \mathcal{W}_{\mathrm{fc}} := \left( \prod_{l=1}^{L-1} \mathbb{R}^{d_{l-1} \times d_l} \right) \times \mathbb{R}^{d_{L-1}}$ . It is worth noting that when  $G = \{e\}$  is the trivial group, a linear fully-connected network  $f_{\mathrm{fc}}(\mathbf{x}; \mathbf{W})$  is identical with a linear  $G$ -CNN, which is a special case of linear steerable networks. See Remark A.1 for details.
3.3 Group-invariant binary classification
Consider a binary classification data set  $S = \{(\mathbf{x}_i, y_i) : i \in [n]\}$ , where  $\mathbf{x}_i \in \mathbb{R}^{d_0}$  and  $y_i \in \{\pm 1\}, \forall i \in [n]$ . We assume that  $S$  are i.i.d. samples from a  $G$ -invariant distribution  $\mathcal{D}$  defined below.
Definition 3.5. A distribution  $\mathcal{D}$  over  $\mathbb{R}^{d_0} \times \{\pm 1\}$  is said to be  $G$ -invariant with respect to a representation  $\rho_0 \in GL(\mathbb{R}^{d_0})$  if
$$
\left(\rho_ {0} (g) \otimes \operatorname {I d}\right) _ {*} \mathcal {D} = \mathcal {D}, \forall g \in G, \tag {14}
$$
where  $(\rho_0(g)\otimes \mathrm{Id})(\mathbf{x},y)\coloneqq (\rho_0(g)\mathbf{x},y)$ , and  $(\rho_0(g)\otimes \mathrm{Id})_*\mathcal{D}\coloneqq \mathcal{D}\circ (\rho_0(g)\otimes \mathrm{Id})^{-1}$  is the pushforward measure of  $\mathcal{D}$  under  $\rho_0(g)\otimes \mathrm{Id}$ .
It is easy to verify that the Bayes optimal classifier  $f^{*}:\mathbb{R}^{d_{0}}\to \{\pm 1\}$  (the one achieving the smallest population risk) for a  $G$ -invariant distribution  $\mathcal{D}$  is necessarily a  $G$ -invariant function, i.e.,  $f^{*}(g\mathbf{x}) = f^{*}(\mathbf{x}),\forall g\in G$ . Therefore, to learn  $f^{*}$  using (linear) neural networks, it is natural to approximate  $f^{*}$  using an equivariant steerable network
$$
f ^ {*} (\mathbf {x}) \approx \operatorname {s i g n} \left(f _ {\text {s t e e r}} (\mathbf {x}; \mathbf {W})\right) = \operatorname {s i g n} \left(\langle \mathbf {x}, \mathcal {P} _ {\text {s t e e r}} (\mathbf {W}) \rangle\right). \tag {15}
$$
After choosing the exponential loss  $\ell_{\mathrm{exp}}: \mathbb{R} \times \{\pm 1\} \to \mathbb{R}_+$ ,  $\ell_{\mathrm{exp}}(\hat{y}, y) := \exp(-\hat{y}y)$ , as a surrogate loss function, the empirical risk minimization over  $S$  for the steerable network  $f_{\mathrm{steer}}(\mathbf{x}; \mathbf{W})$  becomes
$$
\min  _ {\mathbf {W} \in \mathcal {W} _ {\text {s t e e r}}} \mathcal {L} _ {\mathcal {P} _ {\text {s t e e r}}} (\mathbf {W}; S) := \sum_ {i = 1} ^ {n} \ell_ {\exp} \left(\left\langle \mathbf {x} _ {i}, \mathcal {P} _ {\text {s t e e r}} (\mathbf {W}) \right\rangle , y _ {i}\right) = \sum_ {i = 1} ^ {n} \ell_ {\exp} \left(\left\langle \overline {{\mathbf {x}}} _ {i}, \overline {{\mathcal {P}}} _ {\text {s t e e r}} (\mathbf {W}) \right\rangle , y _ {i}\right). \tag {16}
$$