SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning, pages 242-252. PMLR, 2019.
X. Cheng, Q. Qiu, R. Calderbank, and G. Sapiro. RotDCF: Decomposition of convolutional filters for rotation-equivariant deep networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1gTEj09FX.
L. Chizat and F. Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in neural information processing systems, 31, 2018.
T. Cohen and M. Welling. Group equivariant convolutional networks. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2990-2999, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https://proceedings.mlr.press/v48/cohenc16.html.
T. S. Cohen and M. Welling. Steerable CNNs. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=rJQKYt511.
T. S. Cohen, M. Geiger, and M. Weiler. A general theory of equivariant CNNs on homogeneous spaces. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/b9cfe8b6042cf759dc4c0cccb27a6737-Paper.pdf.
S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675-1685. PMLR, 2019.
B. Elesedy. Group symmetry in pac learning. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022.
B. Elesedy and S. Zaidi. Provably strict generalisation benefit for equivariant models. In International Conference on Machine Learning, pages 2959-2969. PMLR, 2021.
S. Gunasekar, J. Lee, D. Soudry, and N. Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pages 1832-1841. PMLR, 2018a.
S. Gunasekar, J. D. Lee, D. Soudry, and N. Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems, 31, 2018b.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
A G-CNNs
A special case of the steerable networks is the group convolutional neural network (G-CNN), wherein equivariance is achieved through group convolutions. More specifically, for each  $l \in [L - 1]$ , the hidden representation space  $(\mathcal{X}_l, \rho_l)$  is set to
$$
\mathcal {X} _ {l} = \left(\mathbb {R} ^ {d _ {l}}\right) ^ {G} = \left\{\mathbf {x} _ {l}: G \rightarrow \mathbb {R} ^ {d _ {l}} \right\}, \quad \rho_ {l} (g) \mathbf {x} _ {l} (h) := \mathbf {x} _ {l} \left(g ^ {- 1} h\right) \in \mathbb {R} ^ {d _ {1}}, \forall g, h \in G. \tag {41}
$$
The representation  $\rho_{l}\in \mathrm{GL}(\mathcal{X}_{l})$  in (41) is known as the regular representation of  $G$ . Intuitively,  $\mathbf{x}_l\in \mathcal{X}_l$  can be viewed as a matrix of size  $d_{l}\times |G|$ , and  $\rho_l(g)$  is a permutation of the columns of  $\mathbf{x}_l$ . With this choice of  $\{(\mathcal{X}_l,\rho_l)\}_{l = 0}^L$ , a (linear) G-CNN is built as follows.
First layer: the first-layer equivariant map  $\Psi_1^{\mathrm{G - CNN}}:\mathcal{X}_0\to \mathcal{X}_1$  of a G-CNN is defined as
$$
\Psi_ {1} ^ {\mathrm {G - C N N}} (\mathbf {x}; \mathbf {w} _ {1}) (g) = \mathbf {w} _ {1} ^ {\top} g ^ {- 1} \mathbf {x} \in \mathbb {R} ^ {d _ {1}}, \quad \forall \mathbf {x} \in \mathbb {R} ^ {d _ {0}}, \tag {42}
$$
where  $\mathbf{w}_1 = (w_1^{j,k})_{j,k} \in \mathbb{R}^{d_0 \times d_1}$  are the trainable parameters of the first layer. Eq. (42) is called a  $G$ -lifting map as it lifts a Euclidean signal  $\mathbf{x} \in \mathbb{R}^{d_0}$  to a function  $\Psi_1^{\mathrm{G - CNN}}(\mathbf{x}; \mathbf{w}_1)$  on  $G$ .
Hidden layers: For  $l \in \{2, \dots, L - 1\}$ , define  $\Psi_l^{\mathrm{G - CN}} : \mathcal{X}_{l - 1} \to \mathcal{X}_l$  as the group convolutions,
$$
\Psi_ {l} ^ {\mathrm {G - C N N}} (\mathbf {x}; \mathbf {w} _ {l}) (g) = \sum_ {h \in G} \mathbf {w} _ {l} ^ {\top} \left(h ^ {- 1} g\right) \mathbf {x} (h) \in \mathbb {R} ^ {d _ {l}}, \quad \forall \mathbf {x} \in \mathcal {X} _ {l - 1}, \tag {43}
$$
where  $\mathbf{w}_l(g) \in \mathbb{R}^{d_{l-1} \times d_l}, \forall g \in G$ . Equivalently,  $\mathbf{w}_l \in \mathbb{R}^{d_{l-1} \times d_l \times |G|}$  can be viewed as a 3D tensor.
Last layer: Define  $\Psi_L^{\mathrm{G - CNN}}:\mathcal{X}_{L - 1}\to \mathcal{X}_L$  as the group-pooling followed by a fully-connected layer,
$$
\Psi_ {L} ^ {\mathrm {G - C N N}} (\mathbf {x}; \mathbf {w} _ {L}) = \mathbf {w} _ {L} ^ {\top} \frac {1}{| G |} \sum_ {g \in G} \mathbf {x} (g), \quad \forall \mathbf {x} \in \mathcal {X} _ {L - 1}, \tag {44}
$$
where  $\mathbf{w}_L\in \mathbb{R}^{d_{L - 1}}$  is the weight of the last layer. An  $L$  -layer linear G-CNN is then the composition
$$
f _ {\mathrm {G} - \mathrm {C N N}} (\mathbf {x}; \mathbf {W}) = \Psi_ {L} ^ {\mathrm {G} - \mathrm {C N N}} \left(\dots \Psi_ {2} ^ {\mathrm {G} - \mathrm {C N N}} \left(\Psi_ {1} ^ {\mathrm {G} - \mathrm {C N N}} \left(\mathbf {x}; \mathbf {w} _ {1}\right); \mathbf {w} _ {2}\right) \dots ; \mathbf {w} _ {L}\right), \quad \mathbf {W} = [ \mathbf {w} _ {l} ] _ {l = 1} ^ {L}, \tag {45}
$$
where  $\mathbf{W} \in \mathbb{R}^{d_0 \times d_1} \times \left[\prod_{l=2}^{L-1} \mathbb{R}^{d_{l-1} \times d_l \times |G|}\right] \times \mathbb{R}^{d_{L-1}} =: \mathcal{W}_{\mathrm{G-CNN}}$  are the trainable weights.
Remark A.1. As a special case of linear steerable networks, a linear  $G$ -CNN  $f_{G\text{-CNN}}(\mathbf{x}; \mathbf{W})$  (45) can be written as
$$
f _ {\mathrm {G} - \mathrm {C N N}} (\mathbf {x}; \mathbf {W}) = \left\langle \mathbf {x}, \mathcal {P} _ {\mathrm {G} - \mathrm {C N N}} (\mathbf {W}) \right\rangle = \left\langle \overline {{\mathbf {x}}}, \bar {\mathcal {P}} _ {\mathrm {G} - \mathrm {C N N}} (\mathbf {W}) \right\rangle , \tag {46}
$$
$$
\overline {{\mathcal {P}}} _ {\mathrm {G} - \mathrm {C N N}} (\mathbf {W}) := \mathbf {w} _ {1} \left[ \prod_ {l = 2} ^ {L - 1} \left(\sum_ {g \in G} \mathbf {w} _ {l} (g)\right) \right] \mathbf {w} _ {L}, \mathcal {P} _ {\mathrm {G} - \mathrm {C N N}} (\mathbf {W}) := \left[ \frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \right] \overline {{\mathcal {P}}} _ {\mathrm {G} - \mathrm {C N N}} (\mathbf {W}) \tag {47}
$$
$$
f _ {\text {s t e e r}} (\mathbf {x}; \mathbf {W}) = f _ {\text {s t e e r}} (\overline {{\mathbf {x}}}; \mathbf {W}) = \langle \overline {{\mathbf {x}}}, \mathbf {w} _ {1} M \left(\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}\right) \rangle , \tag {48}
$$
Proof. Since the linear steerable network  $f_{\mathrm{steer}}(\mathbf{x}; \mathbf{W})$  is  $G$ -invariant, we have  $f_{\mathrm{steer}}(\mathbf{x}; \mathbf{w}) = f_{\mathrm{steer}}(g\mathbf{x}; \mathbf{w})$  for all  $g \in G$ . Therefore,
$$
f _ {\text {s t e e r}} (\mathbf {x}; \mathbf {W}) = \frac {1}{| G |} \sum_ {g \in G} f _ {\text {s t e e r}} (g \mathbf {x}; \mathbf {W}) = f _ {\text {s t e e r}} \left(\frac {1}{| G |} \sum_ {g \in G} g \mathbf {x}; \mathbf {w}\right) = f _ {\text {s t e e r}} (\overline {{\mathbf {x}}}; \mathbf {W}), \tag {49}
$$
where the second equality is due to the linearity of  $f_{\mathrm{steer}}(\mathbf{x}; \mathbf{w})$  in  $\mathbf{x}$ . We thus have
$$
\begin{array}{l} f _ {\text {s t e e r}} (\mathbf {x}; \mathbf {W}) = f _ {\text {s t e e r}} (\overline {{\mathbf {x}}}; \mathbf {W}) = \Psi_ {L} ^ {\text {s t e e r}} \left(\dots \Psi_ {2} ^ {\text {s t e e r}} \left(\Psi_ {1} ^ {\text {s t e e r}} (\overline {{\mathbf {x}}}; \mathbf {w} _ {1}); \mathbf {w} _ {2}\right) \dots ; \mathbf {w} _ {L}\right) (50) \\ = \left\langle \Psi_ {1} ^ {\text {s t e e r}} (\overline {{\mathbf {x}}}; \mathbf {w}), \Phi \left(\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}\right) \right\rangle , (51) \\ \end{array}
$$
where  $\Phi (\mathbf{w}_2,\dots ,\mathbf{w}_L)\in \mathcal{X}_1 = \left(\mathbb{R}^{d_1}\right)^G$  is multi-linear with respect to  $(\mathbf{w}_1,\dots ,\mathbf{w}_L)$ . Under Assumption 3.1, we have
$$
\begin{array}{l} f _ {\text {s t e e r}} (\mathbf {x}; \mathbf {W}) = \sum_ {g \in G} \left\langle \Psi_ {1} ^ {\text {s t e e r}} (\bar {\mathbf {x}}; \mathbf {w}) (g), \Phi \left(\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}\right) (g) \right\rangle (52) \\ = \sum_ {g \in G} \left\langle \mathbf {w} _ {1} ^ {\top} g ^ {- 1} \bar {\mathbf {x}}, \Phi \left(\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}\right) (g) \right\rangle (53) \\ = \sum_ {g \in G} \left\langle \mathbf {w} _ {1} ^ {\top} \overline {{\mathbf {x}}}, \Phi \left(\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}\right) (g) \right\rangle (54) \\ = \left\langle \overline {{\mathbf {x}}}, \mathbf {w} _ {1} \left[ \sum_ {g \in G} \Phi \left(\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}\right) (g) \right] \right\rangle (55) \\ = \left\langle \overline {{\mathbf {x}}}, \mathbf {w} _ {1} M \left(\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}\right) \right\rangle , (56) \\ \end{array}
$$
where  $M(\mathbf{w}_2, \dots, \mathbf{w}_L) \coloneqq \sum_{g \in G} \Phi(\mathbf{w}_2, \dots, \mathbf{w}_L)(g)$  is also multi-linear in  $(\mathbf{w}_2, \dots, \mathbf{w}_L)$ .
$$
\mathbb {R} _ {G} ^ {d _ {0}} = \bigcap_ {g \in G} \ker (I - g ^ {\top}) = \operatorname {R a n g e} \left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top}\right). \tag {57}
$$
$$
\mathcal {A} (\beta) := \bar {\beta} = \frac {1}{| G |} \sum_ {g \in G} g \beta . \tag {58}
$$
$$
\bar {\beta} = \beta \Longleftrightarrow \beta \in \mathbb {R} _ {G} ^ {d _ {0}}, \quad a n d \| \bar {\beta} \| \leq \| \beta \|, \forall \beta \in \mathbb {R} ^ {d _ {0}}. \tag {59}
$$
Proof. To prove (a), for the first equality, a vector  $\beta \in \mathbb{R}_G^{d_0}$  if and only if  $0 = \beta^\top (I - g)\mathbf{x} = \left\langle (I - g)^\top \beta ,\mathbf{x}\right\rangle ,\forall \mathbf{x}\in \mathbb{R}^{d_0},\forall g\in G$  . This is equivalent to  $\beta \in \bigcap_{g\in G}\ker (I - g^{\top})$  , and therefore  $\mathbb{R}_G^{d_0} = \bigcap_{g\in G}\ker (I - g^{\top})$
For the second equality, if  $\beta \in \bigcap_{g\in G}\ker (I - g^{\top})$ , then  $(I - g^{\top})\beta = 0,\forall g\in G$ . Hence
$$
\begin{array}{l} 0 = \frac {1}{| G |} \sum_ {g \in G} (I - g ^ {\top}) \beta = \beta - \frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \beta (60) \\ \Rightarrow \beta = \frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \beta \in \text {R a n g e} \left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top}\right) (61) \\ \end{array}
$$
On the other hand, if  $\beta = \frac{1}{|G|}\sum_{h\in G}h^{\top}\mathbf{y}$  for some  $\mathbf{y}\in \mathbb{R}^{d_0}$ , then for all  $g\in G$
$$
\left(I - g ^ {\top}\right) \boldsymbol {\beta} = \left(I - g ^ {\top}\right) \frac {1}{| G |} \sum_ {h \in G} h ^ {\top} \mathbf {y} = \frac {1}{| G |} \sum_ {h \in G} h ^ {\top} \mathbf {y} - \frac {1}{| G |} \sum_ {h \in G} (h g) ^ {\top} \mathbf {y} = 0. \tag {62}
$$
Thus  $\beta \in \bigcap_{g\in G}\ker (I - g^{\top})$
Point (b) can be easily derived from (57) and (60).
To prove (c), notice that
$$
\mathcal {A} ^ {\top} = \frac {1}{| G |} \sum_ {g \in G} g ^ {\top} = \frac {1}{| G |} \sum_ {g \in G} g ^ {- 1} = \frac {1}{| G |} \sum_ {g \in G} g = \mathcal {A}. \tag {63}
$$
Hence  $\mathcal{A} = \mathcal{A}^{\top}$  is self-adjoint. This combined with point (b) implies that  $\mathcal{A} = \mathcal{A}^{\top}:\beta \mapsto \overline{\beta}$  is an orthogonal projection from  $\mathbb{R}^{d_0}$  onto  $\mathbb{R}_G^{d_0}$ .
C Proof of Theorem 4.1
$$
\boldsymbol {\gamma} ^ {*} = \arg \min  _ {\boldsymbol {\gamma} \in \mathbb {R} ^ {d _ {0}}} \| \boldsymbol {\gamma} \| ^ {2}, \quad \text {s . t .} y _ {i} \left\langle \overline {{\mathbf {x}}} _ {i}, \boldsymbol {\gamma} \right\rangle \geq 1, \forall i \in [ n ]. \tag {64}
$$
$$
\boldsymbol {\beta} ^ {*} = \arg \min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {d _ {0}}} \| \operatorname {P r o j} _ {\text {R a n g e} (\mathcal {A})} \boldsymbol {\beta} \| ^ {2}, \quad \text {s . t .} \boldsymbol {\beta} \in \mathbb {R} _ {G} ^ {d _ {0}}, \text {a n d} y _ {i} \left\langle \mathbf {x} _ {i}, \boldsymbol {\beta} \right\rangle \geq 1, \forall i \in [ n ], \tag {65}
$$
$$
\boldsymbol {\beta} _ {\text {s t e e r}} ^ {\infty} \propto \boldsymbol {\beta} ^ {*} = \arg \min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {d _ {0}}} \| \boldsymbol {\beta} \| ^ {2}, \quad \text {s . t .} \boldsymbol {\beta} \in \mathbb {R} _ {G} ^ {d _ {0}}, \text {a n d} y _ {i} \left\langle \mathbf {x} _ {i}, \boldsymbol {\beta} \right\rangle \geq 1, \forall i \in [ n ]. \tag {66}
$$
Before proving Theorem 4.1, we need the following lemma which holds for general  $L$ -homogeneous networks, of which linear steerable networks are a special case. Note that in what follows,  $\|\cdot\|$  always denotes the Euclidean norm of a tensor viewed as a one-dimensional vector.
Lemma C.1 (paraphrased from Lyu and Li [2020] and Ji and Telgarsky [2020]). Under Assumption 3.7, we have the following results of directional convergence and alignment.
(a)  $\mathcal{L}_{\mathcal{P}_{\mathrm{steer}}}(\mathbf{W}(t);S)\to 0$  as  $t\to \infty$  . Consequently,  $\| \mathbf{W}(t)\| \to \infty$  and  $\| \mathcal{P}_{\mathrm{steer}}(\mathbf{W}(t))\| \to \infty$
(b) The directional convergence and alignment of the parameters  $\mathbf{W}(t)$  and the gradients  $\nabla_{\mathbf{W}}\mathcal{L}_{\mathcal{P}_{\mathrm{steer}}}(\mathbf{W}(t))$
$$
\lim  _ {t \rightarrow \infty} \frac {\mathbf {W} (t)}{\| \mathbf {W} (t) \|} = \mathbf {W} ^ {\infty} = - \lim  _ {t \rightarrow \infty} \frac {\nabla_ {\mathbf {W}} \mathcal {L} _ {\mathcal {P} _ {\text {s t e e r}}} (\mathbf {W} (t) ; S)}{\| \nabla_ {\mathbf {W}} \mathcal {L} _ {\mathcal {P} _ {\text {s t e e r}}} (\mathbf {W} (t) ; S) \|}, \tag {67}
$$
for some  $\mathbf{W}^{\infty}\in \mathcal{W}_{\mathrm{steer}}$  with  $\| \mathbf{W}^{\infty}\| = 1$
(c) The limit  $\mathbf{W}^{\infty}$  is along the direction of a first-order stationary point of the constrained optimization problem
$$
\min  _ {\mathbf {W}} \| \mathbf {W} \| ^ {2}, \quad \text {s . t .} y _ {i} \left\langle \mathbf {x} _ {i}, \mathcal {P} _ {\text {s t e e r}} (\mathbf {W}) \right\rangle \geq 1, \forall i \in [ n ]. \tag {68}
$$
In other words, there exists a scaling factor  $\tau > 0$  such that  $\tau \mathbf{W}^{\infty}$  satisfies the Karush-Kuhn-Tucker (KKT) conditions of (68).
Proof of Theorem 4.1. The first part of the proof is inspired by Gunasekar et al. [2018b]. To prove (a), let  $\mathbf{W}^{\infty} = [\mathbf{w}_l^{\infty}]_{l=1}^{L} = \lim_{t\to \infty}\frac{\mathbf{W}(t)}{\|\mathbf{W}(t)\|}$  be the limit direction of  $\mathbf{W}(t)$ , and let  $\widetilde{\mathbf{W}}^{\infty} = [\widetilde{\mathbf{w}}_l^{\infty}]_{l=1}^{L} = \tau \mathbf{W}^{\infty}$ ,  $\tau > 0$ , be the stationary point of problem (68) by Lemma C.1. The KKT condition for  $\widetilde{\mathbf{W}}^{\infty}$  implies the existence of dual variables  $\alpha_i \geq 0, i \in [n]$ , such that
- Primal feasibility:
$$
y _ {i} \left\langle \mathcal {P} _ {\text {s t e e r}} \left(\widetilde {\mathbf {W}} ^ {\infty}\right), \mathbf {x} _ {i} \right\rangle \geq 1, \quad \forall i \in [ n ]. \tag {69}
$$
- Stationarity:
$$
\widetilde {\mathbf {W}} ^ {\infty} = \nabla_ {\mathbf {W}} \mathcal {P} _ {\text {s t e e r}} \left(\widetilde {\mathbf {W}} ^ {\infty}\right) \cdot \left(\sum_ {i = 1} ^ {n} \alpha_ {i} y _ {i} \mathbf {x} _ {i}\right) \tag {70}
$$
- Complementary slackness:
$$
y _ {i} \left\langle \mathcal {P} _ {\text {s t e e r}} \left(\widetilde {\mathbf {W}} ^ {\infty}\right), \mathbf {x} _ {i} \right\rangle > 1 \Rightarrow \alpha_ {i} = 0. \tag {71}
$$
We claim that  $\gamma^{*} = \overline{\mathcal{P}}_{\mathrm{steer}}(\widetilde{\mathbf{W}}^{\infty})$  is a stationary point of the following problem
$$
\min  _ {\boldsymbol {\gamma} \in \mathbb {R} ^ {d _ {0}}} \| \boldsymbol {\gamma} \| ^ {2}, \quad \text {s . t .} y _ {i} \left\langle \overline {{\mathbf {x}}} _ {i}, \boldsymbol {\gamma} \right\rangle \geq 1, \forall i \in [ n ]. \tag {72}
$$
That is, there exists  $\tilde{\alpha}_i\geq 0,i\in [n]$  , such that the following conditions are satisfied.
$$
y _ {i} \left\langle \overline {{\mathcal {P}}} _ {\text {s t e e r}} \left(\widetilde {\mathbf {W}} ^ {\infty}\right), \bar {\mathbf {x}} _ {i} \right\rangle \geq 1, \quad \forall i \in [ n ]. \tag {73}
$$
$$
\overline {{\mathcal {P}}} _ {\text {s t e e r}} \left(\widetilde {\mathbf {W}} ^ {\infty}\right) = \sum_ {i = 1} ^ {n} \tilde {\alpha} _ {i} y _ {i} \bar {\mathbf {x}} _ {i} \tag {74}
$$
$$
y _ {i} \left\langle \overline {{\mathcal {P}}} _ {\text {s t e e r}} \left(\widetilde {\mathbf {W}} ^ {\infty}\right), \bar {\mathbf {x}} _ {i} \right\rangle > 1 \Longrightarrow \tilde {\alpha} _ {i} = 0. \tag {75}
$$
Indeed, Eq. (73) holds due to Eq. (69) and  $\left\langle \mathcal{P}_{\mathrm{steer}}(\widetilde{\mathbf{W}}^{\infty}),\mathbf{x}_i\right\rangle = \left\langle \overline{\mathcal{P}}_{\mathrm{steer}}(\widetilde{\mathbf{W}}^{\infty}),\overline{\mathbf{x}}_i\right\rangle$  . Moreover, by the definition of  $\mathcal{P}_{\mathrm{steer}}(\mathbf{W})$  and  $\overline{\mathcal{P}}_{\mathrm{steer}}(\mathbf{W})$  in (10), we have for any  $\mathbf{z}\in \mathbb{R}^{d_0}$
$$
\nabla_ {\mathbf {w} _ {1}} \bar {\mathcal {P}} _ {\text {s t e e r}} (\mathbf {W}) \cdot \mathbf {z} = \mathbf {z} \left[ M \left(\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}\right) \right] ^ {\top} \tag {76}
$$
$$
\nabla_ {\mathbf {w} _ {1}} \mathcal {P} _ {\text {s t e e r}} (\mathbf {W}) \cdot \mathbf {z} = \left(\frac {1}{| G |} \sum_ {g \in G} g\right) \mathbf {z} [ M (\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}) ] ^ {\top} = \bar {\mathbf {z}} [ M (\mathbf {w} _ {2}, \dots , \mathbf {w} _ {L}) ] ^ {\top}. \tag {77}
$$
Therefore Eq. (70) implies
$$
\widetilde {\mathbf {w}} _ {1} ^ {\infty} = \nabla_ {\mathbf {w} _ {1}} \mathcal {P} _ {\text {s t e e r}} \left(\widetilde {\mathbf {W}} ^ {\infty}\right) \cdot \left(\sum_ {i = 1} ^ {n} \alpha_ {i} y _ {i} \mathbf {x} _ {i}\right) = \left(\sum_ {i = 1} ^ {n} \alpha_ {i} y _ {i} \overline {{\mathbf {x}}} _ {i}\right) \cdot \left[ M \left(\widetilde {\mathbf {w}} _ {2} ^ {\infty}, \dots , \widetilde {\mathbf {w}} _ {L} ^ {\infty}\right) \right] ^ {\top} \tag {78}
$$
$$
\begin{array}{l} \overline {{\mathcal {P}}} _ {\mathrm {s t e e r}} (\widetilde {\mathbf {W}} ^ {\infty}) = \widetilde {\mathbf {w}} _ {1} ^ {\infty} M (\widetilde {\mathbf {w}} _ {2} ^ {\infty}, \dots , \widetilde {\mathbf {w}} _ {L} ^ {\infty}) = \left(\sum_ {i = 1} ^ {n} \alpha_ {i} y _ {i} \overline {{\mathbf {x}}} _ {i}\right) \| M (\widetilde {\mathbf {w}} _ {2} ^ {\infty}, \dots , \widetilde {\mathbf {w}} _ {L} ^ {\infty}) \| ^ {2} \\ = c \sum_ {i = 1} ^ {n} \alpha_ {i} y _ {i} \bar {\mathbf {x}} _ {i}, \tag {79} \\ \end{array}
$$
where  $c = \| M(\widetilde{\mathbf{w}}_2^\infty, \dots, \widetilde{\mathbf{w}}_L^\infty)\|^2 \geq 0$ . From (73) we know that  $\overline{\mathcal{P}}_{\mathrm{steer}}(\widetilde{\mathbf{W}}^\infty) \neq 0$  and hence  $c > 0$ . Let  $\tilde{\alpha}_i = c\alpha_i, \forall i \in [n]$ . Then  $\tilde{\alpha}_i \geq 0$ , and the stationarity (74) is satisfied due to Eq. (79), and the complementary slackness also holds due to  $\tilde{\alpha}_i$  being a positive scaling of  $\alpha_i$  for all  $i \in [n]$ . Therefore  $\gamma^* = \overline{\mathcal{P}}_{\mathrm{steer}}(\widetilde{\mathbf{W}}^\infty)$  is a stationary point of (72). Since problem (72) is strongly convex,  $\gamma^* = \overline{\mathcal{P}}_{\mathrm{steer}}(\widetilde{\mathbf{W}}^\infty)$  is in fact the unique minimizer of (72). Hence the limit direction of the predictor  $\mathcal{P}_{\mathrm{steer}}(\mathbf{W}(t))$  is
$$
\begin{array}{l} \beta_ {\text {s t e e r}} ^ {\infty} = \lim  _ {t \rightarrow \infty} \frac {\mathcal {P} _ {\text {s t e e r}} (\mathbf {W} (t))}{\| \mathcal {P} _ {\text {s t e e r}} (\mathbf {W} (t)) \|} = \lim  _ {t \rightarrow \infty} \frac {\left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top}\right) \overline {{\mathcal {P}}} _ {\text {s t e e r}} (\mathbf {W} (t))}{\left\|\left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top}\right) \overline {{\mathcal {P}}} _ {\text {s t e e r}} (\mathbf {W} (t)) \right\|} (80) \\ = \lim  _ {t \rightarrow \infty} \frac {\left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top}\right) \overline {{\mathcal {P}}} _ {\text {s t e e r}} \left(\frac {\tau \mathbf {W} (t)}{\| \mathbf {W} (t) \|}\right)}{\left\|\left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top}\right) \overline {{\mathcal {P}}} _ {\text {s t e e r}} \left(\frac {\tau \mathbf {W} (t)}{\| \mathbf {W} (t) \|}\right)\right\|} = \frac {\left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top}\right) \overline {{\mathcal {P}}} _ {\text {s t e e r}} \left(\widetilde {\mathbf {W}} ^ {\infty}\right)}{\left\|\left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top}\right) \overline {{\mathcal {P}}} _ {\text {s t e e r}} \left(\widetilde {\mathbf {W}} ^ {\infty}\right)\right\|} (81) \\ \propto \frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \gamma^ {*}, (82) \\ \end{array}
$$
where the third equality is due to  $\overline{\mathcal{P}}_{\mathrm{steer}}$  being  $L$ -homogeneous (12), and the fourth equality comes from the continuity of  $\overline{\mathcal{P}}_{\mathrm{steer}}$  and that  $\mathcal{P}_{\mathrm{steer}}(\widetilde{\mathbf{W}}^{\infty}) = \left(\frac{1}{|G|}\sum_{g\in G}g^{\top}\right)\overline{\mathcal{P}}_{\mathrm{steer}}\left(\widetilde{\mathbf{W}}^{\infty}\right)\neq 0$  (because otherwise (69) can not hold.)
Finally, if  $G$  acts unitarily on  $\mathcal{X}_0$ , then Eq. (74) combined with Proposition 3.6 implies that  $\gamma^{*} = \overline{\mathcal{P}}_{\mathrm{steer}}(\widetilde{\mathbf{W}}^{\infty}) \in \mathbb{R}_{G}^{d_0}$ . Hence
$$
\beta_ {\text {s t e e r}} ^ {\infty} \propto \frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \boldsymbol {\gamma} ^ {*} = \frac {1}{| G |} \sum_ {g \in G} g \boldsymbol {\gamma} ^ {*} = \overline {{\boldsymbol {\gamma} ^ {*}}} = \boldsymbol {\gamma} ^ {*}, \tag {83}
$$
where the last equality is again due to Proposition 3.6. This concludes the proof of (a).
To prove (b), we first show that problem (65) has a unique minimizer. To prove this, notice that  $\mathrm{Proj}_{\mathrm{Range}(\mathcal{A})}:\mathbb{R}_G^{d_0}\to \mathrm{Range}(\mathcal{A})$  is injective; indeed, for any  $\beta \in \mathbb{R}_G^{d_0}$
$$
\operatorname {P r o j} _ {\text {R a n g e} (\mathcal {A})} \beta = 0 \Longrightarrow \beta \in \operatorname {R a n g e} (\mathcal {A}) ^ {\perp} = \ker \left(\mathcal {A} ^ {\top}\right). \tag {84}
$$
Thus  $0 = \mathcal{A}^{\top}\beta = \beta$ , where the second equality is due to  $\mathcal{A}^{\top}$  being a projection onto  $\mathrm{Range}(\mathcal{A}^{\top}) = \mathbb{R}_G^{d_0}$  (Proposition 3.6) and  $\beta \in \mathbb{R}_G^{d_0}$ . Therefore the objective function in (65) is strongly convex on  $\mathbb{R}_G^{d_0}$  and there exists a unique minimizer.
To show  $\beta_{\mathrm{steer}}^{\infty} \propto \beta^{*}$ , it suffices to verify that  $\frac{1}{|G|} \sum_{g \in G} g^{\top} \gamma^{*} = \beta^{*}$  is the minimizer of (65). To this end, we notice that
$\frac{1}{|G|}\sum_{g\in G}g^{\top}\pmb{\gamma}^{*} = \mathcal{A}^{\top}\pmb{\gamma}^{*}\in \mathbb{R}_{G}^{d_{0}}$  by Proposition 3.6;
for all  $i\in [n]$
$$
y _ {i} \left\langle \mathbf {x} _ {i}, \frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \boldsymbol {\gamma} ^ {*} \right\rangle = y _ {i} \left\langle \overline {{\mathbf {x}}} _ {i}, \boldsymbol {\gamma} ^ {*} \right\rangle \geq 1. \tag {85}
$$
Thus  $\frac{1}{|G|}\sum_{g\in G}g^{\top}\pmb{\gamma}^{*}$  satisfies the constraints in (65). Assume for the sake of contradiction that  $\frac{1}{|G|}\sum_{g\in G}g^{\top}\pmb{\gamma}^{*}\neq \pmb{\beta}^{*}$ , then by the uniqueness of the minimizer of (65) we have