SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\| \widetilde {\gamma} \| := \left\| \operatorname {P r o j} _ {\text {R a n g e} (\mathcal {A})} \beta^ {*} \right\| <   \left\| \operatorname {P r o j} _ {\text {R a n g e} (\mathcal {A})} \frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \gamma^ {*} \right\|, \tag {86}
$$
where  $\widetilde{\gamma} \coloneqq \mathrm{Proj}_{\mathrm{Range}(\mathcal{A})}\beta^{*}$ . We make the following two claims to be proved shortly:
- Claim 1:  $y_{i}\langle \widetilde{\gamma},\overline{\mathbf{x}}_{i}\rangle = y_{i}\langle \boldsymbol{\beta}^{*},\mathbf{x}_{i}\rangle \geq 1, \forall i\in [n]$ .
- Claim 2:  $\operatorname{Proj}_{\operatorname{Range}(\mathcal{A})}\frac{1}{|G|}\sum_{g\in G}g^{\top}\pmb{\gamma}^{*} = \pmb{\gamma}^{*}$ .
These two claims combined with (86) imply that  $\widetilde{\gamma}$  satisfies the constraint in (64) and has a smaller norm compared to  $\gamma^{*}$ ,  $\|\widetilde{\gamma}\| < \|\gamma^{*}\|$ . This contradicts the fact that  $\gamma^{*}$  is the minimizer of problem (64). Therefore we have  $\frac{1}{|G|} \sum_{g \in G} g^{\top} \gamma^{*} = \beta^{*}$ .
If we assume in addition that  $\rho_0$  is unitary on  $\mathcal{X}_0$ , then  $\mathcal{A} = \mathcal{A}^{\top}:\beta \mapsto \overline{\beta}$  is an orthogonal projection from  $\mathbb{R}^{d_0}$  onto  $\mathbb{R}_G^{d_0} = \mathrm{Range}(\mathcal{A})$ . Therefore  $\operatorname{Proj}_{\mathrm{Range}(\mathcal{A})}\beta = \beta$  for  $\beta \in \mathbb{R}_G^{d_0}$ , and hence problems (65) and (66) are equivalent.
Finally, we prove the above two claims.
Proof of Claim 1:  $y_{i}\langle \widetilde{\gamma},\overline{\mathbf{x}}_{i}\rangle = y_{i}\langle \boldsymbol{\beta}^{*},\mathbf{x}_{i}\rangle \geq 1,\forall i\in [n]$
Since  $\widetilde{\gamma} = \mathrm{Proj}_{\mathrm{Range}(\mathcal{A})}\beta^{*}$ , we have
$$
\widetilde {\gamma} - \boldsymbol {\beta} ^ {*} \in \operatorname {R a n g e} (\mathcal {A}) ^ {\perp} = \ker \left(\mathcal {A} ^ {\top}\right). \tag {87}
$$
$$
\mathcal {A} ^ {\top} \widetilde {\gamma} = \mathcal {A} ^ {\top} \beta^ {*} = \beta^ {*}, \tag {88}
$$
where the second equality is due to  $\beta^{*}\in \mathbb{R}_{G}^{d_{0}} = \mathrm{Range}(\mathcal{A}^{\top})$  and  $\mathcal{A}^{\top} = \mathcal{A}^{\top}\circ \mathcal{A}^{\top}$  is a projection; cf. Proposition 3.6. Therefore, for all  $i\in [n]$ ,
$$
y _ {i} \left\langle \widetilde {\gamma}, \overline {{\mathbf {x}}} _ {i} \right\rangle = y _ {i} \left\langle \mathcal {A} ^ {\top} \widetilde {\gamma}, \mathbf {x} _ {i} \right\rangle = y _ {i} \left\langle \boldsymbol {\beta} ^ {*}, \mathbf {x} _ {i} \right\rangle \geq 1. \tag {89}
$$
Proof of Claim 2:  $\operatorname{Proj}_{\operatorname{Range}(\mathcal{A})}\mathcal{A}^\top \pmb{\gamma}^* = \pmb{\gamma}^*$ .
We first note that both  $\mathrm{Proj}_{\mathrm{Range}(\mathcal{A})}\mathcal{A}^\top \pmb{\gamma}^*$  and  $\pmb{\gamma}^{*}$  are in the affine space  $\mathcal{A}^{\top}\pmb{\gamma}^{*} + \ker (\mathcal{A}^{\top})$ . Indeed,
$$
\left(\mathcal {A} ^ {\top} \boldsymbol {\gamma} ^ {*} - \operatorname {P r o j} _ {\text {R a n g e} (\mathcal {A})} \mathcal {A} ^ {\top} \boldsymbol {\gamma} ^ {*}\right) \in \operatorname {R a n g e} (\mathcal {A}) ^ {\perp} = \ker \left(\mathcal {A} ^ {\top}\right), \tag {90}
$$
$$
\mathcal {A} ^ {\top} \left(\gamma^ {*} - \mathcal {A} ^ {\top} \gamma^ {*}\right) = \mathcal {A} ^ {\top} \gamma^ {*} - \mathcal {A} ^ {\top} \mathcal {A} ^ {\top} \gamma^ {*} = 0. \tag {91}
$$
This also implies
$$
\mathcal {A} ^ {\top} \left(\operatorname {P r o j} _ {\text {R a n g e} (\mathcal {A})} \mathcal {A} ^ {\top} \boldsymbol {\gamma} ^ {*}\right) = \mathcal {A} ^ {\top} \boldsymbol {\gamma} ^ {*}. \tag {92}
$$
Moreover, for any  $i\in [n]$
$$
\begin{array}{l} y _ {i} \left\langle \operatorname {P r o j} _ {\text {R a n g e} (\mathcal {A})} \mathcal {A} ^ {\top} \boldsymbol {\gamma} ^ {*}, \overline {{\mathbf {x}}} _ {i} \right\rangle = y _ {i} \left\langle \mathcal {A} ^ {\top} \left(\operatorname {P r o j} _ {\text {R a n g e} (\mathcal {A})} \mathcal {A} ^ {\top} \boldsymbol {\gamma} ^ {*}\right), \mathbf {x} _ {i} \right\rangle (93) \\ = y _ {i} \left\langle \mathcal {A} ^ {\top} \boldsymbol {\gamma} ^ {*}, \mathbf {x} _ {i} \right\rangle = y _ {i} \left\langle \boldsymbol {\gamma} ^ {*}, \overline {{\mathbf {x}}} _ {i} \right\rangle \geq 1. (94) \\ \end{array}
$$
Hence  $\mathrm{Proj}_{\mathrm{Range}(\mathcal{A})}\mathcal{A}^\top \gamma^*$  also satisfies the constraint in (64). Since the orthogonal projection  $\mathrm{Proj}_{\mathrm{Range}(\mathcal{A})}\mathcal{A}^\top \gamma^*$  achieves the minimal norm among all vectors in the affine space  $\mathcal{A}^\top \gamma^* + \ker (\mathcal{A}^\top)$  which includes  $\gamma^*$ , this can only happen when  $\mathrm{Proj}_{\mathrm{Range}(\mathcal{A})}\mathcal{A}^\top \gamma^* = \gamma^*$  is the unique minimizer of (64).
This concludes the proof of Theorem 4.1.
D Proofs in Section 5
$$
\beta_ {\text {s t e e r}} ^ {\infty} = \beta_ {\mathrm {f c}} ^ {\infty}. \tag {95}
$$
Proof. By Theorem 4.1, a positive scaling  $\gamma^{*} = \tau \beta_{\mathrm{steer}}^{\infty}$  of  $\beta_{\mathrm{steer}}^{\infty}$  satisfies the KKT condition of (22). That is, there exists  $\alpha_{i}\geq 0,i\in [n]$ , such that
$$
y _ {i} \left\langle \gamma^ {*}, \bar {\mathbf {x}} _ {i} \right\rangle \geq 1, \quad \forall i \in [ n ]. \tag {96}
$$
$$
\boldsymbol {\gamma} ^ {*} = \sum_ {i = 1} ^ {n} \alpha_ {i} y _ {i} \bar {\mathbf {x}} _ {i} \tag {97}
$$
$$
y _ {i} \left\langle \boldsymbol {\gamma} ^ {*}, \overline {{\mathbf {x}}} _ {i} \right\rangle > 1 \Longrightarrow \alpha_ {i} = 0. \tag {98}
$$
Using Corollary 4.3, we only need to show that  $\gamma^{*}$  is also the solution of
$$
\arg \min  _ {\boldsymbol {\gamma} \in \mathbb {R} ^ {d _ {0}}} \| \boldsymbol {\gamma} \| ^ {2}, \quad \text {s . t .} y _ {i} \left\langle g \mathbf {x} _ {i}, \boldsymbol {\gamma} \right\rangle \geq 1, \forall i \in [ n ], \forall g \in G. \tag {99}
$$
That is, there exists  $\tilde{\alpha}_{i,g}\geq 0,\forall i\in [n],\forall g\in G$  , such that
$$
y _ {i} \left\langle \gamma^ {*}, g \mathbf {x} _ {i} \right\rangle \geq 1, \quad \forall i \in [ n ], \forall g \in G. \tag {100}
$$
$$
\boldsymbol {\gamma} ^ {*} = \sum_ {i = 1} ^ {n} \sum_ {g \in G} \tilde {\alpha} _ {i, g} y _ {i} g \mathbf {x} _ {i} \tag {101}
$$
$$
y _ {i} \left\langle \gamma^ {*}, g \mathbf {x} _ {i} \right\rangle > 1 \Longrightarrow \tilde {\alpha} _ {i, g} = 0. \tag {102}
$$
Indeed, we set  $\tilde{\alpha}_{i,g} = \frac{1}{|G|}\alpha_{i} \geq 0$ . Since  $\gamma^{*} \in \mathbb{R}_{G}^{d_{0}}$  if  $\rho_{0}$  is unitary (this can also be observed from Eq. (97)), we have  $y_{i}\left\langle \boldsymbol{\gamma}^{*}, g\mathbf{x}_{i}\right\rangle = y_{i}\left\langle \boldsymbol{\gamma}^{*}, \mathbf{x}_{i}\right\rangle, \forall g \in G, \forall i \in [n]$ . Hence we have primal feasibility (100) from (96),
$$
y _ {i} \left\langle \boldsymbol {\gamma} ^ {*}, g \mathbf {x} _ {i} \right\rangle = y _ {i} \left\langle \boldsymbol {\gamma} ^ {*}, \frac {1}{| G |} \sum_ {h \in G} h g \mathbf {x} _ {i} \right\rangle = y _ {i} \left\langle \boldsymbol {\gamma} ^ {*}, \overline {{\mathbf {x}}} _ {i} \right\rangle \geq 1, \quad \forall i \in [ n ], \forall g \in G. \tag {103}
$$
Stationarity (101) holds since
$$
\sum_ {i = 1} ^ {n} \sum_ {g \in G} \tilde {\alpha} _ {i, g} y _ {i} g \mathbf {x} _ {i} = \sum_ {i = 1} ^ {n} \sum_ {g \in G} \frac {1}{| G |} \alpha_ {i} y _ {i} g \mathbf {x} _ {i} = \sum_ {i = 1} ^ {n} \alpha_ {i} y _ {i} \bar {\mathbf {x}} _ {i} = \boldsymbol {\gamma} ^ {*}, \tag {104}
$$
where the last equality comes from (97). Finally, if  $y_{i}\left\langle \gamma^{*},g\mathbf{x}_{i}\right\rangle >1$ , then (103) and (98) imply that
$$
\tilde {\alpha} _ {i, g} = \frac {1}{| G |} \alpha_ {i} = 0. \tag {105}
$$
This proves the condition for complementary slackness (102).
$$
\beta_ {\text {s t e e r}} ^ {\infty} \propto \left(\frac {1}{| G |} \sum_ {g \in G} \rho_ {0} (g) ^ {\top} \rho_ {0} (g)\right) ^ {1 / 2} \beta_ {\mathrm {f c}} ^ {\rho_ {0}, \infty}. \tag {106}
$$
Proof. The proof is similar to that of Corollary 5.1. By Theorem 4.1, the limit direction  $\beta_{\mathrm{steer}}^{\infty}$  is proportional to  $\frac{1}{|G|}\sum_{g\in G}g^{\top}\pmb{\gamma}^{*}$ , where  $\pmb{\gamma}^{*}$  satisfies the same KKT condition ((96), (97) and (98)) of problem (22) as in the proof of Corollary 5.1. On the other hand, note that the loss function  $\mathcal{L}_{\mathcal{P}_{\mathrm{fc}}}^{\rho_0}(\mathbf{W};S_{\mathrm{aug}})$  (34) for the modified fully-connected network on the augmented data set  $S_{\mathrm{aug}}$  is equivalent to
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {P} _ {\mathrm {f c}}} ^ {\rho_ {0}} (\mathbf {W}; S _ {\mathrm {a u g}}) = \sum_ {g \in G} \sum_ {i = 1} ^ {n} \ell_ {\exp} \left(\langle g \mathbf {x} _ {i}, \mathcal {P} _ {\mathrm {f c}} (\mathbf {W}) \rangle_ {\rho_ {0}}, y _ {i}\right) (107) \\ = \sum_ {g \in G} \sum_ {i = 1} ^ {n} \ell_ {\exp} \left(\left\langle A g \mathbf {x} _ {i}, \mathcal {P} _ {\mathrm {f c}} (\mathbf {W}) \right\rangle , y _ {i}\right) (108) \\ = \mathcal {L} _ {\mathcal {P} _ {\mathrm {f c}}} (\mathbf {W}; \widetilde {S} _ {\mathrm {a u g}}), (109) \\ \end{array}
$$
$$
A = \left(\frac {1}{| G |} \sum_ {g \in G} \rho_ {0} (g) ^ {\top} \rho_ {0} (g)\right) ^ {1 / 2}, \quad \widetilde {S} _ {\mathrm {a u g}} = \left\{\left(A g \mathbf {x} _ {i}, y _ {i}\right): i \in [ n ], g \in G \right\}. \tag {110}
$$
Therefore, by Corollary 4.3,  $\beta_{\mathrm{fc}}^{\rho_0,\infty}$  is proportional to the solution  $\widetilde{\gamma}_{\mathrm{aug}}^{*}$  of
$$
\widetilde {\gamma} _ {\text {a u g}} ^ {*} = \arg \min  _ {\boldsymbol {\gamma} \in \mathbb {R} ^ {d _ {0}}} \| \boldsymbol {\gamma} \| ^ {2}, \quad \text {s . t .} y _ {i} \left\langle A g \mathbf {x} _ {i}, \boldsymbol {\gamma} \right\rangle \geq 1, \forall i \in [ n ], \forall g \in G. \tag {111}
$$
We claim that  $\widetilde{\gamma}_{\mathrm{aug}}^{*} = A^{-1}\left(\frac{1}{|G|}\sum_{g\in G}g^{\top}\gamma^{*}\right)$ . To see this, let  $\widetilde{\alpha}_{i,g} = \alpha_{i},\forall i\in [n]$ , where  $\alpha_{i}\geq 0$  is the dual variable for  $\gamma^{*}$  of problem (22). We verify below that the KKT conditions for problem (111) are satisfied for the primal-dual pair  $A^{-1}\left(\frac{1}{|G|}\sum_{g\in G}g^{\top}\gamma^{*}\right)$  and  $(\widetilde{\alpha}_{i,g})_{i\in [n],g\in G}$ .
- Primal feasibility: for any  $i \in [n]$  and  $h \in G$ ,
$$
\begin{array}{l} y _ {i} \left\langle A ^ {- 1} \left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \boldsymbol {\gamma} ^ {*}\right), A h \mathbf {x} _ {i} \right\rangle = y _ {i} \left\langle \frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \boldsymbol {\gamma} ^ {*}, h \mathbf {x} _ {i} \right\rangle (112) \\ = y _ {i} \left\langle \boldsymbol {\gamma} ^ {*}, \left(\frac {1}{| G |} \sum_ {g \in G} g\right) h \mathbf {x} _ {i} \right\rangle (113) \\ = y _ {i} \left\langle \boldsymbol {\gamma} ^ {*}, \overline {{\mathbf {x}}} _ {i} \right\rangle \geq 1, (114) \\ \end{array}
$$
where the first equality is due to  $A^{\top} = A$  being symmetric, and the last inequality is due to (96).
$$
\begin{array}{l} A ^ {- 1} \left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \boldsymbol {\gamma} ^ {*}\right) = A ^ {- 1} \left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top}\right) \sum_ {i = 1} ^ {n} \alpha_ {i} y _ {i} \overline {{\mathbf {x}}} _ {i} (115) \\ = A ^ {- 1} \frac {1}{| G | ^ {2}} \left(\sum_ {g \in G} g ^ {\top}\right) \sum_ {i = 1} ^ {n} \alpha_ {i} y _ {i} \sum_ {h \in G} h \mathbf {x} _ {i} (116) \\ = A \cdot A ^ {- 2} \frac {1}{| G | ^ {2}} \sum_ {i = 1} ^ {n} \alpha_ {i} y _ {i} \sum_ {g \in G} \sum_ {h \in G} g ^ {\top} h \mathbf {x} _ {i} (117) \\ = A \cdot A ^ {- 2} \frac {1}{| G |} \sum_ {i = 1} ^ {n} \alpha_ {i} y _ {i} \left(\frac {1}{| G |} \sum_ {g \in G} g ^ {\top} g\right) \sum_ {h \in G} h \mathbf {x} _ {i} (118) \\ = \sum_ {i = 1} ^ {n} \frac {\alpha_ {i}}{| G |} y _ {i} \sum_ {h \in G} A h \mathbf {x} _ {i} (119) \\ = \sum_ {i = 1} ^ {n} \sum_ {g \in G} \tilde {\alpha} _ {i, g} y _ {i} A g \mathbf {x} _ {i}, (120) \\ \end{array}
$$
where the first equality is due to (97), and the last equality comes from the definition of the dual variable  $\widetilde{\alpha}_{i,g} = \frac{1}{|G|}\alpha_{i},\forall i\in [n],\forall g\in G$
- Complementary slackness: if  $y_{i} \left\langle A^{-1}\left(\frac{1}{|G|}\sum_{g \in G} g^{\top}\boldsymbol{\gamma}^{*}\right), Ag\mathbf{x}_{i} \right\rangle > 1$  for some  $i \in [n]$  and  $g \in G$ , then (114) and (98) imply that  $\widetilde{\alpha}_{i,g} = \frac{1}{|G|}\alpha_{i} = 0$ .
Hence  $\widetilde{\gamma}_{\mathrm{aug}}^{*} = A^{-1}\left(\frac{1}{|G|}\sum_{g\in G}g^{\top}\gamma^{*}\right)$  is indeed the solution of (111). Therefore,
$$
\boldsymbol {\beta} _ {\text {s t e e r}} ^ {\infty} \propto \frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \boldsymbol {\gamma} ^ {*} = A \boldsymbol {\gamma} _ {\text {a u g}} ^ {*} \propto A \boldsymbol {\beta} _ {\mathrm {f c}} ^ {\rho_ {0}, \infty}. \tag {121}
$$
E Proofs of Theorem 6.1
$$
M _ {\text {s t e e r}} := \min  _ {i \in [ n ], g \in G} y _ {i} \left\langle \boldsymbol {\beta} _ {\text {s t e e r}} ^ {\infty}, g \mathbf {x} _ {i} \right\rangle , \quad M _ {\mathrm {f c}} := \min  _ {i \in [ n ], g \in G} y _ {i} \left\langle \boldsymbol {\beta} _ {\mathrm {f c}} ^ {\infty}, g \mathbf {x} _ {i} \right\rangle . \tag {122}
$$
Proof. By Corollary 4.3 and Corollary 5.1, we have
$$
\boldsymbol {\beta} _ {\text {s t e e r}} ^ {\infty} \propto \boldsymbol {\gamma} _ {\text {s t e e r}} ^ {*} = \arg \min  _ {\boldsymbol {\gamma} \in \mathbb {R} ^ {d _ {0}}} \| \boldsymbol {\gamma} \| ^ {2}, \quad \text {s . t .} y _ {i} \left\langle \boldsymbol {\gamma}, g \mathbf {x} _ {i} \right\rangle \geq 1, \forall g \in G, \forall i \in [ n ], \tag {123}
$$
$$
\boldsymbol {\beta} _ {\mathrm {f c}} ^ {\infty} \propto \boldsymbol {\gamma} _ {\mathrm {f c}} ^ {*} = \arg \min  _ {\boldsymbol {\gamma} \in \mathbb {R} ^ {d _ {0}}} \| \boldsymbol {\gamma} \| ^ {2}, \quad \text {s . t .} y _ {i} \left\langle \boldsymbol {\gamma}, \mathbf {x} _ {i} \right\rangle \geq 1, \forall i \in [ n ]. \tag {124}
$$
Moreover, the margin  $M_{\mathrm{steer}}$  of the steerable network  $\beta_{\mathrm{steer}}^{\infty}$  on the augmented data set  $S_{\mathrm{aug}}$  is  $M_{\mathrm{steer}} = \frac{1}{\|\gamma_{\mathrm{steer}}^{*}\|}$ . Consider the following three cases.
- Case 1:  $\min_{i\in [n],g\in G}y_i\langle \gamma_{\mathrm{fc}}^*,g\mathbf{x}_i\rangle \geq 1$ . In this case,  $\gamma_{\mathrm{fc}}^{*}$  also satisfies the more restrictive constraint in (123). Therefore  $\gamma_{\mathrm{fc}}^{*} = \gamma_{\mathrm{steer}}^{*}$ , which implies
$$
\beta_ {\text {s t e e r}} ^ {\infty} = \beta_ {\mathrm {f c}} ^ {\infty}, \quad \text {a n d} \quad M _ {\text {s t e e r}} = M _ {\mathrm {f c}}. \tag {125}
$$
- Case 2:  $\min_{i\in [n],g\in G}y_i\langle \gamma_{\mathrm{fc}}^*,g\mathbf{x}_i\rangle \leq 0$ . This implies that  $\gamma_{\mathrm{fc}}^{*}$ , and hence  $\beta_{\mathrm{fc}}^{\infty}$  has a nonpositive margin on  $D_{\mathrm{aug}}$ . Thus
$$
M _ {\mathrm {f c}} \leq 0 <   M _ {\text {s t e e r}}. \tag {126}
$$
- Case 3:  $0 < \min_{i\in [n],g\in G}y_{i}\left\langle \boldsymbol{\gamma}_{\mathrm{fc}}^{*},g\mathbf{x}_{i}\right\rangle < 1$ . In this case, define
$$
\widetilde {\gamma} _ {\mathrm {f c}} ^ {*} := \frac {\gamma_ {\mathrm {f c}} ^ {*}}{\min  _ {i \in [ n ] , g \in G} y _ {i} \left\langle \gamma_ {\mathrm {f c}} ^ {*} , g \mathbf {x} _ {i} \right\rangle}. \tag {127}
$$
Then  $\widetilde{\gamma}_{\mathrm{fc}}^{*}$  satisfies the constraint in (123), and hence
$$
\left\| \widetilde {\gamma} _ {\mathrm {f c}} ^ {*} \right\| \geq \left\| \gamma_ {\text {s t e e r}} ^ {*} \right\|. \tag {128}
$$
$$
\begin{array}{l} M _ {\mathrm {f c}} = \min  _ {i \in [ n ], g \in G} y _ {i} \left\langle \boldsymbol {\beta} _ {\mathrm {f c}} ^ {\infty}, g \mathbf {x} _ {i} \right\rangle = \frac {\min  _ {i \in [ n ] , g \in G} y _ {i} \left\langle \boldsymbol {\gamma} _ {\mathrm {f c}} ^ {*} , g \mathbf {x} _ {i} \right\rangle}{\left\| \boldsymbol {\gamma} _ {\mathrm {f c}} ^ {*} \right\|} = \frac {1}{\left\| \widetilde {\boldsymbol {\gamma}} _ {\mathrm {f c}} ^ {*} \right\|} (129) \\ \leq \frac {1}{\| \boldsymbol {\gamma} _ {\text {s t e e r}} ^ {*} \|} = M _ {\text {s t e e r}}. (130) \\ \end{array}
$$
F Proof of Theorem 6.3
To prove Theorem 6.3, we need first the following preliminaries.
Lemma F.1. If  $\mathcal{D}$  is  $G$ -invariant and linearly separable, then  $\mathcal{D}$  can be linearly separated by a  $G$ -invariant classifier. That is, there exists  $\beta \in \mathbb{R}_G^{d_0}$  such that
$$
\mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ y \langle \mathbf {x}, \boldsymbol {\beta} \rangle \geq 1 ] = 1. \tag {131}
$$
Proof. Since  $\mathcal{D}$  is linearly separable, there exists  $\beta_0\in \mathbb{R}^{d_0}$  such that  $\mathbb{P}_{(\mathbf{x},y)\sim \mathcal{D}}[y\langle \mathbf{x},\beta_0\rangle \geq 1] = 1$ . Let  $\beta = \frac{1}{|G|}\sum_{g\in G}g^{\top}\beta_{0}\in \mathbb{R}_{G}^{d_{0}}$ , and we aim to show that  $\mathcal{D}$  can also be linearly separated by  $\beta$ . Indeed, for all  $(\mathbf{x},y)\in \mathbb{R}^{d_0}\times \{\pm 1\}$ ,
$$
\begin{array}{l} y \langle \mathbf {x}, \boldsymbol {\beta} \rangle <   1 \Longrightarrow y \left\langle \mathbf {x}, \frac {1}{| G |} \sum_ {g \in G} g ^ {\top} \boldsymbol {\beta} _ {0} \right\rangle <   1 \Longrightarrow y \left\langle \frac {1}{| G |} \sum_ {g \in G} g \mathbf {x}, \boldsymbol {\beta} _ {0} \right\rangle <   1 (132) \\ \Longrightarrow \exists g \in G, \text {s . t .} y \langle g \mathbf {x}, \beta_ {0} \rangle <   1. (133) \\ \end{array}
$$
Let  $E = \{(\mathbf{x},y):y\langle \mathbf{x},\beta_0\rangle < 1\} \subset \mathbb{R}^{d_0}\times \{\pm 1\}$  , then
$$
\begin{array}{l} \mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ y \langle \mathbf {x}, \boldsymbol {\beta} \rangle <   1 ] \leq \sum_ {g \in G} \mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ y \langle g \mathbf {x}, \boldsymbol {\beta} _ {0} \rangle <   1 ] (134) \\ = \sum_ {g \in G} \mathcal {D} \left\{\left(\mathbf {x}, y\right): \left(\rho_ {0} (g) \otimes \operatorname {I d}\right) (\mathbf {x}, y) \in E \right\} (135) \\ = \sum_ {g \in G} \left(\rho_ {0} (g) \otimes \operatorname {I d}\right) _ {*} \mathcal {D} (E) = \sum_ {g \in G} \mathcal {D} (E) (136) \\ = \sum_ {g \in G} \mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} [ y \langle \mathbf {x}, \boldsymbol {\beta} _ {0} \rangle <   1 ] = 0, (137) \\ \end{array}
$$
where the second equality in (136) is due to  $\mathcal{D}$  being  $G$ -invariant. Therefore  $\mathcal{D}$  can be separated by  $\beta \in \mathbb{R}_G^{d_0}$ .
Definition F.2 (Empirical Rademacher complexity). Let  $\mathcal{F} \subset \mathbb{R}^{\mathcal{Z}}$  be a class of real-valued functions on  $\mathcal{Z}$ , and let  $S = (\mathbf{z}_1, \dots, \mathbf{z}_n)$  be a set of  $n$  samples from  $\mathcal{Z}$ . The (empirical) Rademacher complexity  $\widehat{\mathcal{R}}_n(\mathcal{F}, S)$  of  $\mathcal{F}$  over  $S$  is defined as
$$
\widehat {\mathcal {R}} _ {n} (\mathcal {F}, S) := \frac {1}{n} \mathbb {E} _ {\boldsymbol {\sigma} \sim \{\pm 1 \} ^ {n}} \left[ \sup  _ {f \in \mathcal {F}} \sum_ {i = 1} ^ {n} \sigma_ {i} f \left(\mathbf {z} _ {i}\right) \right], \tag {138}
$$
where  $\sigma = (\sigma_{1},\dots ,\sigma_{n})$  are the i.i.d. Rademacher variables satisfying  $\mathbb{P}[\sigma_i = 1] = \mathbb{P}[\sigma_i = -1] = 1 / 2$ .
Lemma F.3 (Talagrand's Contraction Lemma, Lemma 8 in Mohri and Medina [2014]). Let  $\Phi_i: \mathbb{R} \to \mathbb{R}$ ,  $i \in [n]$ , be 1-Lipschitz functions,  $\mathcal{F} \subset \mathbb{R}^{\mathcal{Z}}$  be a function class, and  $S = (\mathbf{z}_1, \dots, \mathbf{z}_n) \in \mathcal{Z}^n$  be a set of  $n$  samples from  $\mathcal{Z}$ . Then
$$
\widehat {\mathcal {R}} _ {n} (\mathcal {F}, S) \geq \frac {1}{n} \mathbb {E} _ {\boldsymbol {\sigma} \sim \{\pm 1 \} ^ {n}} \left[ \sup  _ {f \in \mathcal {F}} \sum_ {i = 1} ^ {n} \sigma_ {i} \Phi_ {i} \circ f (\mathbf {z} _ {i}) \right]. \tag {139}
$$
Lemma F.4. Let  $\mathcal{H} = \left\{\mathbf{x}\mapsto \langle \beta ,\mathbf{x}\rangle :\beta \in \mathbb{R}_G^{d_0},\| \beta \| \leq B\right\}$  be the function space of  $G$  -invariant linear functions of bounded norm, and let  $S_{\mathbf{x}} = \{\mathbf{x}_1,\dots ,\mathbf{x}_n\} \subset \mathbb{R}^{d_0}$  be a set of  $n$  samples from  $\mathbb{R}^{d_0}$ . Then
$$
\widehat {\mathcal {R}} _ {n} \left(\mathcal {H}, S _ {\mathbf {x}}\right) \leq \frac {B \max  _ {i \in [ n ]} \| \overline {{\mathbf {x}}} _ {i} \|}{\sqrt {n}} \tag {140}
$$
Proof. By the definition of empirical Rademacher complexity (138), we have
$$
\begin{array}{l} n \widehat {\mathcal {R}} _ {n} (\mathcal {H}, S _ {\mathbf {x}}) = \mathbb {E} _ {\boldsymbol {\sigma}} \left[ \sup  _ {\boldsymbol {\beta} \in \mathbb {R} _ {G} ^ {d _ {0}}, \| \boldsymbol {\beta} \| \leq B} \sum_ {i = 1} ^ {n} \sigma_ {i} \langle \boldsymbol {\beta}, \mathbf {x} _ {i} \rangle \right] (141) \\ = \mathbb {E} _ {\boldsymbol {\sigma}} \left[ \sup  _ {\boldsymbol {\beta} \in \mathbb {R} _ {G} ^ {d _ {0}}, \| \boldsymbol {\beta} \| \leq B} \sum_ {i = 1} ^ {n} \sigma_ {i} \langle \boldsymbol {\beta}, \overline {{\mathbf {x}}} _ {i} \rangle \right] (142) \\ = \mathbb {E} _ {\boldsymbol {\sigma}} \sup  _ {\boldsymbol {\beta} \in \mathbb {R} _ {G} ^ {d _ {0}}, \| \boldsymbol {\beta} \| \leq B} \left\langle \boldsymbol {\beta}, \sum_ {i = 1} ^ {n} \sigma_ {i} \bar {\mathbf {x}} _ {i} \right\rangle (143) \\ = B \mathbb {E} _ {\boldsymbol {\sigma}} \left\| \sum_ {i = 1} ^ {n} \sigma_ {i} \overline {{\mathbf {x}}} _ {i} \right\| (144) \\ \leq B \left(\mathbb {E} _ {\boldsymbol {\sigma}} \left\| \sum_ {i = 1} ^ {n} \sigma_ {i} \overline {{\mathbf {x}}} _ {i} \right\| ^ {2}\right) ^ {1 / 2}, (145) \\ \end{array}
$$
where (142) is due to  $\beta \in \mathbb{R}_G^{d_0}$ . Since  $\sigma_1, \dots, \sigma_n$  are i.i.d., we have
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\sigma}} \left\| \sum_ {i = 1} ^ {n} \sigma_ {i} \overline {{\mathbf {x}}} _ {i} \right\| ^ {2} = \mathbb {E} _ {\boldsymbol {\sigma}} \left[ \sum_ {i, j} \sigma_ {i} \sigma_ {j} \langle \overline {{\mathbf {x}}} _ {i}, \overline {{\mathbf {x}}} _ {j} \rangle \right] (146) \\ = \sum_ {i \neq j} \left\langle \overline {{\mathbf {x}}} _ {i}, \overline {{\mathbf {x}}} _ {j} \right\rangle \mathbb {E} _ {\boldsymbol {\sigma}} \sigma_ {i} \sigma_ {j} + \sum_ {i = 1} ^ {n} \left\| \overline {{\mathbf {x}}} _ {i} \right\| ^ {2} \mathbb {E} _ {\boldsymbol {\sigma}} \sigma_ {i} ^ {2} (147) \\ = \sum_ {i = 1} ^ {n} \| \overline {{\mathbf {x}}} _ {i} \| ^ {2} \leq n \max  _ {i \in [ n ]} \| \overline {{\mathbf {x}}} _ {i} \| ^ {2}. (148) \\ \end{array}
$$
Eq. (145) combined with (148) completes the proof.
We also need the following standard result on the generalization bound based on the Rademacher complexity [Shalev-Shwartz and Ben-David, 2014].
Lemma F.5. Let  $\mathcal{H}$  be a set of hypotheses,  $\mathcal{Z} = \mathbb{R}^{d_0} \times \{\pm 1\}$  be the set of labeled samples, and
$$
\ell : \mathcal {H} \times \mathcal {Z} \rightarrow [ 0, \infty), \quad (h, \mathbf {z}) \mapsto \ell (h, \mathbf {z}), \tag {149}
$$
be a loss function satisfying  $0 \leq \ell(h, \mathbf{z}) \leq c$  for all  $\mathbf{z} \in \mathcal{Z}$  and  $h \in \mathcal{H}$ . Define the function class  $\mathcal{F} = \ell(\mathcal{H}, \cdot) \coloneqq \{\mathbf{z} \mapsto \ell(h, \mathbf{z}) : h \in \mathcal{H}\}$ . Then, for any  $\delta > 0$  and any distribution  $\mathcal{D}$  on  $\mathcal{Z}$ , we have with probability at least  $1 - \delta$  over i.i.d. samples  $S = (\mathbf{z}_1, \dots, \mathbf{z}_n) \sim \mathcal{D}^n$  that
$$
\sup  _ {h \in \mathcal {H}} \mathcal {L} _ {\mathcal {D}} (h) - \mathcal {L} _ {S} (h) \leq 2 \mathcal {R} _ {n} (\mathcal {F}) + c \sqrt {\frac {\log (1 / \delta)}{2 n}}, \tag {150}
$$