SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $X$  be an  $n$ -dimensional space, Definition 4.2 (Equation 2) defines  $k(Gx)$  as an integral over  $Gx \subseteq X$ , which is a  $m$ -dimensional sub-manifold of  $X$ . In Theorem 4.3, Equation 4 rewrites the error function (Equation 1) as an iterated integral over the orbit  $Gx$  and then the fundamental domain  $F$  using Definition 4.1. In the discrete group case,  $m$  would be 0, Equation 2 is an integral of a 0-form in a 0-manifold, which is a sum:
$$
k (G x) = \min  _ {y \in Y} \sum_ {z \in G x} p (z) \mathbb {1} (f (z) \neq y) = \min  _ {y \in Y} \sum_ {g \in G} p (g x) \mathbb {1} (f (g x) \neq y) \tag {8}
$$
C Proof of Proposition 4.7
Proof. Consider the integral of probability density inside  $Gx$ , for a given  $y$ , it can be separated into two groups:
$$
\begin{array}{l} \int_ {G x} p (z) d z = \int_ {G x} p (z) \mathbb {1} (f (z) = y) d z \\ + \int_ {G x} p (z) \mathbb {1} (f (z) \neq y) d z. \\ \end{array}
$$
We can then rewrite  $k(Gx)$  in Equation 2 as:
$$
k (G x) = \min  _ {y \in Y} \left[ \int_ {G x} p (z) d z - \int_ {G x} p (z) \mathbb {1} (f (z) = y) d z \right]. \tag {9}
$$
Letting  $(Gx)_y = \{x' \in Gx \mid f(x') = y\} = f^{-1}(y) \cap Gx$ , Equation 9 can be written as:
$$
k (G x) = \min  _ {y \in Y} \left[ \int_ {G x} p (z) d z - \int_ {(G x) _ {y}} p (z) d z \right]
$$
$$
\int_ {G x} p (z) d z - \max _ {y \in Y} \int_ {(G x) _ {y}} p (z) d z.
$$
Theorem 4.3 can be rewritten as:
$$
\begin{array}{l} \operatorname {e r r} (h) \geq \int_ {F} \Big (\int_ {G x} p (z) d z - \max _ {y \in Y} \int_ {(G x) _ {y}} p (z) d z \Big) d x \\ \geq \int_ {F} \int_ {G x} p (z) d z - \int_ {F} \max  _ {y \in Y} \int_ {(G x) _ {y}} p (z) d z \\ \geq 1 - \int_ {F} \max  _ {y \in Y} | (G x) _ {y} | p (x) d x. \tag {10} \\ \end{array}
$$
The first term in Equation 10 uses the fact that  $X = \bigcup_{x \in F} Gx$  so the integral of the probability of the orbits of all points in the fundamental domain is the integral of the probability of the input domain  $X$  which is 1. The second term of Equation 10 uses  $p(gx) = p(x)$  so the integration of  $p(z)$  on  $(Gx)_y$  becomes  $p(x)$  times the range of the limit which is the size of  $(Gx)_y, |(Gx)_y|$ .
Now consider a partition of  $F = \coprod_{q} F_{q}$  where  $F_{q} = \{x \in F : (\max_{y \in Y} |(Gx)_{y}|) / |Gx| = q\}$ . We can rewrite Equation 10 as:
$$
\operatorname {e r r} (h) \geq 1 - \int_ {F} q | G x | p (x) d x \tag {11}
$$
$$
\geq 1 - \sum_ {q} \int_ {F _ {q}} q | G x | p (x) d x \tag {12}
$$
$$
\geq 1 - \sum_ {q} q \int_ {F _ {q}} | G x | p (x) d x. \tag {13}
$$
Equation 11 uses the definition of  $q$ . Equation 12 separates the integral over  $F$  into the partition of  $F$ . Equation 13 moves  $q$  out from the integral because it is a constant inside the integral. Consider the definition of  $c_{q}$ , we have:
$$
\begin{array}{l} c _ {q} = \mathbb {P} (x \in X _ {q}) \\ = \int_ {X _ {q}} p (x) d x \\ = \int_ {F _ {q}} \int_ {G x} p (z) d z d x (14) \\ = \int_ {F _ {q}} | G x | p (x) d x. (15) \\ \end{array}
$$
Equation 14 uses  $X_{q} = \bigcup_{x\in F_{q}}Gx$ . Equation 15 uses  $p(x) = p(gx)$ . Now we can write Equation 13 as:
$$
\operatorname {e r r} (h) \geq 1 - \sum_ {q} q c _ {q}.
$$
D Proof of Theorem 4.9
Define  $q(gx)\in \mathbb{R}^{n\times n}$  such that
$$
Q _ {G x} q (g x) = p (g x) \rho_ {Y} (g) ^ {T} \rho_ {Y} (g) \alpha (x, g). \tag {16}
$$
In particular,  $q(gx)$  exists when  $Q_{Gx}$  is full rank. It follows that  $Q_{Gx}\int_{G}q(gx)dg = Q_{Gx}$ . Moreover,  $Q_{Gx}$  and  $q(gx)$  are symmetric matrix.
Proof. The error function (Equation 1) can be written
$$
\begin{array}{l} \operatorname {e r r} (h) = \mathbb {E} _ {x \sim p} [ | | f (x) - h (x) | | _ {2} ^ {2} ] \\ = \int_ {X} p (x) | | f (x) - h (x) | | _ {2} ^ {2} d x \\ = \int_ {x \in F} \int_ {g \in G} p (g x) | | f (g x) - h (g x) | | _ {2} ^ {2} \alpha (x, g) d g d x. \\ \end{array}
$$
Denote  $e(x) = \int_{G} p(gx) ||f(gx) - h(gx)||_2^2 \alpha(x, g) dg$ . Since  $h$  is  $G$ -equivariant, for each  $x \in F$  the value  $c = h(x) \in \mathbb{R}^n$  of  $h$  at  $x$  determines the value of  $h$  across the whole orbit  $h(gx) = gh(x) = gc$  for  $g \in G$ . Then  $e(x)$  can be written
$$
\begin{array}{l} e (x) = \int_ {G} p (g x) | | f (g x) - g c | | _ {2} ^ {2} \alpha (x, g) d g \\ = \int_ {G} p (g x) | | g (g ^ {- 1} f (g x) - c) | | _ {2} ^ {2} \alpha (x, g) d g \\ = \int_ {G} (g ^ {- 1} f (g x) - c) ^ {T} p (g x) g ^ {T} g \alpha (x, g) (g ^ {- 1} f (g x) - c) d g \\ = \int_ {G} (g ^ {- 1} f (g x) - c) ^ {T} Q _ {G x} q (g x) (g ^ {- 1} f (g x) - c) d g. \tag {17} \\ \end{array}
$$
Taking the derivative of  $e(x)$  with respect to  $c$  we have
$$
\begin{array}{l} \frac {\partial e (x)}{\partial c} = \int_ {G} \left(\left(Q _ {G x} q (g x)\right) ^ {T} + \left(Q _ {G x} q (g x)\right)\right) \left(c - g ^ {- 1} f (g x)\right) d g \\ = \int_ {G} 2 Q _ {G x} q (g x) \left(c - g ^ {- 1} f (g x)\right) d g. \\ \end{array}
$$
Setting  $\partial e(x) / \partial c = 0$  we can find an equation for  $c^*$  which minimizes  $e(x)$
$$
\begin{array}{l} Q _ {G x} \int_ {G} q (g x) d g \cdot c ^ {*} = Q _ {G x} \int_ {G} q (g x) g ^ {- 1} f (g x) d g \\ Q _ {G x} c ^ {*} = Q _ {G x} \mathbf {E} _ {G} [ f, x ]. \tag {18} \\ \end{array}
$$
Substituting  $c^*$  into Equation 17 we have
$$
\begin{array}{l} e (x) \geq \int_ {G} \left(g ^ {- 1} f (g x) - c ^ {*}\right) ^ {T} Q _ {G x} q (g x) \left(g ^ {- 1} f (g x) - c ^ {*}\right) d g \\ = \int_ {G} (g ^ {- 1} f (g x)) ^ {T} Q _ {G x} q (g x) (g ^ {- 1} f (g x)) \\ - \left(c ^ {* T} Q _ {G x} q (g x) g ^ {- 1} f (g x)\right) ^ {T} \tag {19} \\ - c ^ {* T} Q _ {G x} q (g x) g ^ {- 1} f (g x) \\ + c ^ {* T} Q _ {G x} q (g x) c ^ {*} d g. \\ \end{array}
$$
The term  $\int_{G}c^{*T}Q_{Gx}q(gx)g^{-1}f(gx)dg$  could be simplified as
$$
\int_ {G} c ^ {* T} Q _ {G x} q (g x) g ^ {- 1} f (g x) d g = \int_ {G} \mathbf {E} _ {G} [ f, x ] Q _ {G x} q (g x) g ^ {- 1} f (g x) d g. \tag {20}
$$
Notice that  $Q_{Gx}$ , and  $q(gx)$  are symmetric matrix
$$
\begin{array}{l} \int_ {G} c ^ {* T} Q _ {G x} q (g x) c ^ {*} d g = \int_ {G} c ^ {* T} q (g x) Q _ {G x} c ^ {*} d g \\ = \int_ {G} \mathbf {E} _ {G} ^ {T} [ f, x ] Q _ {G x} q (g x) \mathbf {E} _ {G} [ f, x ] d g. \\ \end{array}
$$
Thus Equation 19 becomes
$$
\begin{array}{l} e (x) \geq \int_ {G} \left(g ^ {- 1} f (g x)\right) ^ {T} Q _ {G x} q (g x) \left(g ^ {- 1} f (g x)\right) \\ - \left(\mathbf {E} _ {G} ^ {T} [ f, x ] Q _ {G x} q (g x) g ^ {- 1} f (g x)\right) ^ {T} \\ - \mathbf {E} _ {G} ^ {T} [ f, x ] Q _ {G x} q (g x) g ^ {- 1} f (g x) \\ + \mathbf {E} _ {G} ^ {T} [ f, x ] Q _ {G x} q (g x) \mathbf {E} _ {G} [ f, x ] d g \\ = \int_ {G} p (g x) | | f (g x) - g \mathbf {E} _ {G} [ f, x ] | | _ {2} ^ {2} \alpha (x, g) d g. \\ \end{array}
$$
Taking the integral over the fundamental domain  $F$  we have
$$
\begin{array}{l} \operatorname {e r r} (h) = \int_ {F} e (x) \\ \geq \int_ {F} \int_ {G} p (g x) | | f (g x) - g \mathbf {E} _ {G} [ f, x ] | | _ {2} ^ {2} \alpha (x, g) d g d x. \tag {21} \\ \end{array}
$$
E Proof of Corollary 4.10
Proof. When  $\rho_{Y}$  is an orthogonal representation, we have  $\rho_{Y}(g)^{T}\rho_{Y}(g) = I_{n}$ , i.e., the identity matrix. Then  $q(gx)$  can be written as  $q(gx) = s(gx)\mathrm{Id}$  where  $s(gx)$  is a scalar. Since  $\int_{G}q(gx)dg = \mathrm{Id}$ , we can re-define  $q(gx)$  to drop  $\mathrm{Id}$  and only keep the scalar, then  $q_{x}(g)$  can be viewed as a probability density function of  $g$  because now  $\int_{G}q_{x}(g) = 1$ .
With  $q_{x}(g)$  being the probability density function,  $\mathbf{E}_G[f,x]$  (Equation 6) naturally becomes the mean  $\mathbb{E}_G[f_x]$  where  $g\sim q_{x}$ .
Now consider  $e(x) = \int_{G} p(gx) || f(gx) - g\mathbf{E}_{G}[f_{x}]||_{2}^{2}\alpha (x,g)dg$  in Theorem 4.9, it can be written as
$$
\begin{array}{l} e (x) = \int_ {G} p (g x) | | f (g x) - g \mathbb {E} _ {G} [ f _ {x} ] | | _ {2} ^ {2} \alpha (x, g) d g \\ = \int_ {G} p (g x) | | g \left(g ^ {- 1} f (g x) - \mathbb {E} _ {G} [ f _ {x} ]\right) | | _ {2} ^ {2} \alpha (x, g) d g \\ = \int_ {G} p (g x) \left(g ^ {- 1} f (g x) - \mathbb {E} _ {G} [ f _ {x} ]\right) ^ {T} \rho_ {Y} (g) ^ {T} \rho_ {Y} (g) \left(g ^ {- 1} f (g x) - \mathbb {E} _ {G} [ f _ {x} ]\right) \alpha (x, g) d g. \\ \end{array}
$$
Since  $\rho_Y(g)^T\rho_Y(g) = I_n$ , we have
$$
\begin{array}{l} e (x) = \int_ {G} p (g x) \left(g ^ {- 1} f (g x) - \mathbb {E} _ {G} [ f _ {x} ]\right) ^ {T} \left(g ^ {- 1} f (g x) - \mathbb {E} _ {G} [ f _ {x} ]\right) \alpha (x, g) d g \\ = \int_ {G} p (g x) | | g ^ {- 1} f (g x) - \mathbb {E} _ {G} [ f _ {x} ] | | _ {2} ^ {2} \alpha (x, g) d g. \tag {22} \\ \end{array}
$$
From Equation 5 we have  $p(gx)\alpha(x, g) = Q_{Gx}q(gx)$ . Substituting in Equation 22 we have
$$
e (x) = \int_ {G} Q _ {G x} q (g x) | | g ^ {- 1} f (g x) - \mathbb {E} _ {G} [ f _ {x} ] | | _ {2} ^ {2} d g.
$$
Since  $Q_{Gx} = \int_G p(gx)\alpha (a,g)dg$  when  $\rho_Y(g)^T\rho_Y(g) = I_n$ , we have
$$
\begin{array}{l} e (x) = Q _ {G x} \int_ {G} q _ {x} (g) | | g ^ {- 1} f (g x) - \mathbb {E} _ {G} [ f _ {x} ] | | _ {2} ^ {2} d g \\ = Q _ {G x} \mathbb {V} _ {G} [ f _ {x} ]. \tag {23} \\ \end{array}
$$
Now consider  $Q_{Gx}$  (Equation 5), when  $\rho_Y(g)^T\rho_Y(g) = I_n$ , it can be written
$$
\begin{array}{l} Q _ {G x} = \int_ {G} p (g x) \alpha (x, g) d g \\ = \int_ {G x} p (z) d z \\ = p (G x). \\ \end{array}
$$
Replacing  $Q_{Gx}$  with  $p(Gx)$  in Equation 23 then taking the integral of  $e(x)$  over the fundamental domain gives the result.
F Lower Bound of Equivariant Regression when  $\rho_{Y} = \mathrm{Id}$
Proposition F.1. When  $\rho_{Y} = \mathrm{Id}$ , the error of  $h$  has lower bound  $\mathrm{err}(h) \geq \int_{F} p(Gx) \mathbb{V}_{Gx}[f] dx$ , which is the same as Theorem 4.8.
Proof. Consider Equation 5, when  $\rho_Y(g) = \mathrm{Id}$ , we have
$$
Q _ {G x} = \int_ {G} p (g x) \alpha (x, g) d g.
$$
Exchange the integration variable using  $z = gx$  we have
$$
Q _ {G x} = \int_ {G x} p (z) d z. \tag {24}
$$
Consider  $\mathbb{E}_G[f_x] = \int_G q_x(g)g^{-1}f(gx)dg$ . When  $\rho_Y(g) = \mathrm{Id}$ , it becomes
$$
\mathbb {E} _ {G} [ f _ {x} ] = \int_ {G} q (g x) f (g x) d g.
$$
Substituting  $q(gx)$  with Equation 5, considering  $\rho_{Y}(g) = \mathrm{Id}$ , we have
$$
\mathbb {E} _ {G} [ f _ {x} ] = \int_ {G} Q _ {G x} ^ {- 1} p (g x) f (g x) \alpha (x, g) d g.
$$
Exchange the integration variable using  $z = g x$  we have
$$
\mathbb {E} _ {G} [ f _ {x} ] = \int_ {G x} Q _ {G x} ^ {- 1} p (z) f (z) d z.
$$
Substituting Equation 24 we have
$$
\begin{array}{l} \mathbb {E} _ {G} [ f _ {x} ] = \int_ {G x} \frac {p (z)}{\int_ {G x} p (z) d z} f (z) d z \\ = \mathbb {E} _ {G x} [ f ]. \\ \end{array}
$$
Similarly, we can proof  $\mathbb{V}_G[f_x] = \mathbb{V}_{Gx}[f]$ , thus when  $\rho_Y = \mathrm{Id}$ , Corollary 4.10 is Theorem 4.8.
Non-equivariant model class  $C_2$ -equivariant model class
σT	supfn∈FN	1/m	∑i=1mσifn(xi)	supfn∈FN	1/m	∑i=1mσifn(xi)
[-1,-1,-1,-1]	1	1
[-1,-1,-1,+1]	1	1
[-1,-1,+1,-1]	1	1
[-1,-1,+1,+1]	1	0.75
[-1,+1,-1,-1]	1	1
[-1,+1,-1,+1]	1	1
[-1,+1,+1,-1]	1	1
[-1,+1,+1,+1]	1	1
[+1,-1,-1,-1]	1	1
[+1,-1,-1,+1]	1	1
[+1,-1,+1,+1]	1	1
[+1,+1,-1,-1]	1	0.75
[+1,+1,-1,+1]	1	1
[+1,+1,-1,+1]	1	1
[+1,+1,+1,+1]	1	1
R_S	1	31/32
(a) Correct
(b) Incorrect
(c) Extrinsic
Figure 11: The correct, incorrect, and extrinsic data distribution in the Swiss Roll experiment.
(d) Extrinsic Invariant View
G Rademacher Complexity of Harmful Extrinsic Equivariance Example
Let  $S = \{x^{1}, x^{2}, x^{3}, x^{4}\}$ , where the labels are  $y^{1}, y^{2} = +1$  and  $y^{3}, y^{4} = -1$ . We consider two model classes  $\mathcal{F}_N$ , the set of all linear models, and  $\mathcal{F}_E$ , the set of all linear models equivariant to  $C_2$ , and compute their empirical Rademacher complexity on  $S$ .
For the data  $S$ , an extrinsically equivariant linear model class has lower empirical Rademacher complexity than its unconstrained linear counterpart, demonstrating that extrinsic equivariance can be harmful to learning.
H Additional Experiments
H.1 Swiss Roll Experiment
Figure 11 and Figure 12 show the actual data distribution for the Swiss Roll experiment in Section 6.1. In the incorrect distribution, the data in the two  $z$  planes form two spirals with different labels but the same shape. The equivariance is incorrect because if we translate one spiral to the other spiral's plane, they will overlap but their labels are different. In the correct distribution, there are two different 'dashed' spirals copied into two  $z$ -planes. The equivariance is correct because after a  $z$ -translation, both the data and their labels exactly overlap. In all three cases, we assume the data has a uniform distribution. Figure 13b shows the ternary plot of MLP for all different  $c$ ,  $ir$ ,  $er$ , where the performance of MLP decreases as the correct ratio increases. Figure 13a shows an inverse trend: the
(a)  $c = .5, i = .5, e = 0$
(b)  $c = .5, i = 0, e = .5$
(c)  $c = 0,i = .5,e = .5$
(d)  $c = .5,i = .25,e = .25$
Figure 12: Data distribution example with different correct ratio  $(c)$ , incorrect ratio  $(ir)$ , and extrinsic ratio  $(er)$  values.
(a)
(b)
Figure 13: The ternary plot of the invariant network (a) and unconstrained network (b) with different correct, incorrect, and extrinsic ratio.
performance of INV increases as the correct ratio increases. Moreover, both extrinsic and incorrect equivariance harms the performance of INV, but incorrect equivariance is more devastating because the error is limited by a theoretical lower bound.
H.2 Square Experiment
We consider the environment shown in Example 4.6. We vary  $m \in \{0.2, 0.4, 0.6, 0.8, 1\}$  and  $c \in \{0, 0.2, 0.4, 0.6, 0.8, 1\}$ . We train an  $u$ -invariant network and evaluate its test performance with the theoretical lower bound  $\mathrm{err}(h) \geq (1 - c) \times (1 - m)$ . Figure 14 shows the test error of the trained network compared with the theoretical lower bound. The highest difference is below  $3\%$ , demonstrating the correctness of our theory.
H.3 Regression Experiment
In this experiment, we validate our theoretical error lower bound for invariant and equivariant regression (Theorem 4.8, 4.9) in an environment similar to Example 4.11. Consider a regression task \( f: \mathbb{R} \times \mathcal{X} \to \mathbb{R}^2 \) given by \( (\theta, x) \mapsto y \), where \( \mathcal{X} = \{x_0, x_1, x_2, x_3\} \). The group \( g \in G = C_4 = \{e, r, r^2, r^3\} \) acts on \( (\theta, x) \) by \( g(\theta, x) = (\theta, gx) \) through permutation: \( x_1 = rx_0; x_2 = rx_1; x_3 =
Invariant Network	Equivariant Network
Empirical/Theoretical	1.002 ±0.000	1.001 ±0.000
Table 2: Empirical  $\operatorname{err}\left( h\right)$  divided by theoretical  $\operatorname{err}\left( h\right)$  for invariant regression and equivariant regression. Results are averaged over 100 runs with different  $f$  for each regression. Empirical regression error matches theoretical error.
Figure 14: Result of the square experiment in terms of the  $L_{1}$  distance between the network error and the theoretical lower bound in percentage. Each cell corresponds to an experiment with a particular correct ratio (c) and majority label ratio (m). Results are averaged over 10 runs.
$rx_{2};x_{0} = rx_{3}$ . Let  $r^k\in G$  acts on  $y$  by  $\rho_Y(g) = \left( \begin{array}{cc}\cos g & -\sin g\\ \sin g & \cos g \end{array} \right)$  where  $g = k\pi /2$ . Note that fixing a single value of  $\theta$  gives Example 4.11; in other words, this experiment has infinitely many orbits where each orbit is similar to Example 4.11.
We generate random polynomial function  $f$  that is not equivariant, i.e.,  $\exists (\theta, x)$  s.t.  $g \cdot f(\theta, x) \neq \rho_Y(g)y$ . Then we try to fit  $f$  using a  $G$ -invariant network and a  $G$ -equivariant network. We measure their error compared with the theoretical lower bound given by Theorem 4.8 and 4.9. As is shown in Table 2, both the invariant network and the equivariant network achieve an error rate nearly the same as our theoretical bound. The empirical error is slightly higher than the theoretical error due to the neural network fitting error. Please refer to I.4 for more experiment details.
I Experiment Details
This section describes the details of our experiments. All of the experiment is performed using a single Nvidia RTX 2080 Ti graphic card.
I.1 Swiss Roll Experiment
In the Swiss Roll Experiment in Section 6.1, we use a three-layer MLP for the unconstrained network. For the  $z$ -invariant network, we use a network with two DSS [37] layers to implement the  $z$ -invariance, each containing two FC layers. We train the networks using the Adam [24] optimizer with a learning rate of  $10^{-3}$ . The batch size is 128. In each run, there are 200 training data, 200 validation data, and 200 test data randomly sampled from the data distribution. The network is trained for a minimal of 1000 epochs and a maximum of 10000 epochs, where the training is terminated after there is no improvement in the classification success rate in the validation set for a consecutive of 1000 epochs. We report the test success rate of the epoch model with the highest validation success rate.
I.2 Square Experiment
In the Square Experiment in Section H.2, we use a network with two DSS [37] layers to implement the horizontal invariance, where each layer contains two FC layers. We train the networks using the Adam [24] optimizer with a learning rate of  $10^{-3}$ . The batch size is 128. In each run, there are 1000 training data, 200 validation data, and 200 test data randomly sampled from the data distribution. The network is trained for a minimal of 1000 epochs and a maximum of 10000 epochs, where the training is terminated after there is no improvement in the classification success rate in the validation set for a consecutive of 1000 epochs. We report the test success rate of the epoch model with the highest validation success rate.
I.3 Digit Classification Experiment
In the Digit Classification Experiment in Section 6.2, we use two similar five-layer convolutional networks for the  $D_{4}$ -invariant network and the CNN, where the  $D_{4}$ -invariant network is implemented using the e2cnn package [60]. Both networks have the similar amount of trainable parameters. We train the networks using the Adam [24] optimizer with a learning rate of  $5 \times 10^{-5}$  and weight decay
Figure 15: The robotic experiment setup and the  $D_{1}$ -equivariant policy network.
of  $10^{-5}$ . The batch size is 256. In each run, there are 5000 training data, 1000 validation data, and 1000 test data randomly sampled from the data distribution. The network is trained for a minimal of 50 epochs and a maximum of 1000 epochs, where the training is terminated after there is no improvement in the classification success rate in the validation set for a consecutive of 50 epochs. We report the test success rate of the epoch model with the highest validation success rate.
I.4 Regression Experiment
In the regression experiment, we validate our theoretical error lower bound for invariant and equivariant regression (Theorem 4.8, 4.9) by comparing empirical network fitting error and the theoretical fitting error of a function  $f$ . Specifically, the function  $f$  maps a distance  $\theta$  and an index  $x$  pair to a vector  $y$ :
$$
f: \mathbb {R} \times \mathcal {X} \rightarrow \mathbb {R} ^ {2}, \text {g i v e n b y} (\theta , x) \mapsto y \tag {25}
$$
where  $\mathcal{X} = \{x_0, x_1, x_2, x_3\}$ . The group  $g \in G = C_4 = \{e, r, r^2, r^3\}$  acts on  $(\theta, x)$  by  $g(\theta, x) = (\theta, gx)$  through permuting the index  $x: x_1 = rx_0; x_2 = rx_1; x_3 = rx_2; x_0 = rx_3$ . Let  $r^k \in G$  acts on vector  $y$  by rotation  $\rho_Y(g) = \begin{pmatrix} \cos g & -\sin g \\ \sin g & \cos g \end{pmatrix}$  where  $g = k\pi / 2$ .
We construct function  $f$  in the following way: for each  $x \in \mathcal{X}$ , choose  $l_x: \mathbb{R} \to \mathbb{R}^2$  and define  $f(\theta, x) = l_x(\theta)$ . Notice that when  $l_{gx} = \rho_Y(g)l_x(\theta)$ ,  $f$  is  $G$ -equivariant. We define  $l_x(\theta) = (p_x(\theta), q_x(\theta))$  where  $p_x$  and  $q_x$  are cubic polynomials of  $x$ , i.e.,  $p_x$  with coefficients  $a, b, c, d$  will be  $p_x = ax^3 + bx^2 + cx + d$ . We choose  $p_x$  and  $q_x$  with different coefficients for each  $x$  such that  $f$  is not equivariant, i.e.,  $l_{gx} \neq \rho_Y(g)l_x(\theta)$ . For each run, we generate a function  $f$ , sample data  $\theta, x$ , and evaluate the data obtaining  $y$ . Then we train neural networks using L2 loss till converge. Eventually, we sample another set of data to evaluate the empirical L2 error as well as the theoretical L2 error.
I.5 Robotic Experiment
In the robotic manipulation experiment, the state  $s$  is defined as a top-down RGBD image of the workspace centered at the gripper's position (Figure 15 middle). The action  $a = (x,y,z,\theta ,\lambda)$  is defined as the change of position  $(x,y,z)$  and top-down orientation  $(\theta)$  of the gripper, with the
gripper open width  $(\lambda)$ . For a  $D_{1} = \{1, g\}$  group where  $g$  represents a horizontal flip, the group action on the state space  $gs$  is defined as flipping the image; the group action on the action space  $ga$  is defined as flipping the  $y$  and  $\theta$  action and leaving the other action components unchanged,  $ga = (x, -y, z, -\theta, \lambda)$ . We define a  $D_{1}$ -equivariant policy network  $\pi: s \mapsto a$  using e2cnn [60], where the output action of  $\pi$  will flip accordingly when the input image is flipped (Figure 15 bottom). We train the network using the Adam [24] optimizer with a learning rate of  $10^{-3}$  and weight decay of  $10^{-5}$ . For each run, we train the network for a total of 20k training steps, where we perform evaluation for 100 episodes every 2k training steps. We report the highest success rate of the 10 evaluations as the result of the run.
We develop the experimental environments in the PyBullet [12] simulator, based on the BullatArm benchmark [57]. In the Stacking (correct equivariance) and Sorting (incorrect equivariance) experiment, we gather a total of 400 episodes of demonstrations, where  $400c$  of them are Stacking and the rest  $400(1 - c)$  are Sorting. In evaluation, the task follows the same distribution, where  $100c$  of the evaluation episodes are Stacking and the rest are Sorting. Notice that the agent can distinguish the Stacking and Sorting tasks because the object colors are different for the two tasks (green and blue for stacking, yellow and red for sorting). In the Sorting (extrinsic equivariance) experiment, we also use 400 episodes of demonstrations.
Specifically, in Sorting, the cube and the triangle are initially placed randomly, within a distance of  $\pm 1.5cm$  from the horizontal mid-line of the workspace. The objective is to push the triangle at least  $9cm$  toward left and to push the cube at least  $9cm$  toward right, while ensuring that both objects remain within the boundaries of workspace. In Stacking, two blocks are randomly initialized on the floor of the workspace. The goal is to pick up the triangle and place it on top of the cube. The workspace has a size of  $30cm \times 30cm \times 25cm$ .