SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\alpha_ {Y} (i, y) = \phi \left(\alpha_ {X} (i, \phi^ {- 1} (y))\right) = \phi \left(\phi^ {- 1} (y)\right) = y
$$
$$
\begin{array}{l} \alpha_ {Y} (h g, y) = \phi (\alpha_ {X} (h g, \phi^ {- 1} (y))) \\ = \phi \left(\alpha_ {X} \left(h, \alpha_ {X} (g, \phi^ {- 1} (y))\right)\right) \\ = \phi \left(\alpha_ {X} \left(h, \phi^ {- 1} \circ \phi \left(\alpha_ {X} \left(g, \phi^ {- 1} (y)\right)\right)\right)\right) \\ = \alpha_ {Y} (h, \phi (\alpha_ {X} (g, \phi^ {- 1} (y)))) \\ = \alpha_ {Y} (h, \alpha_ {Y} (g, y)). \\ \end{array}
$$
We can now state a proposition telling us that if a composite function is equivariant, then it must be a composite of equivariant functions.
Proposition D.4. Assume that we are given a group  $G$  and two sets  $X$  and  $Z$  with group actions  $\alpha_{X}$  and  $\alpha_{Z}$ . Assume also that we are given a further set  $Y$  and two functions  $\phi : X \to Y$  and  $\psi : Y \to Z$  such that their composition  $\psi \circ \phi : X \to Z$  is  $G$ -equivariant w.r.t.  $\alpha_{X}$  and  $\alpha_{Z}$ .
1. If  $\psi$  is invertible we can define a group action  $\alpha_{Y}$  on  $Y$  such that  $\phi$  and  $\psi$  are  $G$ -equivariant with respect to this action.
2. If  $\psi$  is not invertible, introduce  $Y' = \phi(X) / \sim$  where  $\sim$  is the equivalence relation identifying elements of  $Y$  that map to the same element of  $Z$ , i.e.,
$$
y _ {1} \sim y _ {2}: \Longleftrightarrow \psi (y _ {1}) = \psi (y _ {2}).
$$
Then we can define a group action  $\alpha_{Y'}$  on  $Y'$  such that  $\phi$  and  $\psi$  are equivariant when seen as functions to/from  $Y'$ .
Proof. Assume first that  $\psi$  is invertible. From Lemma D.3 we get a group action  $\alpha_{Y}$  on  $Y$  defined by
$$
\alpha_ {Y} (g, y) = \psi^ {- 1} (\alpha_ {Z} (g, \psi (y))).
$$
Lemma D.3 also states that  $\psi$  is equivariant with this choice of  $\alpha_{Y}$ . Next we show that  $\phi$  is equivariant, which follows from the fact that  $\psi \circ \phi$  is equivariant:
$$
\begin{array}{l} \alpha_ {Y} (g, \phi (x)) = \psi^ {- 1} \left(\alpha_ {Z} (g, \psi \circ \phi (x))\right) \\ = \psi^ {- 1} (\psi \circ \phi (\alpha_ {X} (g, x))) \\ = \phi (\alpha_ {X} (g, x)). \\ \end{array}
$$
If  $\psi$  is not invertible, then  $\psi$  is anyway invertible as a map  $\psi : Y' \to \psi \circ \phi(X)$ . Note that the action  $\alpha_Z$  is well defined even when restricting from  $Z$  to  $\psi \circ \phi(X)$ , because by the equivariance of  $\psi \circ \phi$ ,  $\alpha_Z$  can't move an element from  $\psi \circ \phi(X)$  to  $Z \setminus \psi \circ \phi(X)$ :
$$
\alpha_ {Z} (g, \psi \circ \phi (x)) = \psi \circ \phi (\alpha_ {X} (g, x)) \in \psi \circ \phi (X).
$$
Hence the earlier argument for constructing  $\alpha_{Y}$  works for constructing a group action  $\alpha_{Y'}$  on  $Y'$  such that the maps  $\phi$  and  $\psi$  (appropriately restricted) become equivariant.
Looking back at Example 2.1, with  $\phi = W_{1}$ ,  $\psi = W_{2} \mathrm{ReLU}(\cdot)$ , we see that  $Y'$  in that case would be a single point space since  $W_{2}$  maps everything to 0. The action  $\alpha_{Y'}$  would hence be trivial.
As mentioned before, the main problem with changing  $Y$  to  $Y'$  as above is that if  $X, Y, Z$  are feature spaces in a neural network  $\psi \circ \phi$ , where  $\phi$  and  $\psi$  themselves might be decomposable into multiple linear layers and activation functions, there is no guarantee that  $Y'$  will be a vector space or that  $\phi$  and  $\psi$  can still be expressed as decompositions into linear layers and activation functions when changing from  $Y$  to  $Y'$ . Still, the intuition that the equivariance should be preserved layerwise holds in the interpretation provided by Proposition D.4.
E Network layers with bias
In this section we generalize Proposition 2.3 to work for layers with bias, i.e.  $x \mapsto Wx + t$  for some bias vector  $t$ . The discussion up to Section 2.1 works with biases as well, as does Lemma 2.2 (as it regards ReLU and not the layers themselves). For Proposition 2.3 it gets more complicated. What makes the proof of easy is that given an invertible linear layer  $\ell(x) = Wx$ , a group representation  $\rho(g)$  on  $x$  is transferred to a group representation  $\alpha(g, x) = \ell(\rho(g)\ell^{-1}(x)) = W\rho(g)W^{-1}x$  on  $\ell(x)$  w.r.t. which  $W$  is equivariant. If we consider an affine layer  $\ell(x) = Wx + t$  with bias vector  $t$ , then we can play the same game, but  $\rho(g)$  is not transferred to a group representation (linear action) but instead to the affine action  $\alpha(g, x) = W\rho(g)W^{-1}x + (I - W\rho(g)W^{-1})t$  on  $\ell(x)$ . This means that we can not apply Lemma 2.2. We can however find a generalization as follows:
Lemma E.1. If  $A, B$  are  $n \times n$  invertible matrices and  $a$  and  $b$  are  $n$ -vectors, such that  $\operatorname{ReLU}(Ax + a) = B\operatorname{ReLU}(x) + b$  for all  $x \in \mathbb{R}$ , then  $a = b = 0$ .
Proof. Inserting  $x = 0$  yields  $\operatorname{ReLU}(a) = b$ . Inserting  $x = A^{-1}a$  yields  $\operatorname{ReLU}(2a) = B\operatorname{ReLU}(A^{-1}a) + b$  so that  $b = B\operatorname{ReLU}(A^{-1}a)$ . Inserting  $x = -A^{-1}a$  yields  $0 = B\operatorname{ReLU}(-A^{-1}a) + b$  so  $b = -B\operatorname{ReLU}(-A^{-1}a)$ . Combined we have that  $B^{-1}b = \operatorname{ReLU}(A^{-1}a) = -\operatorname{ReLU}(-A^{-1}a)$  so that  $B^{-1}b$  must be zero and hence so must  $b$ . Finally, inserting  $x = -2A^{-1}a$  yields  $\operatorname{ReLU}(-a) = B\operatorname{ReLU}(-2A^{-1}a) + b$  so that  $\operatorname{ReLU}(-a) = -b = 0$  which combined with  $\operatorname{ReLU}(a) = b = 0$  gives  $a = 0$ .
This lemma shows that if ReLU commutes with affine actions, then the affine actions must in fact be linear and so Lemma 2.2 applies. This shows that Proposition 2.3 holds with affine layers as well. We thank the anonymous reviewer who prompted this generalization.