SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We describe some concrete examples of partial and approximate equivariance using the language we used in section 3.2 while sourcing them from existing literature. But first, we expand a little on our equivariance error notation.
B.1 Equivariance error notation
Recall that the action of elements of an ambient group  $G$  over the product space  $\mathcal{Z} = \mathcal{X}\times \mathcal{Y}$  may be written as follows: For coordinates  $z = x\times y$  we may write  $g\cdot z = (g\cdot x,g\cdot y)$ , and thus for  $\widetilde{f}:\mathcal{X}\to \mathcal{Y}$  and  $f(x,y) = \ell (\widetilde{f} (x),y)$ , we have the action
$$
(g \cdot f) (z) := f (g \cdot z) = \ell (\widetilde {f} (g \cdot x), g \cdot y).
$$
For the equivariance error of  $g, f$ , interpreted as "the error of  $f$ 's approximate equivariance under the action by  $g$ ", we get the following, which is valid in the common situations in which  $\ell(y, y') \geq 0$  in general with  $\ell(y, y) = 0$  for all  $y \in \mathcal{V}$ :
$$
\operatorname {e e} (f, g) := \| g \cdot f - f \| _ {\infty} = \sup  _ {x, y} \left| \ell (\widetilde {f} (g \cdot x), g \cdot y) - \ell (\widetilde {f} (x), y) \right| \geq \sup  _ {x} \ell (\widetilde {f} (g \cdot x), g \cdot \widetilde {f} (x)), \tag {B.1}
$$
where the last inequality follows by restricting the supremum from  $\mathcal{X} \times \mathcal{Y}$  to the graph of  $\widetilde{f}$ , namely by imposing  $y = \widetilde{f}(x)$ .
In several recent works, the equivariance error is defined simply by comparing  $g \cdot \widetilde{f}(x)$  and  $\widetilde{f}(g \cdot x)$ , as in the rightmost term of (B.1), thus it is lower than the one found here. We provide a justification for our definition of the equivariance error:
- The loss  $\ell$  is the integrative part of the model, thus a definition for equivariance error which does not include it will only detect partial information concerning the influence of symmetries.
- The notion of equivariance error defined via  $\ell$  simplifies the comparison between  $\widetilde{f}$  and data distributions  $\mathcal{D}$ .
B.2 Examples
B.2.1 Imperfect translation equivariance in classical CNNs
We consider here the most common examples of group equivariant convolutional networks (GCNNs), which are the usual Convolutional Neural Networks (CNNs) for computer vision tasks. We follow observations from [27] and [11], and connect the underlying ideas to Theorem 6.1.
Setting of the problem. We consider a usual CNN layer, keeping in mind a segmentation task, where both  $\mathcal{X} = \mathcal{Y}$  represent spaces of images. More precisely, we think of images as pixel values at integer-coordinate positions taken from the index set  $\mathbb{Z} \times \mathbb{Z}$ . We also assume that the relevant information of each image only comes from a square of size  $n \times n$  pixels, outside which the pixel values are taken to be 0. We consider the application of a single convolution kernel/filter, of  $k \times k$  pixels (with  $k$  a small odd number). One typically applies padding by a layer of 0's of size  $(k - 1)/2$  on the perimeter of the  $n \times n$  square, after which convolution with the kernel is computed on the  $n \times n$  central pixels of the resulting  $(n + k - 1) \times (n + k - 1)$  padded input image. The output relevant information is restricted to a  $n \times n$  square, outside which pixel values are set to 0 again, via padding.
Metric on  $\mathcal{X}$ . As a natural choice of distance over  $\mathcal{X}$  we may consider  $L^2$ -difference between pixel-value functions, or interpret pixel values as probability densities, and use Wasserstein distance, or consider other ad-hoc image metrics.
Group action: translations. The group acting on our "pixel-value functions" is the group of translations with elements from  $\mathbb{Z} \times \mathbb{Z}$ . We expect the following invariance for the segmentation function  $f: \mathcal{X} \to \mathcal{X}$ :
$$
f (v \cdot x) = v \cdot f (x),
$$
where  $x \in \mathcal{X}$  represents an image with pixel values assigned to integer coordinates and  $v \in \mathbb{Z}^2$  is a translation vector and (in two alternative notations)  $v \cdot x = \tau_v(x)$  is the result of translating all pixel values of  $x$  by  $v$ .
Deformation properties of the action. If we take the previously mentioned distance functions on  $\mathcal{X}$  and the usual distance induced from  $\mathbb{R}^2$  over translation vectors  $v$ , it is easy to verify that the assumptions of Theorem A.3 about the action of translations hold, and the Lipschitz constants with respect to the metric on  $\mathcal{X}$  only depend on the mismatch near the boundary, due to "zero pixels moving in" and to "interior pixels moving out" of our  $n \times n$  square, and being truncated. The ensuing bounds only depend on the precise distance that we introduce use on  $\mathcal{X}$ .
More realistic actions. An alternative more realistic definition of  $\mathbb{Z} \times \mathbb{Z}$ -action consists of defining  $v \cdot x$  as the truncation of  $\tau_v(x)$  where, for pixels outside our "relevant"  $n \times n$  square we set pixel value to 0 after the translation.
Problems near the boundary. Nevertheless, the updated translation action, will move pixel values of 0, coming from pixels outside the  $n \times n$  square, and will create artificial zero pixel values inside the image  $v \cdot x$ , different than the values that would be present in real data.
Imperfect equivariance of data. Also, even in the latter more realistic alternative, the above translation equivariance is not respecting by segmentation input-output pairs coming from finite  $n \times n$  images, since, independently of  $n$ , the boundary pixel positions translated by  $v$ , fall outside the original image.
Approximate stabilizer. In any case, we have to restrict the choices of  $v$  to integer-coordinate elements of a (subset of a)  $n \times n$  square, containing only the translations that are relating "real" segmentations that appear within our  $n \times n$  relevant window. It is thus
natural to restrict  $\mathsf{Stab}_{\epsilon}$  to only include vectors in a smaller subset of  $\mathbb{Z} \times \mathbb{Z}$ , of cardinality
$$
\left| \operatorname {S t a b} _ {\epsilon} \right| \leq n ^ {2}.
$$
The value of error  $\epsilon$  may quantify the allowed error (or noisiness) for our model sets.
Reducing to finite  $G$  and computing  $\lambda$ . For the sake of computing  $\lambda$  in our Theorem 6.1, we can observe that input and output values outside the "padding perimeter" given by a  $(n + k - 1) \times (n + k - 1)$  square, are irrelevant, and thus we may actually periodize the images and consider the images as subsets of a padded torus  $(\mathbb{Z} / (n + k - 1)\mathbb{Z})^2$ , which we consider as acting on itself by translations. In this case  $G \simeq (\mathbb{Z} / (n + k - 1)\mathbb{Z})^2$ , so that
$$
| G | = (n + k - 1) ^ {2} \quad \mathrm {a n d t h u s} \quad \lambda = \frac {| \mathsf {S t a b} _ {\epsilon} |}{(n + k - 1) ^ {2}}.
$$
Further extensions. In [27], it is argued that convolutional layers in classical CNNs are not fully translation-equivariant, and can encode positional information, due to the manner in which boundary padding is implemented. Possible solutions are increasing the padding to size  $k$  (so that the padded image is a square of size  $(n + 2k) \times (n + 2k)$ ) or extending images by periodicity, via so-called "circular padding" (which transforms each image into a space equivalent to the torus  $(\mathbb{Z} / n\mathbb{Z})^2$ ). In either case, the application of actions by translations by vectors that are too large compared to the image size of  $n \times n$ , will increase the mismatch between model equivariance and data equivariance.
Stride and downsampling. In [11], a different equivariance error for classical CNNs is studied, related to the use of stride  $>1$  in order to lower the output dimensions of CNN layer outputs. If for example we use stride 2 when defining layer operation  $f:\mathcal{X}\to \mathcal{V}$ , then  $\mathcal{V}$  will have relevant pixel values only in an  $n / 2\times n / 2$ -square, and we apply the  $k\times k$  convolution kernel only at positions with coordinates in  $(2\mathbb{Z})\times (2\mathbb{Z})$  from image  $x$ . In this case we require that translations by group elements  $v\in (2\mathbb{Z})\times (2\mathbb{Z})$  on  $x$  have the effect of a translation by  $v / 2\in \mathbb{Z}$  on the output. However for shifts in the input via vectors  $v$  that do not have two even coordinates, we may not have an explicit corresponding action on the output, and in [11] a solution via adaptive polyphase sampling is proposed. A possibility for studying the best polyphase sampling strategy via almost equivariance, would be to include a bound for equivariance error  $\epsilon >0$  and consider the optimization problem of finding the polyphase approximation that minimizes theoretical or empirical quantifications of  $\epsilon$ . As a benchmark (modelled on the case of infinite images without boundary effects) one could compare the above to the action via  $\mathsf{Stab}_0 = (2\mathbb{Z})\times (2\mathbb{Z})$  which has  $\lambda = 1 / 4$  within the ambient group  $G = \mathbb{Z}\times \mathbb{Z}$ . Our Theorem 6.1 can be used to compare the effects of increasing or decreasing  $\epsilon, \lambda$ , in terms of data symmetry.
B.2.2 Partial equivariance in GCNNs
In this section, we connect the main results from [40] to our setup. In [40], one of the main motivating examples was to consider rotations applied to a handwritten digit and revert them. The underlying group action was via  $SO(2)$  and only rotations of angles between  $[-60^{\circ}, 60^{\circ}]$  were permitted in one case, which allowed us to not confound rotated digits "3" and "9" for example.
The above task can be formulated on a space of functions  $f: \mathcal{X} \to \mathcal{Y}$  in which  $\mathcal{X}$  represents the space of possible images and  $\mathcal{Y}$  the labels. Elements  $(Y_d, Y_\theta) \in \mathcal{Y}$  include a digit classification label  $Y_d$  and a rotation angle value  $Y_\theta$ .
We consider actions by group  $G = SO(2) = \{R_{\phi} : \phi \in \mathbb{R} / 360\mathbb{Z}\}$ , where  $R_{\phi}$  is the rotation matrix by angle  $\phi$ , and the group operation corresponds to summing the angles, modulo  $360^{\circ}$  (or in radians, modulo  $2\pi$ ). The action of  $R_{\phi}$  over  $\mathcal{X}$  would be by rotation as usual ( $R_{\theta}$  sends image  $x \in \mathcal{X}$  to  $R_{\theta} \cdot x$ , now rotated by  $\theta$ ), and over  $\mathcal{Y}$  we consider the action by
$$
R _ {\phi} \left(Y _ {d}, Y _ {\theta}\right) = \left(Y _ {d}, Y _ {\theta} + \phi\right),
$$
i.e. the restriction of the action on the digit label leaves it invariant and the restriction of the action on the angle label is non-trivial, giving a shift on the label.
The optimum labelling assigns to  $x$  a label  $y^{*}(x)$  enjoying precise equivariance under the above definitions of the actions, and thus we are allowed to permit equivariance error  $\epsilon = 0$ . However as mentioned above, applying rotations outside the range  $\theta \in [-60^{\circ}, 60^{\circ}]$  to the data would surely bring us outside the labelled data distribution, thus we are led to take
$$
\epsilon = 0, \quad \operatorname {S t a b} _ {0} = \left\{R _ {\theta}: \theta \in \left[ - 6 0 ^ {\circ}, 6 0 ^ {\circ} \right] \right\}.
$$
We then have that  $|\mathsf{Stab}_0| / |SO(2)| = 1/3$ , with respect to the natural Haar measure on rotations. It is natural to think of the set of  $\mathsf{Stab}_0$ -action representatives of images  $\mathcal{X}_0^0$  given as the "unrotated" images. If we take a digit image that is rotated, say by  $20^\circ$ , from its "base" version, and we apply a rotation of  $50^\circ$  to it (i.e., an element of  $\mathsf{Stab}_0$ ), then we reach the version of the image now excessively rotated by  $70^\circ$ . This means that without further modification, considering model symmetries with  $\mathsf{Stab}_0$  taken to be independent of the point, would automatically generate some error when tested on the data. While decreasing the threshold angle in the definition of  $\mathsf{Stab}_0$  from  $60^\circ$  would limit this effect, it will also decrease generalization error in the model. The study of point-dependent invariance sets  $\mathsf{Stab}_{\epsilon}$  is interesting in view of this example application, but it is outside the scope of the current approximation/generalization bounds and is left for future work.
B.2.3 Possible applications to partial equivariance in Reinforcement Learning
The use of approximate invariances for RL applications was considered in [23, Sec. 6] via soft equivariance constraints allowing better generalization for an agent moving in a geometric environment. While imposing approximate equivariance for memoryless  $G$ -action for groups such as  $G = SO(2)$ ,  $\mathbb{Z}_2$  has produced positive results, it may be interesting, in analogy to the previous section, to include memory, and thus restrict the choices of group actions across time steps. Note that for a temporal evolution of  $T$  steps, the group action by  $G$  acting independently at each step would produce a  $T$ -interval action via the product group  $G^T$ , and allowing for a partial action via  $\mathsf{Stab}_{\epsilon}$ , with possibly increased fitness to evolving data. More precise time-dependence prescriptions and consequences within  $Q$ -learnig are left to future work.
C Discussion of [20]
In section 2 we mentioned [20], which considers PAC-style bounds under model symmetry. [20] works with compact groups and argues how the learning problem in such a setup reduces to only working with a set of reduced orbit representatives, which leads to a generalization gain. This message of [20] is similar to ours, although we work with a more general setup. However, we noted that the main theorem in [20] has an error. Here, we briefly sketch the issue with the proof.
One of the main quantities of interest in the main theorem of [20] is  $D_{\tau}(\mathcal{X}, \mathcal{H})$  (notation from their paper), which directly comes from the following bound, and is claimed to have a linear dependence on  $Cov(\mathcal{X}, \rho, \delta)$ . Again, for the sake of easier verification, we follow their notation. Crucially, note that [20] uses the notation  $Cov$  as analogous to our  $\mathcal{N}$ :
$$
C o v (\mathcal {H}, \| \cdot \| _ {L _ {\infty}}, 2 C \delta + \kappa) \leq C o v (\mathcal {X}, \rho , \delta) \sup  _ {x \in \mathcal {X}} C o v (\mathcal {H} (x), \| \cdot \| _ {\infty}, \kappa)
$$
However, the correct application of the Kolmogorov-Tikhomirov estimate shows that the reasoning in the proof should yield a dependence which is exponential in  $Cov(\mathcal{X}, \rho, \delta)$ , not linear. To see this, set  $s = 2$  (sufficient for our purposes) in equation 238 in [54] (page 186). In other words, it is not possible to cover Lipschitz functions in infinity norm by only using constant functions.