SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A General Framework for Robust  $G$ -Invariance in  $G$ -Equivariant Networks
Sophia Sanborn sanborn@ucsb.edu
Nina Miolane  
ninamiolane@ucsb.edu
Department of Electrical and Computer Engineering  
UC Santa Barbara
We introduce a general method for achieving robust group-invariance in group-equivariant convolutional neural networks ( $G$ -CNNs), which we call the  $G$ -triple-correlation ( $G$ -TC) layer. The approach leverages the theory of the triple-correlation on groups, which is the unique, lowest-degree polynomial invariant map that is also complete. Many commonly used invariant maps—such as the max—are incomplete: they remove both group and signal structure. A complete invariant, by contrast, removes only the variation due to the actions of the group, while preserving all information about the structure of the signal. The completeness of the triple correlation endows the  $G$ -TC layer with strong robustness, which can be observed in its resistance to invariance-based adversarial attacks. In addition, we observe that it yields measurable improvements in classification accuracy over standard Max  $G$ -Pooling in  $G$ -CNN architectures. We provide a general and efficient implementation of the method for any discretized group, which requires only a table defining the group's product structure. We demonstrate the benefits of this method for  $G$ -CNNs defined on both commutative and non-commutative groups— $SO(2)$ ,  $O(2)$ ,  $SO(3)$ , and  $O(3)$  (discretized as the cyclic  $C8$ , dihedral  $D16$ , chiral octahedral  $O$  and full octahedral  $O_h$  groups)—acting on  $\mathbb{R}^2$  and  $\mathbb{R}^3$  on both  $G$ -MNIST and  $G$ -ModelNet10 datasets.
The pooling operation is central to the convolutional neural network (CNN). It was originally introduced in the first CNN architecture—Fukushima's 1980 Neocognitron [17]—and remained a fixture of the model since. The Neocognitron was directly inspired by the canonical model of the visual cortex as a process of hierarchical feature extraction and local pooling [25, 1]. In both the neuroscience and CNN model, pooling is intended to serve two purposes. First, it facilitates the local-to-global coarse-graining of structure in the input. Second, it facilitates invariance to local changes—resulting in network activations that remain similar under small perturbations of the input. In this way, CNNs construct hierarchical, multi-scale features that have increasingly large extent and increasing invariance.
The pooling operation in traditional CNNs, typically a local max or average, has remained largely unchanged over the last forty years. The variations that have been proposed in the literature [40, 56] mostly tackle its coarse-graining purpose, improve computational efficiency, or reduce overfitting, but do not seek to enhance its properties with respect to invariance. Both max and avg operations are reasonable choices to fulfill the goal of coarse-graining within CNNs and  $G$ -CNNs. However, they are excessively imprecise and lossy with respect to the goal of constructing robust representations of objects that are invariant only to irrelevant visual changes. Indeed, the max and avg operations are invariant to many natural image transformations such as translations and rotations, but also
Figure 1: Achieving Robust  $G$ -Invariance in  $G$ -CNNs with the  $G$ -Triple-Correlation. The output of a  $G$ -Convolutional layer is equivariant to the actions of  $G$  on the domain of the signal. To identify signals that are equivalent up to group action, the layer can be followed by a  $G$ -Invariant map that eliminates this equivariance. In  $G$ -CNNs, Max  $G$ -Pooling is a commonly used for this purpose. Taking the maximum of the  $G$ -Convolutional equivariant output is indeed invariant to the actions of the group. However, it is also lossy: many non-equivalent output vectors have the same maximum. Our method—the  $G$ -Triple-Correlation is the lowest-order polynomial invariant map that is complete [46]. As a complete invariant, it preserves all information about the signal structure, removing only the action of the group. Our approach thus provides a new foundation for achieving robust  $G$ -Invariance in  $G$ -CNNs.
to unnatural transformations, including pixel permutations, that may destroy the image structure. This excessive invariance has been implicated in failure modes such as vulnerability to adversarial perturbations [20, 26], and a bias towards textures rather than objects [4]. To overcome these challenges and enable robust and selective invariant representation learning, there is a need for novel computational primitives that selectively parameterize invariant maps for natural transformations.
Many of the transformations that occur in visual scenes are due to the actions of groups. The appreciation of this fact has led to the rise of group-equivariant convolutional networks (G-CNNs) [8] and the larger program of Geometric Deep Learning [6]. While this field has leveraged the mathematics of group theory to attain precise generalized group-equivariance in convolutional network layers, the pooling operation has yet to meet its group theoretic grounding. Standardly, invariance to a group  $G$  is achieved with a simple generalization of max pooling: Max  $G$ -Pooling [8]—see Fig. 1 (top-right). However, this approach inevitably suffers from the lossiness of the max operation.
Here, we unburden the pooling operation of the dual duty of invariance and coarse-graining, by uncoupling these operations into two steps that can be performed with precision. We retain the standard max and avg pooling for coarse-graining, but introduce a new method for robust  $G$ -invariance via the group-invariant triple correlation —see Fig. 1 (bottom-right). The group-invariant triple correlation is the lowest-order complete operator that can achieve exact invariance [32]. As such, we propose a general framework for robust  $G$ -Invariance in  $G$ -Equivariant Networks. We show the advantage of this approach over standard max  $G$ -pooling in several  $G$ -CNN architectures. Our extensive experiments demonstrate improved scores in classification accuracy in traditional benchmark datasets as well as improved adversarial robustness.
We first cover the fundamentals of group-equivariant neural networks—also known as  $G$ -CNNs, or  $G$ -Equivariant Networks—before introducing the framework for  $G$ -Invariant Pooling.
2.1 Mathematical Prerequisites
The construction of  $G$ -CNNs requires mathematical prerequisites of group theory, which we recall here. The interested reader can find details in [23].
Groups. A group  $(G,\cdot)$  is a set  $G$  with a binary operation  $\cdot$ , which we can generically call the product. The notation  $g_{1}\cdot g_{2}$  denotes the product of two elements in the set; however, it is standard to omit
the operator and write simply  $g_{1}g_{2}$  — a convention we adopt here. Concretely, a group  $G$  may define a class of transformations. For example, we can consider the group of two-dimensional rotations in the plane—the special orthogonal group  $SO(2)$ —or the group of two-dimensional rotations and translations in the plane—the special euclidean group  $SE(2)$ . Each element of the group  $g \in G$  defines a particular transformation, such as one rotation by  $30^{\circ}$  or one rotation by  $90^{\circ}$ . The binary operation  $\cdot$  provides a means for combining two particular transformations—for example, first rotating by  $30^{\circ}$  and then rotating by  $90^{\circ}$ . In mathematics, for a set of transformations  $G$  to be a group under the operation  $\cdot$ , the four axioms of closure, associativity, identity and inverse must hold. These axioms are recalled in Appendix A.
Group Actions on Spaces. We detail how a transformation  $g$  can transform elements of a space, for example how a rotation of  $30^{\circ}$  indeed rotates a vector in the plane by  $30^{\circ}$ . We say that the transformations  $g$ 's act on (the elements of) a given space. Specifically, consider  $X$  a space, such as the plane. A group action is a function  $L: G \times X \to X$  that maps  $(g, x)$  pairs to elements of  $X$ . We say a group  $G$  acts on a space  $X$  if the following properties of the action  $L$  hold:
1. Identity: The identity  $e$  of the group  $G$  "does nothing", i.e., it maps any element  $x \in X$  to itself. This can be written as:  $\tilde{L}(e, x) = x$ .
2. Compatibility: Two elements  $g_1, g_2 \in G$  can be combined before or after the map  $L$  to yield the same result, i.e.,  $L(g_1, L(g_2, x)) = L(g_1 g_2, x)$ . For example, rotating a 2D vector by  $30^\circ$  and then  $40^\circ$  yields the same result as rotating that vector by  $70^\circ$  in one time.
For simplicity, we will use the shortened notation  $L_{g}(x)$  to denote  $L(g,x)$  the action of the transformation  $g$  on the element  $x$ .
Some group actions  $L$  have additional properties and turn the spaces  $X$  on which they operate into homogeneous spaces. Homogeneous spaces play an important role in the definition of the  $G$ -convolution in  $G$ -CNNs, so that we recall their definition here. We say that  $X$  is a homogeneous space for a group  $G$  if  $G$  acts transitively on  $X$ —that is, if for every pair  $x_{1}, x_{2} \in X$  there exists an element of  $g \in G$  such that  $L_{g}(x_{1}) = x_{2}$ . The concept can be clearly illustrated by considering the surface of a sphere, the space  $S^2$ . The sphere  $S^2$  is a homogeneous space for  $SO(3)$ , the group of orthogonal  $3 \times 3$  matrices with determinant one that define 3-dimensional rotations. Indeed, for every pair of points on the sphere, one can define a 3D rotation matrix that takes one to the other.
Group Actions on Signal Spaces. We have introduced essential concepts from group theory, where a group  $G$  can act on any abstract space  $X$ . Moving towards building  $G$ -CNNs, we introduce how groups can act on spaces of signals, such as images. Formally, a signal is a map  $f: \Omega \to \mathbb{R}^c$ , where  $\Omega$  is called the domain of the signal and  $c$  denotes the number of channels. The space of signals itself is denoted  $L_2(\Omega, \mathbb{R}^c)$ . For example,  $\Omega = \mathbb{R}^2$  or  $\mathbb{R}^3$  for 2D and 3D images. Gray-scale images have one channel ( $c = 1$ ) and color images have the 3 red-green-blue channels ( $c = 3$ ).
Any action of a group of transformations  $G$  on a domain  $\Omega$  yields an action of that same group on the spaces of signals defined on that domain, i.e., on  $L_{2}(\Omega, \mathbb{R}^{c})$ . For example, knowing that the group of 2D rotations  $SO(2)$  acts on the plane  $\Omega = \mathbb{R}^2$  allows us to define how  $SO(2)$  rotates 2D gray-scale images in  $L_{2}(\mathbb{R}^{2}, \mathbb{R}^{c})$ . Concretely, the action  $L$  of a group  $G$  on the domain  $\Omega$  yields the following action of  $G$  on  $L_{2}(\Omega, \mathbb{R}^{c})$ :
$$
L _ {g} [ f ] (u) = f \left(L _ {g ^ {- 1}} (u)\right), \quad \text {f o r a l l} u \in \Omega \text {a n d f o r a l l} g \in G. \tag {1}
$$
We use the same notation  $L_{g}$  to refer to the action of the transformation  $g$  on either an element  $u$  of the domain or on a signal  $f$  defined on that domain, distinguishing them using [.]. for the signal case. We note that the domain of a signal can be the group itself:  $\Omega = G$ . In what follows, we will also consider actions on real signals defined on a group, i.e., on signals such as  $\Theta : G \to \mathbb{R}$ .
Invariance and Equivariance. The concepts of group-invariance and equivariance are at the core of what makes the  $G$ -CNNs desirable for computer vision applications. We recall their definitions here. A function  $\psi : X \mapsto Y$  is  $G$ -invariant if  $\psi(x) = \psi(L_g(x))$ , for all  $g \in G$  and  $x \in X$ . This means that group actions on the input space have no effect on the output. Applied to the group of rotations acting on the space of 2D images  $X = L_2(\Omega, \mathbb{R}^c)$  with  $\Omega = \mathbb{R}^2$ , this means that a  $G$ -invariant function  $\psi$  produces an input that will stay the same for any rotated version of a given signal. For example, whether the image contains the color red is invariant with respect to any rotation of that
image. A function  $\psi : X \mapsto Y$  is  $G$ -equivariant if  $\psi(L_g(x)) = L_g'(\psi(x))$  for all  $g \in G$  and  $x \in X$ , where  $L$  and  $L'$  are two different actions of the group  $G$ , on the spaces  $X$  and  $Y$  respectively. This means that a group action on the input space results in a corresponding group action of the same group element  $g$  on the output space. For example, consider  $\psi$  that represents a neural network performing a foreground-background segmentation of an image. It is desirable for  $\psi$  to be equivariant to the group of 2D rotations. This equivariance ensures that, if the input image  $f$  is rotated by  $30^\circ$ , then the output segmentation  $\psi(f)$  rotates by  $30^\circ$  as well.
2.2  $G$ -Equivariant Networks
$G$ -CNNs are built from the following fundamental building blocks:  $G$ -convolution, spatial pooling, and  $G$ -pooling. The  $G$ -convolution is equivariant to the action of the group  $G$ , while the  $G$ -pooling achieves  $G$ -invariance. Spatial pooling achieves coarse-graining. We review the group-specific operations here. The interested reader can find additional details in [8, 10], which include the definitions of these operations using the group-theoretic framework of principal bundles and associated vector bundles.
2.2.1  $G$ -Convolution
In plain language, a standard translation-equivariant convolutional neural network layer sweeps filters across a signal (typically, an image), translating the filter and then taking an inner product with the signal to determine the similarity between a local region and the filter.  $G$ -CNNs [8] generalize this idea, replacing translation with the action of other groups that define symmetries in a machine learning task—for example, rotating a filter, to determine the presence of a feature in various orientations.
Consider a signal  $f$  defined on a domain  $\Omega$  on which a group  $G$  acts. A neural network filter is a map  $\phi : \Omega \to \mathbb{R}^c$  defined with the same domain  $\Omega$  and codomain  $\mathbb{R}^c$  as the signal. A  $G$ -convolutional layer is defined by a set of filters  $\{\phi_1, \dots, \phi_K\}$ . For a given filter  $k$ , the layer performs a  $G$ -convolution with the input signal  $f$ :
$$
\Theta_ {k} (g) = \left(\phi_ {k} * f\right) (g) = \int_ {u \in \Omega} \phi_ {k} \left(L _ {g ^ {- 1}} (u)\right) f (u) d u, \quad \forall g \in G, \tag {2}
$$
by taking the dot product in  $\mathbb{R}^c$  of the signal with a transformed version of the filter. In practice, the domain  $\Omega$  of the signal is discretized, such that the  $G$ -convolutional layer becomes:
$$
\Theta_ {k} (g) = \sum_ {u \in \Omega} \phi_ {k} \left(L _ {g ^ {- 1}} (u)\right) f (u), \quad \forall g \in G. \tag {3}
$$
The output of one filter  $k$  is therefore a map  $\Theta_k: G \to \mathbb{R}$ , while the output of the whole layer with  $K$  filters is  $\Theta: G \to \mathbb{R}^K$  defined as  $\Theta(g) = [\Theta_1(g), \dots, \Theta_K(g)]$  for all  $g \in G$ . The  $G$ -convolution therefore outputs a signal  $\Theta$  whose domain has necessarily become the group  $\Omega = G$  and whose number of channels is the number of convolutional filters  $K$ .
The  $G$ -convolution is equivariant to the action of the group on the domain of the signal  $f$  [8]. That is, the action of  $g$  on the domain of  $f$  results in a corresponding action on the output of the layer. Specifically, consider a filter  $\phi_k$ , we have:
$$
\phi_ {k} * L _ {g} [ f ] = L _ {g} ^ {\prime} \left[ \phi_ {k} * f \right], \quad \forall g \in G, \tag {4}
$$
where  $L_{g}$  and  $L_{g}^{\prime}$  represent the actions of the same group element  $g$  on the functions  $f$  and  $\phi_k * f$  respectively. This property applies for the  $G$ -convolutions of the first layer and of the next layers [8].
2.2.2  $G$ -Pooling
Invariance to the action of the group is achieved by pooling over the group  $(G$ -Pooling) [8]. The pooling operation is typically performed after the  $G$ -convolution, so that we restrict its definition to signals  $\Theta$  defined over a group  $G$ . In  $G$ -pooling, a max typically is taken over the group elements:
$$
\mu_ {k} = \max  _ {g \in G} \Theta_ {k} (g). \tag {5}
$$
$G$ -pooling extracts a single real scalar value  $\mu_{k}$  from the full feature vector  $\Theta_{k}$ , which has  $|G|$  values, with  $|G|$  the size of the (discretized) group  $G$  as shown in Fig. 1. When the group  $G$  is a grid
discretizing  $\mathbb{R}^n$ , max  $G$ -Pooling is equivalent to the standard spatial max pooling used in translation-equivariant CNNs, and it can be used to achieve coarse-graining. More generally,  $G$ -Pooling is  $G$ -invariant, as shown in [8]. However, we argue here that it is excessively  $G$ -invariant. Although it achieves the objective of invariance to the group action, it also loses substantial information. As illustrated in Fig. 1, many different signals  $\Theta$  may yield same result  $\mu$  through the  $G$ -pooling operation, even if these signals do not share semantic information. This excessive invariance creates an opportunity for adversarial susceptibility. Indeed, inputs  $f$  can be designed with the explicit purpose of generating a  $\mu_k$  that will fool a neural network and yield an unreasonable classification result. For this reason, we introduce our general framework for robust, selective  $G$ -invariance.
3 The  $G$ -Triple-Correlation Layer for Robust  $G$ -Invariance
We propose a  $G$ -Invariant layer designed for  $G$ -CNNs that is complete—that is, it preserves all information about the input signal except for the group action. Our approach leverages the theory of the triple correlation on groups [32] and applies it to the design of robust neural network architectures. Its theoretical foundations in signal processing and invariant theory allows us to generally define the unique  $G$ -invariant maps of lowest polynomial order that are complete, hence providing a general framework for selective, robust  $G$ -invariance in  $G$ -CNNs [46].
3.1 The  $G$ -Triple-Correlation Layer
The  $G$ -Triple-Correlation  $(G\text{-TC})$  on a real signal  $\Theta : G \to \mathbb{R}$  is the integral of the signal multiplied by two independently transformed copies of it [32]:
$$
\tau_ {\Theta} \left(g _ {1}, g _ {2}\right) = \int_ {g \in G} \Theta (g) \Theta \left(g g _ {1}\right) \Theta \left(g g _ {2}\right) d g. \tag {6}
$$
This definition holds for any locally compact group  $G$  on which we can define the Haar measure  $dg$  used for integration purposes [28]. This definition above is applicable to the  $G$ -CNNs where  $\Theta$  is a collection of scalar signals over the group. We show in Appendix B that we can extend the definition to steerable  $G$ -CNNs where  $\Theta$  can be an arbitrary field [9].
In the equation above, the  $G$ -TC is computed for a pair of group elements  $g_{1}, g_{2}$ . In practice, we sweep over all pairs in the group. Appendix C illustrates the triple correlation on three concrete groups. Importantly, the  $G$ -triple-correlation is invariant to the action of the group  $G$  on the signal  $\Theta$  [28], as shown below.
Proposition 1. Consider a signal  $\Theta : G \mapsto \mathbb{R}^c$ . The  $G$ -Triple-Correlation  $\tau$  is  $G$ -invariant:
$$
\tau_ {L g [ \Theta ]} = \tau_ {\Theta}, \quad f o r a l l g \in G, \tag {7}
$$
where  $L_{g}$  denotes an action of a transformation  $g$  on the signal  $\Theta$ .
The proof is recalled in Appendix D. We propose to achieve  $G$ -invariance in a  $G$ -CNN by applying the  $G$ -Triple-Correlation (G-TC) to the output  $\Theta$  of a  $G$ -convolutional layer. Specifically, we apply the  $G$ -TC to each real scalar valued signal  $\Theta_{k}$  that comes from the  $G$ -convolution of filter  $\phi_{k}$ , for  $k \in \{1, \dots, K\}$ . We only omit the subscript  $k$  for clarity of notations. In practice, we will use the triple correlation on discretized groups, where the integral is replaced with a summation:
$$
T _ {\Theta} \left(g _ {1}, g _ {2}\right) = \sum_ {g \in G} \Theta (g) \Theta \left(g g _ {1}\right) \Theta \left(g g _ {2}\right), \tag {8}
$$
for  $\Theta$  a scalar valued function defined over  $G$ . While it seems that the layer computes  $T_{\Theta}(g_1, g_2)$  for all pairs of group elements  $(g_1, g_2)$ , we note that the real scalars  $\Theta(gg_1)$  and  $\Theta(gg_2)$  commute so that only half of the pairs are required. We will see that we can reduce the number of computations further when the group  $G$  possesses additional properties such as commutativity.
We note that the triple correlation is the spatial dual of the bispectrum, which has demonstrated robustness properties in the context of deep learning with bispectral neural networks [42]. The goal of bispectral neural networks is to learn an unknown group  $G$  from data. The bispectral layer proposed in [42] assumes an MLP architecture. Our work is the first to generalize the use of bispectral invariants to convolutional networks. Here, we assume that the group  $G$  is known in advance, and
exploit the theoretical properties of the triple correlation to achieve robust invariance. One path for future extension may be to combine our approach with the learning approach of [42], to parameterize and learn the group  $G$  that defines a  $G$ -Equivariant and  $G$ -Invariant layer.
3.2 Selective Invariance through Completeness
We show here that the proposed  $G$ -triple-correlation is guaranteed to preserve all information aside from any equivariant component due to the group action on the input domain. This crucial property distinguishes our proposed layer from standard  $G$ -Pooling methods, which collapse signals and lose crucial information about the input (Figure 1). In contrast with standard, excessively invariant  $G$ -pooling methods, we show here that our  $G$ -TC layer is instead selectively  $G$ -invariant thanks to its completeness property [54, 29, 31], defined here:
Proposition 2. Every integrable function with compact support  $G$  is completely identified—up to group action—by its  $G$ -triple-correlation. We say that the  $G$ -triple-correlation is complete.
Mathematically, an operator  $\mathcal{T}$  is complete for a group action  $L$  if the following holds: for every pair of signals  $\Theta_{1}$  and  $\Theta_{2}$ , if  $\mathcal{T}(\Theta_1) = \mathcal{T}(\Theta_2)$  then the signals are equal up to the group action, that is: there exists a group element  $h$  such that  $\Theta_{2} = L_{h}[\Theta_{1}]$ .