SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proof of the completeness of the  $G$ -triple-correlation is only valid under a precise set of assumptions [32] (Theorem 2). As we seek to integrate the  $G$ -triple-correlation to enhance robustness in neural networks, we investigate here the scope of these assumptions. First, the assumptions are not restrictive on the type of groups  $G$  that can be used. Indeed, the proof only requires the groups to be Tatsuyaama duality groups and the groups of interest in this paper meet this condition. This includes all locally compact commutative groups, all compact groups including the groups of rotations, the special orthogonal groups  $SO(n)$ , and groups of translations and rotations, the special euclidean groups  $SE(n)$ . Second, the assumptions are not restrictive on the types of signals. Indeed, the signal only needs to be such that any of its Fourier transform coefficients are invertible. For example, when the Fourier transform coefficients are scalar values, this means that we require these scalars to be non-zero. In practical applications on real image data with noise, there is a probability 0 that the Fourier transform coefficients of the input signal will be exactly 0 (scalar case) or non-invertible (matrix case). This is because the group of invertible matrices is dense in the space of matrices. Therefore, this condition is also verified in the applications of interest and more generally we expect the property of completeness of our  $G$ -TC layer to hold in practical neural network applications.
3.3 Uniqueness
The above two subsections prove that our  $G$ -Triple Correlation layer is selectively  $G$ -invariant. Here, we note that our proposed layer is the lowest-degree polynomial layer that can achieve this goal. In invariant theory, it is observed that the  $G$ -Triple Correlation is the only third-order polynomial invariant (up to change of basis) [46]. Moreover, it is the lowest-degree polynomial invariant that is also complete. It thus provides a unique and minimal-complexity solution to the problem of robust invariance within this function class.
3.4 Computational Complexity
The  $G$ -Triple Correlation enjoys some symmetries that we can leverage to avoid computing it for each pair of group elements (which would represent  $|G|^2$  computations), hence making the feedforward pass more efficient. We summarize these symmetries here.
Proposition 3. Consider two transformations  $g_1, g_2 \in G$ . The  $G$ -Triple Correlation of a real signal  $\Theta$  has the following symmetry:
$$
T _ {\Theta} (g _ {1}, g _ {2}) = T _ {\Theta} (g _ {2}, g _ {1}).
$$
If  $G$  is commutative, the  $G$ -Triple Correlation of a real signal has the following additional symmetries:
$$
T _ {\Theta} (g _ {1}, g _ {2}) = T _ {\Theta} (g _ {1} ^ {- 1}, g _ {2} g _ {1} ^ {- 1}) = T _ {\Theta} (g _ {2} g _ {1} ^ {- 1}, g _ {1} ^ {- 1}) = T _ {\Theta} (g _ {2} ^ {- 1}, g _ {1} g _ {2} ^ {- 1}) = T _ {\Theta} (g _ {1} g _ {2} ^ {- 1}, g _ {2} ^ {- 1}).
$$
The proofs are given in [39] for the group of translations. We extend them to any locally compact group  $G$  in Appendix E. In practice, these symmetries mean that even if there are theoretically
$|G|^{2}$  computations, this number immediately reduces to  $\frac{|G|(|G| + 1)}{2}$  and further reduces if the group  $G$  of interest is commutative. In addition, more subtle symmetries can be exploited to reduce the computational cost to linear  $|G| + 1$  for the case of one-dimensional cyclic groups [34] by considering the spectral dual of the  $G$ -TC: the bispectrum. We provide a computational approach to extend this reduction to more general, non-commutative groups in Appendix F. The theory supporting our approach has yet to be extended to this general case. Thus, there is an opportunity for new theoretical work that further increases the computational efficiency of the  $G$ -Triple-Correlation.
The Triple Correlation. The triple correlation has a long history in signal processing [48, 5, 39]. It originally emerged from the study of the higher-order statistics of non-Gaussian random processes, but its invariance properties with respect to translation have been leveraged in texture statistics [53] and data analysis in neuroscience [13], as well as early multi-layer perceptron architectures in the 1990's [12, 33]. The triple correlation was extended to groups beyond translations in [32], and its completeness with respect to general compact groups was established in [30]. To the best of our knowledge, the triple correlation has not previously been introduced as a method for achieving invariance in convolutional networks for either translation or more general groups.
Pooling in CNNs. Pooling in CNNs typically has the dual objective of coarse graining and achieving local invariance. While invariance is one desiderata for the pooling mechanism, the machinery of group theory is rarely employed in the computation of the invariant map itself. As noted in the introduction, max and average pooling are by far the most common methods employed in CNNs and  $G$ -CNNs. However, some approaches beyond strict max and average pooling have been explored. Soft-pooling addresses the lack of smoothness of the max function and uses instead a smooth approximation of it, with methods including polynomial pooling [49] and learned-norm [22], among many others [15, 14, 43, 44, 3, 45, 11, 35]. Stochastic pooling [57] reduces overfitting in CNNs by introducing randomness in the pooling, yielding mixed-pooling [55], max pooling dropout [51], among others [47, 58, 21]
Geometrically-Aware Pooling. Some approaches have been adopted to encode spatial or structural information about the feature maps, including spatial pyramid pooling [24], part-based pooling [59], geometric  $L_{p}$  pooling [16] or pooling regions defined as concentric circles [41]. In all of these cases, the pooling computation is still defined by a max. These geometric pooling approaches are reminiscent of the Max  $G$ -Pooling for  $G$ -CNNs introduced by [8] and defined in Section 2.2.2, without the explicit use of group theory.
Higher-Order Pooling. Average pooling computes first-order statistics (the mean) by pooling from each channel separately and does not account for the interaction between different feature maps coming from different channels. Thus, second-order pooling mechanisms have been proposed to consider correlations between features across channels [38, 19], but higher-orders are not investigated. Our approach computes a third-order polynomial invariant; however, it looks for higher-order correlations within the group rather than across channels and thus treats channels separately. In principle, these approaches could be combined.
5 Experiments & Results
We implement the  $G$ -TC Layer for arbitrary discretized groups with an efficient implementation built on top of the ESCNN library [7, 50], which provides a general implementation of  $E(n)$ -Equivariant Steerable Convolutional Layers. The method is flexibly defined, requiring the user only to provide a (Cayley) table that defines the group's product structure. The code is publicly available at https://github.com/sophiaas/gtc-invariance. Here, we demonstrate the approach on the groups  $SO(2)$ , and  $O(2)$ ,  $SO(3)$ , and  $O(3)$ , discretized as the groups  $C_n$  (cyclic),  $D_n$  (dihedral),  $O$  (chiral octahedral), and  $O_h$  (full octahedral), respectively. ESCNN provides implementations for  $G$ -Conv layers on all of these  $E(n)$  subgroups.
O(2)
Figure 2: Datasets. The  $O(2)$ -MNIST (top) and  $O(3)$ -ModelNet10 (bottom) datasets are generated by applying a random (rotation, reflection) pair to each element of the original datasets. Although we visualize the continuous group here, in practice, we discretize the group  $O(3)$  as the full octahedral group  $O_h$  to reduce computational complexity.  $SO(2)$  and  $SO(3)$  datasets are generated similarly, by applying a random rotation to each datapoint.
O(3)
We examine the performance of the  $G$ -TC over Max  $G$ -Pooling in  $G$ -Equivariant Networks defined on these groups and trained on  $G$ -Invariant classification tasks. For the groups  $SO(2)$  and  $O(2)$  acting on  $\mathbb{R}^2$ , we use the MNIST dataset of handwritten characters [37], and for the groups  $SO(3)$  and  $O(3)$  acting on  $\mathbb{R}^3$ , we use the voxelized ModelNet10 database of 3D objects [52]. We generate  $G$ -MNIST and  $G$ -ModelNet10 datasets by transforming the domain of each signal in the dataset by a randomly sampled group element  $g \in G$  (Figure 2).
In these experiments, we train pairs of models in parameter-matched architectures, in which only the  $G$ -Pooling method differs. Note that the purpose of these experiments is to compare differences in performance between models using Max  $G$ -Pooling vs. the  $G$ -TC—not to achieve SOTA accuracy. Thus, we do not optimize the models for overall performance. Rather, we fix a simple architecture and set of hyperparameters and examine the change in performance that arises from replacing Max  $G$ -Pooling with the  $G$ -TC Layer (Figure 3).
Figure 3: Models. We compare two simple architectures comprised of a single G-Conv block followed by either a Max  $G$ -Pool layer or a  $G$ -TC Layer and an MLP Classifier.
To isolate the effects of the  $G$ -Pooling method, all models are comprised of a single  $G$ -Conv block followed by  $G$ -Pooling (Max or TC) and an MLP Classifier. Notably, while many  $G$ -Conv models in the literature use the semi-direct product of  $G$  with  $\mathbb{R}^n$  —i.e. incorporating the actions of the group  $G$  into a standard translational convolutional model—here, we perform only pure  $G$ -Conv, without translation. Thus, we use filters the same size as the input in all models. The  $G$ -Conv block is comprised of a  $G$ -Conv layer, a batch norm layer, and an optional nonlinearity. For the
Max  $G$ -Pool model, ReLU is used as the nonlinearity. Given the third-order nonlinearity of the TC, we omit the nonlinearity in the  $G$ -Conv block in the TC Model. The  $G$ -TC layer increases the dimensionality of the output of the  $G$ -Conv block; consequently the input dimension of the first layer of the MLP is larger and the weight matrix contains more parameters than for the Max  $G$ -Pool model. To compensate for this, we increase the dimension of the output of the first MLP layer in the Max Model, to match the overall number of parameters.
Evaluation Methods
We evaluate the models in two ways. First, we examine differences in the raw classification accuracy obtained by replacing Max  $G$ -Pooling with the  $G$ -TC Layer. Second, we assess the completeness of the model by optimizing "metameric" stimuli for the trained models—inputs that yield the same pre-classifier representation as a target input, but are perceptually distinct. The completeness evaluation is inspired by a recent paper that incorporates the bispectrum—the spectral dual of the triple correlation—into a neural network architecture trained to yield  $G$ -invariant representations for  $G$ -transformed data [42]. In this work, two inputs are considered "perceptually distinct" if they are not in the same group orbit. They find that all inputs optimized to yield the same representation in the bispectral model are identical up to the group action. By contrast, many metameric stimuli can be found for  $E(2)$ -CNN [50], a  $G$ -Equivariant CNN that uses Max  $G$ -Pooling. Given the duality of the bispectrum and the triple correlation, we expect to observe similar "completeness" for  $G$ -CNNs using the  $G$ -TC Layer.
5.1 Classification Performance
We train  $G$ -TC and Max  $G$ -Pooling models on the  $SO(2)$  and  $O(2)$ -MNIST and chiral  $(O)$  and full  $(O_h)$  octahedral voxelized ModelNet10 training datasets and examine their classification performance on the test set. Full training details including hyperparameters are provided in Appendix G. Table 1 shows the test classification accuracy obtained by the Max- $G$  and  $G$ -TC architectures on each dataset. Accuracy is averaged over four random seeds, with confidence intervals showing standard deviation. We find that the model equipped with  $G$ -TC obtains a significant improvement in overall classification performance—an increase of 1.3, 0.89, 1.84 and 3.49 percentage points on  $SO(2)$ -MNIST,  $O(2)$ -MNIST,  $O$ -ModelNet10 and  $O_h$ -ModelNet10 respectively.
Method	C8-CNN on SO(2)-MNIST	D16-CNN on O(2)-MNIST
Accuracy	Parameters	Accuracy	Parameters
Max G-Pool	95.23 ± 0.15	32,915	92.17% ± 0.23	224,470
G-TC	96.53 ± 0.16	35,218	93.06 % ± 0.09	221,074
Method	O-CNN on O-ModelNet10	Oh-CNN on Oh-ModelNet10
Accuracy	Parameters	Accuracy	Parameters
Max G-Pool	72.17% ± 0.95	500,198	71.73% ± 0.23	1,826,978
G-TC	74.01% ± 0.48	472,066	75.22% ± 0.62	1,817,602
Table 1: Classification Accuracy & Parameter Counts for Models Trained on G-MNIST and G-ModelNet10. Confidence intervals reflect standard deviation over four random seeds per model. The model equipped with G-TC rather than Max G-Pooling obtains significantly improved classification performance on all datasets.
5.2 Completeness
Following the analysis of [42], we next evaluate the completeness of the models trained on the  $G$ -MNIST Dataset. Figure 4 shows inputs optimized to yield the same pre-classifier representation as a set of target images. In line with similar findings from [42], we find that all inputs yielding an identical representations and classifications in the  $G$ -TC Model are within the same group orbit. Notably, the optimized images are identical to the targets, up to the group action. This reflects exactly the completeness of the  $G$ -TC: the  $G$ -TC preserves all signal structure up to the group action.
Thus, any rotated version of the a target will yield the same  $G$ -TC Layer output. By contrast, many "metameric" misclassified stimuli can be found for the Max  $G$ -Pool Model, a consequence of the lossiness of this pooling operation.
Figure 4: Optimized Model Metamers. For each model, 100 targets from the MNIST dataset were randomly selected. 100 inputs were randomly initialized and optimized to yield identical pre-classifier model presentations. All inputs optimized for the  $G$ -TC Model converge to the orbit of the target. By contrast, metamers that bear no semantic relationship to the targets are found for every target in the Max  $G$ -Pooling model.
In this work, we introduced a new method for achieving robust group-invariance in group-equivariant convolutional neural networks. Our approach, the  $G$ -TC Layer, is built on the triple correlation on groups, the lowest-degree polynomial that is a complete group-invariant map [32, 46]. Our method inherits its completeness, which provides measurable gains in robustness and classification performance as compared to the ubiquitous Max  $G$ -Pooling.
This improved robustness comes at a cost: the  $G$ -TC Layer increases the dimension of the output of a  $G$ -Convolutional layer from  $G$  to  $\frac{|G|(|G| + 1)}{2}$ . While the dimension of the discretized groups used in  $G$ -CNNs is typically small, this increase in computational cost may nonetheless deter practitioners from its use. However, there is a path to further reduction in computational complexity provided that we consider its spectral dual: the bispectrum. In [34], an algorithm is provided that exploits more subtle symmetries of the bispectrum to demonstrate that only  $|G| + 1$  terms are needed to provide a complete signature of signal structure, for the one-dimensional cyclic group. In Appendix F, we extend the computational approach from [34] to more general groups and provided a path for substantial reduction in the complexity of the  $G$ -TC Layer, thus expanding its practical utility. Novel mathematical work that grounds our proposed computations in group theory is required to quantify the exact complexity reduction that we provide.
As geometric deep learning is applied to increasingly complex data from the natural sciences [18, 2, 27], we expect robustness to play a critical role in its success. Our work is the first to introduce the general group-invariant triple correlation as a new computational primitive for geometric deep learning. We expect the mathematical foundations and experimental successes that we present here to provide a basis for rethinking the problems of invariance and robustness in deep learning architectures.
The authors thank Christopher Hillar, Bruno Olshausen, and Christian Shewmake for many conversations on the bispectrum and triple correlation, which have helped shape the ideas in this work. Thanks also to the members of the UCSB Geometric Intelligence Lab and to four anonymous reviewers for feedback on earlier versions. Lastly, the authors acknowledge financial support from the UC Noyce Initiative: UC Partnerships in Computational Transformation, NIH R01 1R01GM144965-01, and NSF Grant 2134241.
[45] Alexandros Stergiou, Ronald Poppe, and Grigorios Kalliatakis. "Refining activation downsampling with SoftPool". In: Proceedings of the IEEE/CVF international conference on computer vision. 2021, pp. 10357-10366.
[46] Bernd Sturmfels. Algorithms in invariant theory. Springer Science & Business Media, 2008.
[47] Zhiqiang Tong, Kazuyuki Aihara, and Gouhei Tanaka. “A hybrid pooling method for convolutional neural networks”. In: Neural Information Processing: 23rd International Conference, ICONIP 2016, Kyoto, Japan, October 16–21, 2016, Proceedings, Part II 23. Springer. 2016, pp. 454–461.
[48] J. Tukey. "The spectral representation and transformation properties of the higher moments of stationary time series". In: Reprinted in The Collected Works of John W. Tukey 1 (1953), pp. 165-184.
[49] Zhen Wei et al. "Building detail-sensitive semantic segmentation networks with polynomial pooling". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, pp. 7115-7123.
[50] Maurice Weiler and Gabriele Cesa. "General E(2)-Equivariant Steerable CNNs". In: Conference on Neural Information Processing Systems (NeurIPS). 2019.
[51] Haibing Wu and Xiaodong Gu. "Max-pooling dropout for regularization of convolutional neural networks". In: Neural Information Processing: 22nd International Conference, ICONIP 2015, Istanbul, Turkey, November 9-12, 2015, Proceedings, Part I 22. Springer. 2015, pp. 46-54.
[52] Zhirong Wu et al. "3d shapenets: A deep representation for volumetric shapes". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, pp. 1912-1920.
[53] John I Yellott. "Implications of triple correlation uniqueness for texture statistics and the Julesz conjecture". In: JOSA A 10.5 (1993), pp. 777-793.
[54] JI Yellott Jr and GJ Iverson. "Uniqueness theorems for generalized autocorrelations". In: Journal of the Optical Society of America 9 (1992), pp. 388-404.
[55] Dingjun Yu et al. "Mixed pooling for convolutional neural networks". In: Rough Sets and Knowledge Technology: 9th International Conference, RSKT 2014, Shanghai, China, October 24-26, 2014, Proceedings 9. Springer. 2014, pp. 364-375.
[56] Afia Zafar et al. “A Comparison of Pooling Methods for Convolutional Neural Networks”. In: Applied Sciences 12.17 (Aug. 2022), p. 8643. DOI: 10.3390/app12178643. URL: https://doi.org/10.3390/app12178643.
[57] Matthew D Zeiler and Rob Fergus. "Stochastic pooling for regularization of deep convolutional neural networks". In: arXiv preprint arXiv:1301.3557 (2013).
[58] Shuangfei Zhai et al. "S3pool: Pooling with stochastic spatial sampling". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 4970-4978.
[59] Ning Zhang, Ryan Farrell, and Trever Darrell. “Pose pooling kernels for sub-category recognition”. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE. 2012, pp. 3665–3672.
A Group Axioms
For a set of transformations  $G$  to be a group under the operation  $\cdot$ , these four axioms must hold:
1. Closure: The product of any two elements of the group is also an element of the group, i.e., for all  $g_1, g_2 \in G$ ,  $g_1 g_2 \in G$ .
2. Associativity: The grouping of elements under the operation does not change the outcome, so long as the order of elements is preserved, i.e.,  $(g_{1}g_{2})g_{3} = g_{1}(g_{2}g_{3})$ .
3. Identity: There exists a "do-nothing" identity element  $e$  that such that the product of  $e$  with any other element  $g$  returns  $g$ , i.e.,  $ge = eg = g$  for all  $g \in G$ .
4. Inverse: For every element  $g$ , there exists an inverse element  $g^{-1}$  such that the product of  $g$  and  $g^{-1}$  returns the identity, i.e.  $gg^{-1} = g^{-1}g = e$ .
B The Case of Steerable  $G$ -CNNs
We consider the framework of Steerable  $G$ -CNNs defined in [9]. Consider a group  $G$  that is the semi-direct product  $G = \mathbb{Z}_2 \ltimes H$  of the group of translations  $\mathbb{Z}_2$  and a group  $H$  of transformations that fixes the origin  $0 \in \mathbb{Z}_2$ . Consider the feature map  $\Theta : \mathbb{Z}_2 \to \mathbb{R}^K$  that is the output of a steerable  $G$ -CNN. Here,  $\Theta$  is a field that transforms according to a representation  $\pi$  induced by a representation  $\rho$  of  $H$  on the fiber  $R^K$ .
The  $G$ -TC can be defined on  $\Theta$  by replacing the regular representation by the induced representation. Specifically, replace any  $\Theta(g)$ , i.e., the scalar value of the feature map at group element  $g$  by  $\pi(g)(\Theta)(x)$ , i.e., the vector value of the feature map at  $x$  after a group element  $g$  has acted on it via the representation  $\pi$ :
$$
\tau_ {\Theta} (g _ {1}, g _ {2}) = \int_ {G} \pi (g) (\Theta) (x) ^ {\dagger} \cdot \pi (g _ {1} g) (\Theta) (x) \cdot \pi (g _ {2} g) (\Theta) (x) d g.
$$
Instead of computing the  $G$ -TC for each scalar coordinate  $k$  of  $\Theta$  as in the main text, we directly compute it as a vector. The formulation does not depend on the choice of  $x$  by homogeneity of the domain  $\mathbb{Z}_2$  for the group  $G$ . Importantly, this  $G$ -TC is invariant to the action of the induced representation, see Appendix D.
C The  $G$ -Triple Correlation: Concrete Examples
We show how to compute the  $G$ -Triple Correlation ( $G$ -TC) on three concrete groups. We start with the  $G$ -TC for the group  $\mathbb{R}^2$  of 2D translations.
Example 1. Consider the group of  $2D$  translations  $G = (\mathbb{R}^2, +)$  with the addition as the group law. Consider a signal  $\Theta$  that is a real function defined over  $\mathbb{R}^2$  and can therefore be identified with an image. For any  $x_1, x_2 \in \mathbb{R}^2$ , the  $G$ -TC is given by:
$$
\tau_ {\Theta} \left(x _ {1}, x _ {2}\right) = \int_ {x \in \mathbb {R} ^ {2}} \Theta (x) \Theta \left(x + x _ {1}\right) \Theta \left(x + x _ {2}\right) d x. \tag {9}
$$
Next, we consider the special orthogonal group  $SO(2)$  of 2D rotations.
Example 2. Consider the group of  $2D$  rotations  $G = (SO(2),\cdot)$  where  $SO(2)$  is parameterized by  $[0,2\pi]$  and the composition of rotations  $\cdot$  is the addition of angles modulo  $2\pi$ :
$$
\theta_ {1} \cdot \theta_ {2} \equiv \theta_ {1} + \theta_ {2} [ 2 \pi ]. \tag {10}
$$
For a real signal  $\Theta$  defined over  $G$ , we have:
$$
\tau_ {\Theta} \left(\theta_ {1}, \theta_ {2}\right) = \int_ {\theta \in S O (2)} \Theta (\theta) \Theta \left(\theta + \theta_ {1}\right) \Theta \left(\theta + \theta_ {2}\right) d \theta , \tag {11}
$$
for any  $\theta_1, \theta_2 \in SO(2)$  and the addition is taken modulo  $2\pi$ .
Finally, we compute the  $G$ -TC for the special euclidean group  $SE(2)$  of 2D rotations and translations, i.e., of 2D rigid-body transformations.
Example 3. Consider the group of 2D rigid body transformations  $G = SE(2) = (SO(2) \times \mathbb{R}^2, \cdot)$  equipped with the group composition law:
$$
\left(\theta_ {1}, x _ {1}\right) \cdot \left(\theta_ {2}, x _ {2}\right) \equiv \left(\theta_ {1} + \theta_ {2}, R _ {\theta_ {1}}. x _ {2} + x _ {1}\right), \tag {12}
$$
where  $R_{\theta_1} = \begin{bmatrix} \cos \theta_1 & -\sin \theta_1 \\ \sin \theta_1 & \cos \theta_1 \end{bmatrix}$  is the 2D rotation matrix associated with rotation  $\theta_1$  and the addition of angles  $\theta_1 + \theta_2$  is taken modulo  $2\pi$ .
For a real signal defined on  $SE(2)$  we have: