A General Framework for Robust G-Invariance in G-Equivariant Networks
Authors: Sophia Sanborn
Nina Miolane
Venue: NeurIPS 2023 poster (2023)
Keywords: equivariance
group-equivariant cnns
invariance
pooling
convolutional neural networks
TL;DR: We introduce a novel method for group-pooling in group-equivariant neural networks based on the triple correlation, which improves both accuracy and robustness.

Abstract: We introduce a general method for achieving robust group-invariance in group-equivariant convolutional neural networks ($G$-CNNs), which we call the $G$-triple-correlation ($G$-TC) layer. The approach leverages the theory of the triple-correlation on groups, which is the unique, lowest-degree polynomial invariant map that is also \textit{complete}. Many commonly used invariant maps\textemdash such as the \texttt{max}\textemdash are incomplete: they remove both group and signal structure. A complete invariant, by contrast, removes only the variation due to the actions of the group, while preserving all information about the structure of the signal. The completeness of the triple correlation endows the $G$-TC layer with strong robustness, which can be observed in its resistance to invariance-based adversarial attacks. In addition, we observe that it yields measurable improvements in classification accuracy over standard Max $G$-Pooling in $G$-CNN architectures. We provide a general and efficient implementation of the method for any discretized group, which requires only a table defining the group's product structure. We demonstrate the benefits of this method for $G$-CNNs defined on both commutative and non-commutative groups\textemdash $SO(2)$, $O(2)$, $SO(3)$, and $O(3)$ (discretized as the cyclic $C8$, dihedral $D16$, chiral octahedral $O$ and full octahedral $O_h$ groups)\textemdash acting on $\mathbb{R}^2$ and $\mathbb{R}^3$ on both $G$-MNIST and $G$-ModelNet10 datasets.

A General Framework for Robust G-Invariance in G-Equivariant Networks
Sophia Sanborn sanborn@ucsb. edu
Nina Miolane
www.miiib..eeuu...mmmmmm edu
Department of Electrical and Computer Engineering UC Santa Barbara
Abstract
We introduce general method for achieving robust group-invarianee in group- equivariant convolutional neural networks (G-CNNs), which we call the G-triple- correlation (G-TC) layer. The approach leverages the theory of the triple-correlation on groups, which is the unique, lowest-degree polynomial invariant map that is also complete. Many commonly used invariant maps-such such the max- are incomplete: they remove both group and signal structure. A complete invariant, by contrast, removes only the variation due to the actions of the group, while preserving all information about the structure of the signal. The completeness the triple correlation endows the G-TC layer with strong robustness, which be observed in its resistance to invariance- awvrriacc-bbased adversarial attacks. addition, observe that it yields measurable improvements in classification accuracy over standard Max G-Pooling in G-CNN architectures. We provide a general and efficient implementation of the method for any discretized group, which requires only table defining the group's 's product structure. We demonstrate the benefits of this method for G-CNNs defined on both commutative and non-commutative groups-SO(2), ((2), (3)3), and O(3) (discretized as the cyclic C8, dihedral D16, chiral octahedral O and full octahedral On groups)-acting on IR² and R on both G-MNIST and G-ModelNet10 datasets.
Introduction
The pooling operation is central the convolutional neural network (CNN). It was originally introduced in the first CNN architecture- Setteettrr-Fuuuuhhiimn Fukushima's 1980 Neocognitron [17] and remained
fixture of the model since. The Neocognitron visual cortex as a process of hierarchical feature extraction and local pooling [25] T. In both the neuroscience and CNN model, pooling is intended to serve purposes. First, it facilitates the local-to-global poarse-graining of structure in the input. Second, it facilitates invariance to local resuuting in network activations that remain similar under small perturbations of the input. In this way, CNNs construct hierarchical, multi-scale features that have increasingly large extent and increasing invariance.
directly inspired by the canonical model of the
The pooling operation traditional CNNs, typically a local max or average, has remained largely unchanged over the last forty years. The variations that have been proposed in the literature [40]56] mostly tackle its coarse-graining purpose, improve computational efficiency, or reduce overfitting, but do not seek enhance its properties with respect invariance. Both max and avg operations reasonable choices to fulfill the goal of coarse coarse-graining within CNNs and G-CNNs. However, they excessively imprecise and lossy with respect the goal of constructing robust representations of objects that are invariant only to irrelevant visual changes. Indeed, the max and avg operations invariant many natural image transformations such translations and rotations, but also
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

G-Conv
Domalla G
G-Invariant Map
5
Max G-Pooling (Standard)
Max G-Pouling is insomplele: It loses information
Correlation (Ours)
The G- ta compiete:
It segamies onlits
Filler
G-Conv
Input
Fenture Vector
Figure I: Achieving Robust G-Invariance in G-CNNs with the G-Triple- ciiple-Correeation. The output of G-Convolutional layer is equivariant to the actions of G the domain of the signal. To identify signals that equivalent up group action, the layer G-CNNs, Max G-Pooling is commonly used for this purpose. Taking the maximum of G-Convolutional cquivariant output is indeed invariant to the actions of the group. However, it is also lossy: many non-cquivalent output vectors have the same maximum. Our method-the G-Triple-Correlation Correlation is the lowest-order polynomial invariant map that is complete [46) As complete invariant, preserves all information about the signal structure, removing only the action group. Our approach thus provides new foundation achieving robust Gnlnrriiacce G-CNNs.
followed by G-Invariant map that eliminates this equivariance.
to unnatural transformations, including pixel permutations, that may destroy the image structure. This excessive invariance has been implicated in failure modes such as vulnerability to adversarial perturbations [20] 26). and a bias towards textures rather than objects (4). To overcome these challenges and enable robust and selective invariant representation learning, there is need for novel computational primitives that selectively parameterize invariant maps for natural transformations.
Many of the transformations that occur in visual scenes are due to the actions of groups. The appreciation of this fact has led the rise of group-equivarian convolutional networks (G-CNNs) (8] and the larger program of Geometric Deep Learning mathematics of group theory to attain precise generalized eoou-eqquivaiannee in convolutional network layers, the pooling operation has yet to meet its group theoretic grounding. Standardly, invariance to a group G is achieved with a simple generalization of max pooling: Max G-Pooling (8) see Fig. [ (top-right). However, this approach inevitably suffers from the lossiness of the max operation.
(6). While this field has leveraged the
Here, we unburden the pooling operation of the dual duty of invariance and yoarse-graining by un- coupling these operations into two steps that can be performed with precision. We retain the standard max and avg pooling for coarse- coarse-graining but introduce new method for robust G-invariance via the group-invariant triple correlation -see Fig. o (bottom-righh) The group-invarian triple correlation the lowest-order complete operator that can achieve exact invariance (32). As such, we propose general framework for robust G-Invariance G-Equivariant Networks. We show the advantage of this approach over standard max G-pooling in several G-CNN architectures. Our extensive experi- ments demonstrate improved scores in classification accuracy in traditional benchmark datasets as well as improved adversarial robustness.
Background
We first cover the fundamentals of group-equivariant neural networks-als, known G-CNNs, or G- C--Eqiivariant Networks before introducing the framework for G-Invrriant Pooling.
2.1 Mathematical Prerequisites
The construction G-CNNs requires mathematical prerequisites of group theory, which we recall here. The interested reader can find details in (33
Groups. group (G. ) a set G with a binary operation which we can generically call the product. The notation 91 I2 denotes the product of two elements in the set; however, it is standard to omit

---

the operator and write simply 9192 convention we adopt here. Coneretely, a group G may define a class of transformations. For example, we can consider the group of wwoddimensional rotations in the plane-the special orthogonal group S0(2) or the group of two-dimensional rotations and translations in the plane-the special euclidean group S (2). Each element of the group g E G defines a particular transformation, such as one rotation by 30 or one rotation by 90 The binary operation provides a means for combining two particular transformations example, first rotating by 30 and then rotating by 90" In mathematics, for set of transformations G to be group under the operation the four axioms of closure, associativity, identity and inverse must hold. These axioms recalled in Appendix A
Group Actions on Spaces. We detail how a transformation g can transform elements of a space, for example how a rotation of 30 indeed rotates a vector in the plane by 30". We say that the transformations g's act on (the elements of) a given space. Specifically, consider X a space, such as the plane. A group action is a function I. G x X + X that maps (g. pairs to elements of X. We say a group G acts on a space X if the following properties of the action L hold:
1. Identity: The identity of the group G "does nothing", i.e., it maps any element x E X to itself. This can be written as: [e, = x.
2. Compatibility: Two elements 91 92 E G can be combined before after the map L to yield the same result, i.e., L[g,, L[g2, x)) = [ .)) For example, rotating a 2D vector by 30 and then 40 yields the same result rotating that vector by 70 in one time.
For simplicity, we will use the shortened notation (z) to denote L(g, x) the action of the transfor- mation g on the element
Some group actions have additional properties and turn the spaces X on which they operate into homogeneous spaces. Homogeneous spaces play an important role in the definition of the G-convolution in G-CNNs, so that we recall their definition here. We say that X a homogeneous spuce for a group G if G acts transitively on Xhthat is, if for every pair 22 E X there exists an element of y E G such that (x11) = 22. The concept can be clearly illustrated by considering the surface of a sphere, the space S The sphere S is homogeneous space for 5O(3), the group of orthogonal 3 x 3 matrices with determinant one that define 3 3-dimensional rotations. Indeed, for every pair of points on the sphere, one can define 3D rotation matrix that takes one to the other.
Group Actions on Signal Spaces. We have introduced essential concepts from group theory, where a group G can act on any abstract space X. Moving towards building G-CNNs, we introduce how groups can act on spaces of signals, such as images. Formally, a signal is map f Ω - RR where Q called the domain of the signal and e denotes the number of channels. The space of signals itself is denoted L [2, R.). For example, Ω = TR22 or RR3 for 2D and 3D images. Gray-scale images have one channel (c = 1) and color images have the 3 red-green-blue channels (c 33..
Any action of a group of transformations G on a domain Z yields an action of that same group on the spaces of signals defined on that domain, i.e., on La (S2, Rt). For example, knowing that the group of 2D rotations S0(2) acts on the plane s = 2 allows us to define how SO(2) rotates 2D gray-scale images in L2 (R. RR). Concretely, the action L of a group G on the domain Q yields the following action of G on R*)
(.) = f(..g (u)), for all u E S2 and for all g E G.
We use the same notation T.g to refer to the action of the transformation y on either an element u of the domain or on a signal ff 'eefined on that domain, distinguishing them using [.] for the signal case. We note that the domain of signal can be the group itself: Ω G. In what follows, we will also consider actions on real signals defined on group, i.c., on signals such as θ 9:G: G+++ R.
Invariance and Equivariance. The concepts of group- invarianvv and equivariance are at the core of what makes the G-CNNs desirable for computer vision applications. We recall their definitions here. A function 26 : X + Y is G-invariant if (x)) = e (L,((z)) for all g E G and E X. This means that group actions on the input space have no effect on the output. Applied to the group of rotations acting on the space of 2D images X = ( R$) with Ω = IR,, this means that G-invariant function y produces an input that will stay the same for any rotated version of given signal. For example, whether the image contains the color red is invariant with respect to any rotation of that
3

---

image. A function e : i's G-equivariant if (L,(z)) (v(()) for all g G and x E X, where L and L' are two different actions of the group G. on the spaces X and Y respectively. This means that group action on the input space results in a corresponding group action of the same group element on the output space. For example, consider 3 that represents a neural network performing a
segmentation of image. is desirable for be equivariant to the group of 2D rotations. This equivariance ensures that, if the input image f is rotated by 30" then the output segmentation (f) rotates by 30" as well.
2.2 G-Equivariant Networks
G-CNNs are built from the following fundamental building blocks: G-convolution, spatial pooling, and G-pooling. The G-convolution equivariant to the action of the group G, while the G-pooling achieves G-invariance. Spatial pooling achieves coarse-graining. We review the group-specific operations here. The interested reader can find additional details in [8 10], which include the definitions of these operations using the group group-theoretic theoretic framework of principal bundles and associated vector bundles.
2.2.1 G-Convolution
In plain language, a standard translation-equivarian convolutional neural network layer sweeps filters across a signal (typically, an image). translating the filter and then taking an inner product with the signal to determine the similarity between local region and the filter. G-CNNs [8] generalize this idea, replacing translation with the action of other groups that define symmetries machine learning task-for example, rotating filter, to determine the presence of feature in various orientations.
Consider signal f defined domain f2 on which group G acts. A neural network filter is map 6: Ω + R defined with the same domain Qt and codomain IR as the signal. A G-convolutional layer is defined by a of filters {Ø1,..,,K) For given filter k, the layer performs G-convolution with the input signal f:
@x(g) = (ob () = f (w)
Vg E G,
(2)
JEE
by taking the dot product in R of the signal with a transformed version of the filter. practice, the domain Q of the signal is discretized, such that the G-convolutional layer becomes:
(x(9)=
Σ dx ( Vg G. uEN
(3)
The output of one filter ls is therefore a map O G while the output of the whole layer with K filters is @ G R defined as O(g) O .g....., Ox for all g G. The G-convolution therefore outputs a signal θ whose domain has necessarily become the group Ω = G and whose number of channels is the number of convolutional filters K.
The G-convolution is equivariant to the action of the group on the domain of the signal f (8]. That is, the action of g on the domain of f results in corresponding action the output of the layer. Specifically, consider filter Φk. we have:
de * Lg = LyJ0x= f],
Vg G.
where T. and T', represent the actions of the same group element g on the functions I and Φu * f respectively. This property applies for the G-convolutions of the first layer and of the next layers (8).
2.2.2 G-Pooling
Invariance to the action of the group is achieved by pooling over the group (G-Pooling) 8 The pooling operation is typically performed after the G-convolution, that restrict its definition to signals O defined over a group G. In G-pooling, a max typically is taken over the group elements:
Ak max (g) gEG
G-pooling extracts a single real scalar value 14 from the full feature vector O₁, which has G values, with |G] the size of the (discretized) group G as shown in Fig. [ When the group G is grid

---

discretizing R",, max G-Pooling is equivalent to the standard spatial max pooling used in translation- equivariant CNNs, and it can be used to achieve coarse -raniinn. More generally, G-Pooling is G-invariant, as shown in [8). However, we argue here that it is excessively G-invariant. Although achieves the objective of invariance to the group action, it also loses substantial information. As illustrated in Fig. 1 many different signals & may yield same result y through the G-pooling operation, even if these signals do not share semantic information. This excessive invariance creates an opportunity for adversarial susceptibility. Indeed, inputs f can be designed with the explicit purpose of generating pi that will fool a neural network and yield an unreasonable classification result. For this reason, we introduce our general framework for robust, selective G-invariance.
3 The G-Triple-Correlate Layer for Robust G-Invariance
We propose a G-Invariant layer designed for G-CNNs that is complete- that is, it preserves all information about the input signal except for the group action. Our approach leverages the theory of the triple correlation on groups [32 and applies it to the design of robust neural network architectures. Its theoretical foundations in signal processing and invariant theory allows to generally define the unique G-invariant maps of lowest polynomial order that are complete, hence providing a general framework for selective, robust G-invariance in G-CNNs (46)
3.1 The G-Triple-Correlatiin Layer
The Ttrple-Correlation (G-TC) on a real signal θ G -> IR is the integral of the signal multiplied by two independently transformed copies of it 32
(g) (go:) θ (992) dg. Te 92) = 9EG
(6)
This definition holds for any locally compact group G on which we can define the Haar measure dg used for integration purposes (28). This definition above is applicable the G-CNNs where θ is collection of scalar signals over the group. We show in Appendix B that we can extend the definition
stecrable G-CNNs where θ can be an arbitrary field (9].
In the equation above, the G-TC computed for a pair of group elements g 91. In practice, we sweep over all pairs in the group. Appendix C illustrates the triple correlation on three concrete groups Importantly, the G-triple-correlation is invariant to the action of the group G on the signal θ [28]. as shown below.
Proposition 1. Consider signal θ G + RR The G-Triple Correlatin T is G-invariant:
[L,(9] = TO, for all E G,
where Lg denotes action of a transformation on the signal O
The proof is recalled in Appendix D. We propose to achieve G-invariance a G-CNN by applying the G-Triple-Car Correl-Coorllttion (G-TC) to the output θ of G-convolutional layer. Specifically, we apply the G-TC to each real scalar valued signal θ₂ that comes from the G-convolution of filter Φk. for k (1, K}. We only omit the subscript i for clarity of notations. In practice, will use the triple correlation on discretized groups, where the integral is replaced with a summation:
Te 92) = Σ ((g))((9)))0nn))
(8)
9EG
for θ a scalar valued function defined over G. While it seems that the layer computes (e(1,, 92) for all pairs of group elements (2), we note that the real scalars ((gg1) and θ (ggg) commute so that only half of the pairs are required. We will see that we reduce the number of computations further when the group G possesses additional properties such as commutativity.
We note that the triple correlation is the spatial dual of the bispectrum, which has demonstrated robustness properties in the context of deep learning with bispectral neural networks [42]. The goal of bispectral neural networks is to learn an unknown group G from data. The bispectral layer proposed in [42) assumes an MI. architecture. Our work the first to generalize the use of bispectral invariants to convolutional networks. Here, we assume that the group G' is known in advance, and
5

---

exploit the theoretical properties of the triple correlation to achieve robust invariance, One path for future extension may be to combine our approach with the learning approach of [42]. to parameterize and learn the group G that defines a G-Equivariant and G-Invariant layer.
3.2 Selective Invariance through Completeness
We show here that the proposed G-triple-coree correlation is guaranteed to preserve all information aside from any equivariant component due to the group action on the input domain. This crucial property distinguishes our proposed layer from standard G-Pooling methods, which collapse signals and lose crucial information about the input (Figure 1T. In contrast with standard, excessively invariant G-pooling methods, we show here that our G-TC Tayer is instead selectively G-invariant thanks its completeness property [54] 29 31,, defined here:
Proposition 2. Every integrable function with compact support G is completely identified-yy up to group action by its Corriale-oorrelation. We say that the G-rriple-correlation complete.
Mathematically, an operator T is complete for group action T. if the following holds: for every pair of signals , and O2 if (ee)) = T(22) then the signals are equal up to the group action, that is: there exists group element h such that θ2 L 1
The proof of the completeness of the G-triple-correlation is only valid under precise of assump- tions [32] (Theorem 2). As we seek to integrate the G-triple-correlation to enhance robustness in neural networks, we investigate here the scope of these assumptions. First, the assumptions are not restrictive on the type of groups G that can be used. Indeed, the proof only requires the groups to be Tatsuuma duality groups and the groups of interest in this paper meet this condition. This includes all locally compact commutative groups, all compact groups including the groups of rotations, the special orthogonal groups 5O(n), and groups of translations and rotations, the special euclidean groups (En)). Second, the assumptions are not restrictive on the types of signals. Indeed, the signal only needs to be such that any of its Fourier transform coefficients are invertible. For example, when the Fourier transform coefficients are scalar values, this means that require these scalars to be non-zero. practical applications on real image data with noise, there is a probability 0 that the Fourier transform coefficients of the input signal will be exactly 0 (scalar case) or non-invertible (matrix case). This is because the group of invertible matrices is dense in the space of matrices. Therefore, this condition is also verified in the applications of interest and more generally expect the property of completeness of our G-TC layer to hold in practical neural network applications.
3.3 Uniqueness
The above two subsections prove that our G-Triple Correlation layer is selectively G-invariant. Here, we note that our proposed layer is the lowest-degree polynomial layer that can achieve this goal. In invariant theory, it is observed that the G- Triple Correlation is the only third-order polynomial invariant (up to change of basis) [46). Morcover, it is the lowest-degree polynomial invariant that is also complete. It thus provides unique and minimal-complexin solution to the problem of robust invariance within this function class.
3.4 Computational Complexity
The G-Tiiple Correlation enjoys some symmetries that leverage to avoid computing it for each pair of group elements (which would represent |G|2 computations). hence making the feedforward pass more efficient. We summarize these symmetries here.
Proposition 3. Consider two transformations yı, 92 G. The G-Triple Correlation of a real signal has the following symmetry:
Te(gi, 92) = T9(22.91)
G is commutaaivv, the G-Triple Correlation of a real signal has the following additional symmetries: Te(92 - = Te(gz' Te(92'..197" ) Te (g197 .92
Te(gi 92) = Te(gi
The proofs are given in [39 for the group of translations. We extend them to any locally compact group G in Appendix In practice, these symmetries mean that even if there are theoretically

---

[G| computations, this number immediately reduces to [ and further reduces if the group G' of interest is commutative. computational cost to linear [G] + for the case of one-dimensional cyclic groups [34] by considering the spectral dual of the G-TC: the bispectrum. We provide computational approach to extend this reduction to more general, non-commueative groups in Appendix [ The theory supporting approach has yet to be extended to this general case. Thus, there is an opportunity for new theoretical work that further increases the computational efficiency of the
addition, more subtle symmetries can be exploited reduce the
4 Related Work
The Triple Correlation. The triple correlation has long history in signal processing [48 5 39 It originally emerged from the study of the higher-order statistics of non-Gaussian random processes, but its invariance properties with respect to translation have been leveraged in texture statistics (53) and data analysis in neuroscience [13]. as well as early multi-layer perceptron architectures in the The triple correlation was extended to groups beyond translations in [32], and its completeness with respect to general compact groups was established in [30]. To the best of our knowledge, the triple correlation has not previously been introduced as method for achieving invariance in convolutional networks for either translation or more general groups.
1990's [12 33).
Pooling in CNNs. Pooling in CNNs typically has the dual objective of coarse graining and achieving local invariance. While invariance is one desiderata for the pooling mechanism, the machinery of group theory is rarely employed in the computation of the invariant map itself. As noted in the introduction, max and average pooling are by far the most common methods employed in CNNs and G-CNNs. However, some approaches beyond strict max and average pooling have been explored. addresses the lack of smoothness of the max function and uses instead a smooth approximation of it, with methods including polynomial pooling (49] and learned-norm (22), among
many others [15] 14, 43 44
3 45 11 (35). Stochastic pooling [57) reduces overfitting in CNNs
by introducing randomness
the pooling, yielding mixed-pooling [55] max pooling dropout (5T).
among others 47
58
21
Geometrically-Awwan Pooling, Some approaches have been adopted to encode spatial structural
information about the feature maps, including spatial pyramid pooling (24), part-based pooling (59),
geometric Ip pooling [16] pooling regions defined concentric circles (41). In all of these
cases, the pooling computation is still defined by a max. These geometric pooling approaches are reminiscent of the Max G-Pooling for G-CNNs introduced by [8] and defined in Section 2.2.2
without the explicit use of group theory.
Higher-Order Pooling, Average pooling computes first-order statistics (the mean) by pooling from each channel separately and does not account for the interaction between different feature maps coming from different channels. Thus, second-order pooling mechanisms have been proposed to consider correlations between features across channels [38 [9]. J. but higher-orders are not investigated. Our approach computes third-order polynomial invariant; however, it looks for higher-order correlations within the group rather than across channels and thus treats channels separately. In principle, these approaches could be combined.
5 Experiments & Results
Implementation
We implement the G-TC Layer for arbitrary discretized groups with an efficient implementation built on top of the ESCNN library 77 50). which provides general implementation of Enn)- Equivariant Steerable Convolutional Layers. The method is flexibly defined, requiring the user only to provide a (Cayley) table that defines the group's 's product structure. The code is publicly available /ttttt::///ee ///ithub. com/ sophi aasa/tt-aaaaaee invariance, Here, we demonstrate the approach on the groups S0(2), and 0(2), SO(3), and 0(3), discretized as the groups C, (cyclic), D,, (dihedral), O (chiral octahedral), and On (full octahedral), respectively. [ESCNN provides implementations for G'Conv layers on all of these E(n) subgroups.
7

---

Space
Signal
Group
Dataset
0
d
O
N
O
J 6
N
9 D
2
2
a
d
2
2
Q
J
G P V
3
8
4
5
S
X
9
O Z
W Q
Rotation
Reflection
R2a
A
&
&
N G
7
0(2)
A
Rotation
Reflection
R
0(3)
Figure 2: Datasets. The O(2)-MNIST (top) and -ModelNet10 (bottom) datasets generated by applying a random (rotation, reflection) pair to each element the original datasets. Although we visualize continuous group here, in practice, we discretize the group O(3) as the full octahedral group On to reduce computational complexity. SO(2) and S3(3) datasets are generated similarly, by applying random rotation to each datapoint.
Experimental Design
We examine the performance of the G-TC over Max G-Pooling in G- G-Equivariant Networks defined
on these groups and trained on G-Invariant classification tasks. For the groups S0(2) and ( acting on IR2, and 0(3) acting on R we use the voxelized ModelNet10 database of 3D objects (52). We generate
the MNIST dataset of handwritten characters [37), and for the groups SO(3)
G-MNIST and G-ModeeNet110 datasets by transforming the domain of each signal in the dataset by randomly sampled group element g E G' (Figure
In these experiments, train pairs of models in parameter- matched architectures, in which only the G-Pooling method differs. Note that the purpose of these experiments to compare differences in the G-TC- not achieve SOTA accuracy.
performance between models using Max G-Pooling Thus, we do not optimize the models for overall performance. Rather, we fix a simple architecture and set of hyperparameters and examine the change in performance that arises from replacing Max G-Pooling with the G-TC Layer (Figure
Softmax
Sofmaxx
MLP
Max Pool
TC Lapre
G-Conn
C-CCoor
Figure 3: Models. We compare two simple architectures comprised of a single G-Conv block followed by either a Max G-Pool layer a G-TC Layer and an MLP Classifier.
To isolate the effects of the G-Pooling method, all models are comprised of single G-Conv block followed by G-Pooling (Max or TC) and an MLP Classifier. Notably, while many G-Conv models in the literature use the semi-direct product of with R.e.. incorporating the actions of group G into standard translational convolutional without translation. Thus, we use filters the same size as the input in all models. The G-Conv block is comprised of G-Conv layer, batch norm layer, and optional nonlinearity. For the
here, we perform only pure G-Conv,

---

Max G-Pool model, ReLU is used as the nonlinearity. Given the third-order nonlinearity of the TC, we omit the nonlinearity in the G-Conv block in the TC Model. The G-TC layer increases the dimensionality of the output of the G-Conv block; consequently the input dimension of the first layer of the MIPP is larger and the weight matrix contains more parameters than for the Max G-Pool model. To compensate for this, we increase the dimension of the output of the first MLP layer in the Max Model, to match the overall number of parameters.
Evaluation Methods
We evaluate the models in two ways. First, we examine differences in the raw classification accuracy obtained by replacing Max G-Pooling with the G-TC Layer. Second, we assess the completeness of the model by optimizing "metameric" stimuli for the trained models same pre-classifier representation as a target input, but are perceptually distinct. The completeness evaluation is inspired by a recent paper that incorporates the bispectrum-the spectral dual of the neural network architecture trained to yield G-invariant representations for G-transformed data [42]. In this work, two inputs are considered "perceptually distinct" if they are not in the same group orbit. They find that all inputs optimized to yield the same representation in the bispectral model are identical up to the group action. By contrast, many metameric stimuli can be found for E(2)-CNN (50). G-Equivariant CNN that uses Max G-Pooling, Given the duality of the bispectrum and the triple correlation, we expect to observe similar "completeness" for G-CNNs using the G-TC Layer.
that yield the
triple correlation-inte
5.1 Classification Performance
We train G-TC and Max G-Pooling models on the SO(2) and ((2)-MNIST and chiral (O) and full (O₂) octahedral voxelized ModelNet10 training datasets and examine their classification performance on the test set. Full training details including hyperparameters are provided in Appendix G Table T shows the test classification accuracy obtained by the Max-G and G-TC architectures on each dataset. Accuracy is averaged over four random seeds, with confidence intervals showing standard deviation. We find that the model equipped with G-TC obtains a significant improvement overall classification performance an increase of 1.3, 0.89, 1.84 and 3.49 percentage points on S0(2)- MNIST, (2)-MNNSS, O-ModelNet10 and O Ne0 respectively.
C8-CNN on (0
D16-CNN on 0(2)-MNIST
Method
Aceuracy
Parameters
Accuracy
Parameters
Max G-Pool
95.23 0.15
32,915
92.17% + 0.23
224,470
G-TC
96.53 + 0.16
35,218
93.06 % + 0.09
221,074
O-CNN on O-ModelNet10
-N-CNN on O₂ ModdeleNee110
Method
Accuracy
Parameters
Accuracy
Parameters
Max G-Pool
72.17% = 0.95
500,198
71.73% + 0.23
1.826,978
G-TC
74.01 % + 0.48
472,066
75.22% + 0.62
1,817,602
Table 1: Classification Accuracy & Parameter Counts for Models Trained on G-MNIST and G- ModelNet10. Confidence intervals reflect standard deviation over four random seeds per model. The model equipped with G-TC rather than Max G-Pooling obtains significantly improved classification performance on all datasets.
5.2 Completeness
Following the analysis of [42] J. we next evaluate the completeness of the models trained on the G-MNIST Dataset. Figure 4 shows inputs optimized to yield the same pre-classifier representation as a set of target images. In line with similar findings from (42). we find that all inputs yielding an identical representations and classifications in the G-TC Model are within the same group orbit. Notably, the optimized images are identical to the targets, up to the group action. This reflects exactly the completness of the G-TC: the G-TC preserves all signal structure up to the group action.
4sows

---

Thus, any rotated version of the a target will yield the same G-TC Layer output. By contrast, many "metameric" misclassified stimuli can be found for the Max G-Pool Model, consequence of the lossiness of this pooling operation.
0
O
O
2
M
3
4
J 6
G
S 5
S
5
G
R
8
S
P
Q
P
O
P
N
2 2
2
5
6
Darget
Optimized Metameess
Targota
Metamers
Max G-Pool Model
Corill-Correlation Model
Figure 4: Optimized Model Metamers. For each model, 100 targets from the MNIST dataset were randomly selected. 100 inputs were randomly initalized and optimized to yield identical pre-classifier model presentations. All inputs optimized for the G-TC Model converge to the orbit of the target. By contrast, metamers that bear no semantic relationship to the targets found for every target in the Max G-Pooling model.
6 Discussion
In this work, we introduced a new method for achieving robust group-invariance in group-equivariant convolutional neural networks. Our approach, the G-TC Layer, is built on the triple correlation on groups, the lowest-degree polynomial that is complete group-invariant map [32 46). Our
method inherits its completeness, which provides measurable gains robustness and classification performance as compared to the ubiquitous Max G-Pooling.
This improved robustness comes cost: the G-TC Layer increases the dimension of the output of
G-Convolutional layer from G to [GI[G[+1] While the dimension of the discretized groups used in G-CNNs is typically small, this increase in computational cost may nonetheless deter practitioners from its use. However, there a path to further reduction in computational complexity provided that we consider its spectral dual: the bispectrum. In (34), algorithm is provided that exploits more subtle symmetries of the bispectrum to demonstrate that only |G| + terms are needed to provide a complete signature of signal structure, for the one-dimensional cyclic group. In Appendix F) we extend the computational approach from 34 to more general groups and provided a path for substantial reduction in the complexity of the G-TC Layer, thus expanding its practical utility. Novel mathematical work that grounds our proposed computations in group theory is required to quantify the exact complexity reduction that provide.
As geometric deep learning applied to increasingly complex data from the natural sciences [8 |, we expect robustness to play a critical role in its success. Our work is the first to introduce the general group-invariant triple correlation as a new computational primitive for geometric deep learning. We expect the mathematical foundations and experimental successes that we present here to provide a basis for rethinking the problems of invariance and robustness in deep learning architectures.
10

---

Acknowledgments
The authors thank Christopher Hillar, Bruno Olshausen, and Christian Shewmake for many conversa- tions on the bispectrum and triple correlation, which have helped shape the ideas in this work. Thanks also to the members of the UCSB Geometrie Intelligence Lab and four anonymous reviewers for feedback on earlier versions. Lastly, the authors acknowledge financial support from the UC Noyce Initiative: UC Partnerships in Computational Transformation, NIH RO1 IRO1GMM449655- and NSF Grant 2134241.
References
[1] Edward H. Adelson and James R Bergen. "Spatiotemporal energy models for the perception of motion". In: Josa a 2.2 (1985), pp. 284-299,
[2] Kenneth Atz, Francesca Grisoni, and Gisbert Schneider. "Geometric deep learning on molecu- lar representations" In: Nature Machine Intelligence 3.12 (2021), pp. 1023-1032.
[3] Florentin Bieder, Robin Sandkuhler, and Philippe C Cattin. "Comparison of methods general- In: arXiv preprint arXiv:2 003.07766 (2021).
izing max-and average-pooling
[4] Wieland Brendel and Matthias Bethge. "Approximating enns with hag-of-local-features models works surprisingly well on imagenet". In: arXiv preprint arXiv: 0004.90400000 00760 (2019).
[5] D Brillinger. "Some history of higher-order statistics and spectra". In: Stat. Sin. (1991), pp. 465-476.
[6] Michael M Bronstein et al. "Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges". In: arXiv preprint arXiv:2 13478 (2021).
[7] Gabriele Cesa, Leon Lang, and Maurice Weiler. "A Program Build B(N)-Equivariant Steerable CNNs". In: International Conference on Learning Representations. 2022. URL.: https: fttts:///penrevicee
[8] Taco Cohen and Max Welling. "Group equivariant convolutional networks". In: International conference on machine learning. PMLR. 2016, pp. 2990-2999.
[9] Taco S Cohen and Max Welling. "Steerable cnns". In: arXiv preprint arXiv:1612.08498 (2016).
[10] Taco S. Cohen, Mario Geiger, and Maurice Weiler. "A general theory of equivariant CNNs on homogeneous spaces". In: Advances in Neural Information Processing Systems 32.NeurIPS
(2019). ISSN: 10495258.
[11] Wojeiech Czaja al. "Maximal function pooling with applications". In: Excursions Har- monic Analysis, Volume 6: In Honor of John Benedetto's 's 80th Birthday (2021), pp. 413- 429.
[12] Anastasios Delopoulos, Andreas Tirakis, and Stefanos Kollias. "Invariant image classification using tiple-corrcllaiion-aseed neural networks". In: IEEE Transactions on Neural Networks 5.3 (1994), pp. 392-408.
[13] Sarita S Deshpande, Graham A Smith, and Wim van Drongelen. "Third-order motifs are sufficient to fully and uniquely characterize spatiotemporal neural network activity"'. In: Scientific Reports 13.1 (2023), p. 238.
[14] Hayoung Eom and Heeyoul Choi. Alpha-nntegration Pooling for Convolutional Neural Net- works. 2020. Xrvi 1811 03436 [cs.LG]
[15] Joan Bruna Estrach, Arthur Szlam, and Yann LeCun. "Signal recovery from pooling represen- tations". In: International conference on machine learning. PMLR. 2014, pp. 307-315. [16] Jiashi Feng et al. "Geometric Lp-norm feature pooling for image classification". In: CVPR
2011. IEEE. 2011. pp. 2609-2704
[17] K. Fukushima. "Neocognitron: A Self-organizing Neural Network Model for Mechanism of Pattern Recognition Unaffected by Shift in Position". In: Biological Cybernetics 36 (1980), pp. 193-202.
[18] Pablo Gainza et al. "Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning". In: Nature Methods 17.2 (2020), pp. 184-192.
[19] Zilin Gao et al. "Global second-order pooling convolutional networks". In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2019, pp. 3033-3033. [20] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples". In: arXiv preprint arXiv:1412.657. (2014).
11

---

[21]
Benjamin Graham. "Fractional max- max-poooling". In: arXiv preprint arXiv:1412.6071 (2014).
[22] Caglar Gulcehre et al. "Learned-norm pooling for deep feedforward and recurrent neural net- . In: Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part I 14. Springer. 2014, pp. 530-546.
works".
[23] Brian C Hall. "Lic groups, Lie algebras, and representations". In: Quantum Theory for Mathe- maticians. Springer, 2013, pp. 333-366.
[24] Kaiming He et al. "Spatial pyramid pooling deep convolutional networks for visual recog-
nition". In: IEEE transactions on pattern analysis and machine intelligence 37.9 (2015), pp. 1904-1916.
[25] David H Hubel and Torsten N Wiesel. "Receptive fields of single neurones in the cat's striate cortex". In: The Journal of physiology 148.3 (1959), pp. 574-591.
[26]
Hem-Henrik Jacobsen et al. "Excessive invariance causes adversarial vulnerability". In: arXiv preprint arXiv: /8/ 1.00401 (2018).
[27] Xiangyang Ju et al. "Performance of geometric deep learning pipeline for HL-LHC particle
tracking". In: The European Physical Journal C 81 (2021), pp. 1-14.
[28] R. Kakarala. "A group theoretic approach to the triple correlation". In: IEEE Workshop on higher order statistics. 1993, pp. 28-32.
[29] Ramakrishna Kakarala. "A group-theoretic approach to the triple correlation". In: [1993 Proceedings] IEEE Signal Processing Workshop Higher-Order Statistics. IEEE. 1993, pp. 28-32.
[30] Ramakrishna Kakarala. "Completeness of bispectrum on compact groups". In: arXiv preprint
arXiv:0902.01 0196 (2009).
[31] Ramakrishna Kakarala. "The bispectrum as source of phase-sensitive invariants for Fourier
descriptors: a group-theoretic approach". In: Journal of Mathematical Imaging and Vision 44.3 (2012). pp. 341-353.
[32] Ramakrishna Kakarala. "Triple correlation on groups". PhD thesis. University of California, Irvine, 1992.
[33] Stefanos D Kollias. "A multiresolution neural network approach to invariant image recogni- tion". In: Neurocomputing 12.1 (1996), pp. 35-57.
[34] R. Kondor. Group theoretical methods in machine learning. Columbia University, PhD) Thesis,
2008.
[35] Ashwani Kumar. "Ordinal pooling networks: for preserving information over shrinking feature maps". In: arXiv preprint vrXiiv:104... 02702 (2018).
[36] Y LeCun, Cortes, and C Burges. "The MNIST Dataset of Handwritten Digits (Images)". In: NYU: New York, NY, USA (1999).
[37] Yann LeCun. "The MNIST database of handwritten digits". In: http://yann. lecun. com/exdb/mnist/ (1998).
[38] Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. "Bilinear CNN models for fine- grained visual recognition". In: Proceedings of the IEEE international conference on computer vision. 2015, pp. 1449-1457.
[39] Chrysostomos I. Nikias and Jerry M Mendel. "Signal processing with higher-order spectra". In: IEEE Signal processing magazine 10.3 (1993), pp. 10-37.
[40] Rajendran Nirthika et al. "Pooling in convolutional neural networks for medical image analysis:
survey and an empirical study" In: Neural Computing and Applications 34.7 (Feb. 2022), pp. 5321-5347. DOL: 10. 1007 / =00521- - 022 06953 1007 2007/800521-022-0666 06953-8
URI.: https:://// //s:://dooo doii org 10
[41] Kunlun Qi et al. "Concentric circle pooling in deep convolutional networks for remote sensing scene classification", " In: Remote Sensing 10.6 (2018), p. 934.
[42] Sophia Sanborn et al. "Bispectral Neural Networks". In: International Conference Learning Representations (2023).
[43] Arash Shahriari and Fatih Porikli. "Multipartite pooling for deep convolutional neural net- works". In: arXiv preprint arXiv:1710.07433 (2017).
[44] Zenglin Shi, Yangdong Ye, and Yunpeng Wu. "Rank-based pooling for deep convolutional neural networks". In: Neural Networks 83 (2016), pp. 21-31.
12

---

[45] Alexandros Stergiou, Ronald Poppe, and Grigorios Kalliatakis. "Refining activation downsam- pling with SoftPool". In: Proceedings of the IEEE/CVF international conference on computer vision. 2021, pp. 10357-10366.
[46] Bernd Sturmfels. Algorithms in invariant theory. Springer Science & Business Media, 2008
[47] Zhiqiang Tong, Kazuyuki Aihara, and Gouhei Tanaka. "A hybrid pooling method for convolu- tional neural networks". In: Neural Information Processing: 23rd International Conference, ICONIP 2016, Kyoto, Japan, October 16-21, 2016, Proceedings, Part II 23. Springer. 2016, pp. 454-461.
[48] J. Tukey. "The spectral representation and transformation properties of the higher moments stationary time series". . In: Reprinted in The Collected Works of John W. Tukey (1953),
pp. 165-184.
[49] Zhen Wei al. "Building detail-sensitive semantic segmentation networks with polynomial pooling". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. 2019, pp. 7115-7123.
[50] Maurice Weiler and Gabriele Cesa. "General E(2)--quivariant Steerable CNNs". In: Confer- ence on Neural Information Processing Systems (NeurIPS). 2019.
[51] Haibing Wu and Xiaodong Gu. "Max "Max-pooling dropout for regularization of convolutional neural networks". In: Neural Information Processing: 22nd International Conference, ICONIP 2015, Istanbul, Turkey, November 9-12, 2015, Proceedings, Part 22. Springer. 2015, pp. 46- 54.
[52] Zhirong Wu et al. "3d shapenets: A deep representation for volumetric shapes". . In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, pp. 1912-1922
[53]
John Yellott. "Implications of triple correlation uniqueness for texture statistics and the Julesz conjecture". In: JOSA A 10.5 (1993), pp. 777-793.
[54] JI Yellout Jr and GJ Iverson. "Uniqueness theorems for generalized autocorrelations", In: Journal of the Optical Society of America 9 (1992), pp. 388-404.
[55] Dingjun Yu et al. "Mixed pooling for convolutional neural networks". In: Rough Sets and Knowledge Technology: 9th International Conference, RSKT 2014, Shanghai, China, October 24-26, 2014, Proceedings 9. Springer. 2014, pp. 364-375.
[56] Afia Zafar et al. "A Comparison of Pooling Methods for Convolutional Neural Networks". In: Applied Sciences 12.17 (Aug. 2022), 8643. DOT: 10. 3T.33390//pppp 25980/app12178643
URL: https
//doi. org/ 10. 3390/ app12178645
[57] Matthew D Zeiler and Rob Fergus. "Stochastic pooling for regularization of deep convolutional neural networks". In: arXiv preprint arXiv: 1301,3557 (2013).
[58] Shuangfei Zhai et al. "S3pool: Pooling with stochastic spatial sampling". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 4970-4978.
[59] Ning Zhang, Ryan Farrell, and Trever Darrell. "Pose pooling kernels for sub-category recogni- tion". In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEBE. 2012, pp. 3665-3672.
13

---

Appendices
A Group Axioms
For set of transformations G be a group under the operation these four axioms must hold:
1. Closure: The product of any two elements of the group is also element of the group, i.e., for all 91-92 E G. 9192 E G.
2. Associativity: The grouping of elements under the operation does not change the outcome, so long as the order of elements is preserved, i.e., (9192)93 - 9) (9293).
3. Identity: There exists "do-nothing" identity element e that such that the product of e with any other element y returns y, i.e., ge eg for all G.
4. Inverse: For every element g. there exists an inverse element g such that the product of and g returns the identity, i.e. 99 g cc.
B The Case of Steerable G-CNNS
We consider the framework of Steerable G-CNNs defined 19. Consider group G that is the
semi-direct product G Zz H of the group of translations Z, and a group H of transformations that fixes the origin 0 E Zz. Consider the feature map @ Zg - IR K that is the output of a steerable G-CNN. Here, θ is field that transforms according a representation æ induced by a representation of H on the fiber Rt
The G-TC can be defined on θ by replacing the regular representation by the induced representation. Specifically, replace any (gg) i.c. the scalar value of the feature map at group element g by π (g)( i... the vector value of the feature map at after group element has acted it via the representation :
Tg .2) f(e (z)) a (gig) (0)()) a (g2g) (+)(r)
Instead of computing the G-TC for each scalar coordinate & of as the main text, we directly compute it vector. The formulation does not depend the choice of by homogeneity of the domain Zg for the group G. Importantly, this G-TC is invariant to the action of the induced representation, see Appendix D
C The G-Triple Correlation: Concrete Examples
We show how to compute the G-Triple Correlation (G-TC) on three concrete groups. We start with the G-TC for the group 2 of 2D translations.
Example 1. Consider the group of 2D translations G = GRR with the addition the group law. Consider a signal θ that is real function defined over R.2 and can therefore be identified with an image. For any x1,32 E R2? the G-TC is given by:
Te(
(z)) x1) (or x2) da.
JERRS
Next, we consider the special orthogonal group SO(2) of 2D rotations.
Example 2. Consider the group 2D rotations G (SO(2).)) where SO(2) is parameterized by [0, 2m) and the composition of rotations is the addition of angles modulo 2n:
01 0.0 = 01 + (mm)
(10)
For real signal @ defined over G, we have:
Te(0 02)
(0)0
(11)
for any 01,00 SO(() and the addition is taken modulo 2m.
14

---

Finally, we compute the G-TC for the special euclidean group SE(2) of 2D rotations and translations, ie.. of 2D rigid-body transformations.
Example 3. Consider the group of 2D rigid body transformations G S.((2) (SO(2) R²,-) equipped with the group composition law:
(01, (02, = (01 + 82. Rgg .22 + I1),
(12)
cos 01 - sin 01
where Re,
is the 2D rotation matrix associated with rotation 8r and the
sin 01 cos 01
addition of angles 01 + O2 is laken module 2nt
For real signal defined on SE(2) we have:
((0,z)0 (01, ((0,,) -02,, x2)) d0da
(0.))SR)())
( (0 + 0,, Ro. x)) θ (0 + 02, Rg. + x) d0dr,
(0,))ESE(2)
for any 0, E SO(2) and 11,22 E R
D Invariance of the G-Triple Correlation
Consider a real signal O defined over a group G. The G-Triple Correlation is invariant group actions on the domain of the signal @ as shown in (28).
Proposition 4. Consider two real signals O1 O2 defined over group G. If there exists such that one signal is obtained from the other by group action, i.e., O2 Ln, 1.), then To, = Te.
We recall the proof of (28 below.
Proof. Consider two real signals 11 O2 defined over group G. such that O₂ = La (1 for group action La of group element h. We show that this implies that T0, = T0p-
Taking 91, 92 E G. we have:
1f
Te (9..91.92)
(2(g) (091) O₂ (992) dg
9EG
[1( (0g) La ( (192) dg
9EG
( (bgg:) 9 (hg92) dg
9Ea
(g))) (991) θ (gg2) dg
gEG
Te. (01.92)-
where we use the change of variable hg - g-
This proves the invariance of the G-TC with respect to group actions on the signals.
E
Symmetries of the G-Triple Correlation
The G-Triple Correlation (G-TC) enjoys some symmetries that can leverage avoid computing for each pair of group elements (which would represent [ computations), hence making the feedforward pass more efficient.
These symmetries are given in the main text. We recall them here for completeness.
Proposition 5. Consider transformations 91, 92 E G. The G-Triple Correlation of signal θ has the following symmetry:
(s1)
Toloi 92) = Te(92.91.
15

---

G is commuttatvv,, the G-Triple Correlation of a real signal has the following additional symmetries: (s2)
Tels1,80 (ol91192)= = " =e
.929
.9 (929
Te(9192 992 = Te 91922"
Our proof extends the proof given in 39 for the group of translations.
Proof. Consider two transformations 91, 02 E G.
Symmetry (s1) relies on the fact that (g92) and (gg1) are scalar values that commute:
Te(92 01)
Σ
0(g)0(mn))0(s)) =
Σ
((g)0(on))(m)))
Te
G
G
9EG
gEG
For symmetry (s2), we assume that G is commutative. We prove the first equality:
Te .9291"
(g))
()9(929))))
G
gEG
Σ
(gg')) (1929 (with g' 199 i.e.,, ..e... = g'91)
G
g'EG
((gg)))0g)))(G (G commutative: 929]" = 91 92)
G
g'EG
Σ ((gg)))))))
G
gEG
(19))(())))00)))
(e takes on real values that commute)
G
9EG
Te Te(22: (gg.9g)
The second equality of symmetry (s2) follows using (sl). The third and fourth equality of symmetry (s2) have the same proof.
This result and its proof are also valid for the extension of the G-TC that we propose in Appendix B They naturally emerge by replacing the regular representation by the induced representation in the proof above.
Specifically, consider signal O₂ = π (0o) [01] obtained from the action of go on signal O₁ We want to show that тe, = Te. The key ingredient of the proof is the change of variable within the integral So.. which follows the semi-direct product structure of G
h huo and = (hh)to +
where (bh) is matrix representing h that acts on Ze via matrix multiplication: e.g., a rotation matrix R (()) in the case of Fn)) This concludes the adaptation of the proof for the steerable case.
F Algorithmic Reduction
In this section, we show that we can reduce the complexity of the G-Triple Correlation of real signal. This computational reduction requires that we consider, instead, the spectral dual of the G-TC. the bispectrum (3 In what follows, we consider signal O defined over finite group G. The signal can real or complex valued.
F1 Reduction for Commutative Groups
Consider commutative group G. The bispectrum for signal θ is defined pair of irreducible representations P1, P2 of the group G as:
3(0) p1.02 F(O)) F(O))
C.
(13)
16

---

where F(O) is the Fourier transform of ④ that generalizes the classical Fourier transformation to signals defined over group. We note that, in group theory, the irreducible representations (irreps) of commutative groups map to scalar values. Hence, the bispectrum is complex scalar in this case.
For discrete commutative group, the number of irreps is equal to the size of the group. MkondorThhesin proved that, for cyclic groups, is enough to compute |G] + 1 bispectral co- efficients fully describe the signal & up group action, instead of the |G|2 that would otherwise be required from its definition.
F.2 Reduction for Non- Commutativee Groups
Consider non-commutative group G. The bispectrum of signal @ is defined over pair of irreducible representations P1, P2 of G as:
3(0) = (F((),, @ F(O) (
+
F(O),
C
E CDDDxxD,D;
P1-P2
pEpi Op2
where @ is the tensor product, and is direct
irreps. The unitary Clehsch-Gordan matrix
Oprse is analytically defined for each pair of representations Pl, as:
(p1 @ p2) Ct
+ p(g)
C P21.22
(14)
PEPL 002
We note that the irreps of non-commutative groups map to matrices, hence the bispectrum is complex matrix in this case.
We provide an algorithmic approach reducing the computational complexity of the bispectrum for non-commutative finite groups. We show that bispectral coefficients. That is, we can recover from coefficients 8(0) P1-02 computed for a few, well-chosen irreps P1, P2. In practice, only need compute few bispectral coefficients have complete invariant the signal @ henne reducing the computational complexity of the layer.
recover the signal ( from small subset of its
Generalizing 28), we will show that subset of bispectral coefficients allows to recover the Fourier transform of the signal for every irreducible representation p of the group. This will show that we can recover the signal O itself by applying the inverse Fourier transform.
We first show relationships between the bispectral coefficients and the Fourier coefficients of the signal θ in the following Lemma. We denote the trivial representation of the group G, which is the representation that sends every group element to the scalar
Lemma 1. Consider pa the trivial representation of the group G. Consider any other irreducible representation of dimension D. The bispectral coefficients write:
Spa.po Po (2 EC
Sp.po F(O)), E CDXD.
Here, and in what follows, denote the bispectrum of O i.e., omit the argument @ for clarity of notations.
Proof. For Po the trivial representation, and p irreps, the Clebsh-Jordan (CJ) matrix Cppo is the identity matrix, and the matrix Cppo is the scalar 1.
We compute the bispectral coefficient Bpo.poi
Spo.po = (F(O),o & FF()) C
F(0),)
C
(F(O))p & F(O) po) ) papo F(O) pa
Ct
(Po Pu - Po which irreducible)
Popo
=(F(O)² (Cpppo
Ct
(F(O) is a scalar for which tensor product is multiplication)
PaPo
(FF())
(CJ matrices are equal to 1.)
17

---

Take any irreducible representation of dimension D, we have:
Spm (F(O), & P(O)p.)CCpp
F(O), C
pp
(F(O) scalar and p Po P)
APo
=(0)
(CJ matrices are identity matrices).
Next, summarize main result.
Proposition 6. We recover the Fourier coefficients of a signal ④ from only L bispectral coeffi- cients, where L is a number computed from the Kronecker product table of the group G.
In the proof, we propose algorithmic method that iteratively computes bispectral coefficients until the Fourier coefficients of the signal are all recovered. We note that, for arbitrary groups and their representations, Clebsch-Gordan Gordan (CJ) matrices are not known in general, yet they can be computed numerically. Thus, the proof below assumes that the CJ matrices are given for the group G of interest.
Proof. Algorithmic Approach.
First, we show how we can recover the first Fourier coefficient (DC component), i.e., the Fourier transform of the signal at the trivial representation po. from single bispectral coefficient.
F(O) = f = [ (()roo())d = / @(g)dg eC
(15)
Using Lemma we can recover this Fourier component from the bispectral coefficient Spo-po as:
(F()) pll (l 1) arg arg(F(()).) arg(3pp.mo)
(16)
Next, consider an irreducible representation P1 of dimension D. We seek to recover the Fourier we can recover
coefficient p()) This Fourier coefficient is matrix in CDDD, Using Lemma I from single bispectral coefficient:
F(O) = 3p.p DDD
(17)
F(O),
since we have already recovered the Fourier coefficient F(Q) The matrix F(O)) is hermitian, and thus admits a square-root, that denote F(0)/,
Bppp F(OY, = F(O))
(18)
The square-root F(O), only corresponds to F(O), up to matrix factor. This is an unidentifiability similar the commutative case 28 Specifically, consider the singular value decomposition (SVD) of F(O),;
F(O), U.E.V
F(O) F(O) F(O),
+ =F(O)), = VEVT.
Thus, we have: F(O), = UVt ((O), where U, are unitary matrices that come from the (un- known) SVD decomposition of the (unknown) F(O),. p. By recovering F(O), as F(OY, we fix UVt = I. This is similar to the same way in which 28 fixed ø = 0 (rotation of angle ie., the identity) in the commutative case.
Next, we seek find the remaining Fourier coefficients of the signal @, from a limited subset of the bispectral coefficients. To this aim, denote R the set of irreducible representations of the group
18

---

G. We recall that, for a finite group G, the set R is also finite, with its size equal to the number of conjugacy classes of G.
We consider the following bispectral coefficient:
PPL.P = (F(O),, @ F(O)),
F(O),
C Pipi
LE e0p
mF(O)m @ Fl
F(O))
PR
where ppp. the multiplicity of irreps in the decomposition of Pl, P1. This multiplicity is known it only depends on the group, on the signal O.
We get an equation that allows us recover additional Fourier coefficients of the signal O:
F(O)) PE
Cpp (F(O) @ F())),,) C=t Cipps
(19)
where everything on the right hand side is known. Therefore, every Fourier coefficient F(0), that appears in the decomposition of pi pi into irreducible irreps p can be computed, reading off the elements of the block diagonal matrix defined by the direct sum. We recover the Fourier coefficients F(O), for which pp.p1 O.
We assume that this procedure provides at least one other Fourier coefficient, for an irreps p2, that we fix. We can then compute the following bispectral coefficient:
P .2 (F(O),, @ F(0)m) Crpp Cmip
F(O), Beip
Oprrr
LP Spz
get novel equation:
F(O)) = =01 (F()) @ F(O)pz) pipz
Bpp Cmt pip2'
(20)
PER
where everything on the right-hand side is known. Thus, every Fourier coefficient F(O), that appears in the decomposition of P1 @ P1 into irreducible irreps can be recovered, by reading off the elements of the block diagonal matrix. We get the F(O), for which np.p # O. We iterate this procedure to recover more Fourier coefficients.
Number of bispectral coefficients.
Now, we show that our procedure can indeed recover all of the Fourier coefficients of the signal O Additionally, we show that it only requires a limited number L of bispectral coefficients, where L depends on the group G. Specifically, it depends on the Kronecker product table G, which the [R] x | table of the decomposition of the tensor product of two irreducible representations into a direct sum of elementary irreps. In this table, the element at row i and column j lists all of the multiplicity of the irreps that appear in the decomposition of p; Pj-
The Kronecker table, i.e., any multiplicity my = np, of At: the decomposition أ Ps 3 Pj can be computed with procedure inspired by 99 and described below.
The procedure relies on the character X Tr(p(g)) of the representation f to be decomposed. From group theory, we know that the characters of irreps Pj are orthogonal, in the following sense:
(Xor> Xps)
Σ
Xo. (h)xx) (h) = dij.
(21)
G
hEG
Thus, we can obtain the multiplicity of irrep Pk: in f by computing the inner product with the k-th character:
(xp. Xpu) (eommpp X
Σ
miXpi Xo=
=
Σ mi (xe)) = my.
19

---

A
A
B
B3
E
A
(1,0,00.0)
(0.1,0.0.0)
(0,0, 1,000.00
0.0,0..100)
(0. (0.0,0,0.1 1)
A
0,,1,,0..0
(1.00.0.00
00,0,,1..0)
(0,0,1.0.0)
00.0,00..1)
B1
0,,0,
1,0.0
1,0,0,1,00
(1,0,,0.00
00,0,0,0.1, (0,1,0,0,0)
B2
00,0,,11,0)
(0,0,1.0.00
00,1,0,0,0)
(1,0,0.0.0)
00,0,000,1
00,0,0,0, 0,
(0, 0, 0,
(0, 0,0,0,0,7
(0,0,0,0,1)
1,,10,1,00
Table 2:
Kronecker table for the dihedral group D4 which has 5 irreps called A,, A2. B1, B2 and E.
using the fact that the trace of direct sum equals the sum of the traces (i.e. Xpep = Xp + Xo ) Thus, we can determine the Kronecker product table of interest. For example, the Kronecker product table for the dihedral group D4 shown in Table 2
The Kronecker product table shows us how many bispectral coefficients we need to complete our algorithmic procedure. Our procedure essentially uses a breadth-firg search algorithm on the space of irreducible representations, starting with P1 and using the tensor product with Pi as the mechanism explore the space. Whether this procedure succeeds in including all the irreps in R. might on the group and its Kronecker (tensor) product table. Specifically, consider all the irreps that appear in the row corresponding
P1 in the Kronecker product table. If these irreps do not cover the set R of all possible irreps, then the approach will need more than the tensor products of the form P1 @ Py to get all of the Fourier coefficients of the signal O
We observe in our experiments that this procedure does indeed succeed in computing all of the Fourier coefficients of the signal θ for most of the groups of interest. We provide detailed examples of these computations on our github repository, for the diedral groups D1, D16 and for the chiral octahedral group O. The procedure does not succeed in the case of the full octahedral group O₁, does not succeed.
For the diedral group D4, which has R irreps, our approach allows us recover signal on D4 from only bispectral coefficients, instead of 5" = 25. For the diedral group Die, which has R = 11 irreps, we recover the signal from bispectral coefficients instead of 112 121. For the octahedral group, which has R = 5 irreps, we use 4 bispectral coefficients instead of 52 = 25. This represent a substantial complexity reduction. More theoretical work is needed prove that our approach applies to wide range of discrete groups, or to further generalize it for groups such as O₁,
G Training and Implementation Details
The code to implement all models and experiments in this paper can be found at github.
For all experiments, we use near-identical, invariant map differs. To isolate the effects of the invariant map, all models are comprised of single G-Conv block followed by either Max G-Pooling or the G-TC layer, and an MLP Classifier: Here, we perform only pure G-Conv, without translation. Thus, we use filters the same size as the input in all models. The G-Conv block is comprised of a G-Conv layer, batch norm layer, and an optional nonlincarity. For the Max G-Pool model, ReLU is used as the nonlinearity. Given the third-order nonlinearity of the G-TC, we omit the nonlinearity in the G-Conv block in the G-TC Model. The G-TC layer increases the dimensionality of the output of the G-Cony block; consequently the input dimension of the first layer of the MLP is larger and the weight matrix contains more parameters than for the Max G-Pool model. To compensate for this, we increase the dimension of the output of the first MLP layer in the Max Model, to match the overall number of parameters.
architectures in which only the type of
All models are trained with a cross-entropy loss, using the Adam optimizer, learning rate of 0.00005, weight decay of 0.00001, betas of [0.9, 0.999], epsilon of 10-8, a reduce-on- conltce-o-plateauu plateau learning rate scheduler with a factor of 0.5, patience of epochs, and a minimum learning rate of 0.0.0001. Each model is trained with four random seeds [0, 1, 2, 3], and results are averaged across seeds.
20

---

G.1 MNIST Experiments: S0(2) and (22) on I??
G.1.1 Datasets
The (0(2)-MNIST dataset is generated by applying random 2D planar rotation to each digit in the MNIST (36] training and test datasets. This results in training and test sets with the standard sizes of 60, 000 and 10, 000. For the ((2)-MNIST, each image is randomly flipped and rotated i.e. transformed by a random element of the group O(2). A random 20% of the training dataset is set aside for model validation and is used to tune hyperparameters. The remaining 80% used for training. Images are additionally downsized with interpolation to 16 x 16 pixels.
G.1.2 Models and Training
C8-CNN
TC. The G-Conv block consists of an C8-Conv block using 24 filters followed by Batch Norm. Next, the G-TC Layer applied. The output is raveled and passed to three-layer MLP using 1d Batch Norm and ELU nonlinearity and after each linear layer, with all three layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
Max. The G-Conv block consists of an C8-Conv block using 24 filters followed by Batch Norm and ReLU nonlinearity. Next, Max G-Pooling Layer is applied. The output is raveled and passed to three-layer MLP using 1d Batch Norm and ELU nonlinearity and after each linear layer. The first layer has output dimension 275, to compensate for the difference in output size of the G-TC Layer. The remaining two layers having output dimension 64, Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
D16-CNN
TC. The G-Conv block consists of an D16-Conv block using 24 filters followed by Batch Norm. Next, the G-TC Layer applied. The output is raveled and passed to a three-layer MLP using Id Batch Norm and ELU nonlinearity and after each linear layer, with all three layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
Max. The G-Conv block consists of an D16-Conv block using 24 filters followed by Batch Norm and ReLU nonlinearity. Next, a Max G-Pooling Layer is applied. The output is raveled and passed to a three-layer MI.P using 1d Batch Norm and ELU nonlinearity and after each linear layer. The first layer has output dimension 2, 380, to compensate for the difference in output size of the G-TC Layer. The remaining two layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
G.2 ModelNet10 Experiments: O and O₁ acting on R
G.2.1 Datasets
The ModelNet10 dataset is downsampled and voxelized to a 10 x 10 x 10 grid. for the O-ModelNet10 Dataset, each datapoint is transformed by a random cubic rotation. For the Og-ModelNett0 Dataset, each datapoint is transformed by a random cubic rotation and flip. The standard training and testing sets are used. A random 20% of the training dataset is set aside for model validation and is used to tune hyperparameters. The remaining 80% is used for training.
G.2.2 Models and Training
O-CNN
TC. The G-Conv block consists of an O-Conv block using 24 filters followed by a IID 3D Batch Norm Layer. Next, the G-TC Layer is applied. The output is raveled and passed to a three-layer MLP using 1d Batch Norm and ELU nonlinearity and after each linear layer, with all three layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
Max. The G-Conv block consists of an O-Conv block using 24 filters followed by a IID 3D Batch Norm Layer and ReLU nonlinearity. Next, a Max G-Pooling Layer is applied. The output raveled and passed to three-layer MLP using 1d Batch Norm and ELU nonlinearity and after each linear
21

---

layer. The first layer has output dimension 5, 420, to compensate for the difference in output size of the G-TC Layer. The remaining two layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
Og-CNN
TC. The G-Conv block consists an Oş-Conv block using 24 filters followed by a IID 3D Batch Norm Layer. Next, the G-TC Layer is applied. The output is raveled and passed to three-layer MLP, with all three layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
Max. The G-Conv block consists of -n-Conv block using 24 filters followed by IID 3D Batch Norm I Layer and ReLU nonlinearity. Next, Max G-Pooling Layer is applied. The output is raveled and passed to a three-layer MI.P. The first layer has output dimension 20, 000, to compensate for the difference in output size of the G-TC Layer. The remaining two layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
22


Reviews:
[{"review_id": "k3PALb0Vmv", "reviewer": "Reviewer_E2w2", "summary": "This work proposes to use the triple correlation to achive the group invariance. Unlike max pooling, the triple correlation preserves the entire information from the original signal, which may improve the quality of solutions for downstream tasks. Experiments show effectiveness of the proposed method.", "questions": "1. Since the triple correlation corresponds to bispectrum in frequency space, would it make more sense to optimize in Fourier space (or polar Fourier)?\n\n2. Is the proposed method robust to noisy images? If I look at (8), multiplying three noisy terms may magnify the noise.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method looks very natural. The proposed triple correlation layer achives the group invariance and at the same time preserves the entire information from the original signal. As far as I undertand from this paper, it seems to be the only group-invariant operation that enjoys this nice property in the deep learning literature.\n2. The results from experiments look quite encouraging.", "weaknesses": "1. The novelty of the proposed method seem to be quite limited. It uses a well-known operation and apply it to NN.\n2. The computational complexity of the proposed layer look quite high. Although the authors are able to save half of operations by using symmetry, the order of computation is still O(|G|^2).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer E2w2, \n\nThank you for your review. We appreciate your attention and address your comments and questions below:\n\n**Novelty.** Although the triple-correlation in its translation-invariant form has a long history in signal processing, it is relatively unknown in deep learning. Google scholar reveals only a single paper (from the 90’s) that incorporates the triple-correlation (TC) into neural networks, in a very different context ([Delopoulos, et al, 1994](https://ieeexplore.ieee.org/abstract/document/286911?casa_token=6t3n5fTLMdEAAAAA:BpPufERn_ymV6UkxKtcd3GTb48yVSpfQtoP728NdQIAfcHCU3vOcC3YPxwgovRP3NhfkaO4YXA)). A handful of papers incorporate the bispectrum (BS) into deep learning, but largely as a pre-processing step rather than an architectural feature – with the exception of [Sanborn, et al, (ICLR 2023)](https://arxiv.org/abs/2209.03416). All of these papers use the generic translation-invariant form of the TC/BS – with the exception again of Sanborn, et al, (2023), which is the first to make use of the general group-invariant BS/TC applied to the problem of learning groups from data. Our work is the first to draw the connection between the group-invariant TC and the problem of *pooling and robustness in CNNs*. Here, we propose a novel computational primitive that exploits the structure of the G-convolution to construct invariant maps that significantly improve model accuracy and robustness over standard methods. Although the TC is not new, many great ideas have come from applying the right mathematics to a problem in computer science – such as the idea of a CNN itself :)\n\n**Computational Complexity.** Please see the global response regarding computational complexity.\n\n**Fourier Space.** Indeed, mathematically equivalent computations could be constructed in Fourier space using the bispectrum, the dual of the triple correlation in the frequency domain. The primary reason for performing the computation in the spatial domain is for compatibility with G-convolutional architectures, which live in the spatial domain. Spatial convolution is the dominant paradigm for computer vision, an only a handful of architectures perform equivalent computations in the frequency domain (e.g. Clebsch-Gordan Nets, Kondor et al, 2018). Nonetheless, our pooling method could be formulated in Fourier space for use in these architectures. In fact, Fourier space has some advantages with regard to computational complexity. In particular, for commutative groups, it permits an $O(|G|)$ reduction in the computation of the bispectrum (see [Kondor (2008) Group Theoretic Methods in Machine Learning, pg. 85](https://www.proquest.com/openview/d8ae7d9e47c87bb0acb05bf06ae8b6aa/1?pq-origsite=gscholar&cbl=18750)). For this paper, we chose the spatial formulation to maintain relevance to the existing literature. However, we believe the Fourier formulation is a promising avenue for future work.\n\n**Robustness to Noise.** Great question. The TC actually has special properties with respect to noise. In particular, the TC of a Gaussian signal is exactly zero. For this reason, it was originally used as a way to measure the “non-gaussianity” of a signal (See [Signal Processing with Higher-Order Spectra](https://ieeexplore.ieee.org/abstract/document/221324?casa_token=o59mtEpxSygAAAAA:TTSrggJvk8YjlWE_6UhREEMiZugsAm3OzD9OhCeyOTmp58nmk8-Lfn_zIaWpYScHKsjCzP7rNw) for more details). Thus, the TC has the nice property that it eliminates (Gaussian) noise. More generally, so long as the magnitude of the noise is $ < 1$, it will be reduced by the triple product rather than amplified. Keeping noise within this range can be encouraged with appropriate data normalization.\n\n**Questions for the Reviewer:**\nWe note that you provided ratings of “2” (fair) for both soundness and contribution. Are there concrete results or analyses you like to see that would improve the quality of our paper in these categories? \n\nAgain, we thank you for your time and attention! We look forward to discussing these topics further. \n"}, {"review_id": "JBKWbqHFW9", "reviewer": "Reviewer_NbDh", "summary": "This paper introduces a concept called G-triple-correlation (G-TC) layers, which are used in combination with G-equivariant convolutional layers to achieve G-invariant mapping functions. In contrast to conventional methods such as max or average G-poolings, which result in information loss, the proposed G-TC layers are completely identifiable: they selectively remove only the variations caused by group actions while preserving all the essential content information in images. The invariance and completeness of G-TC layers are proven under certain mild assumptions. The proposed G-TC layers are empirically validated for some representative examples, e.g., SO(2)-MNIST, O(2)-MNIST, SO(3)-ModelNet, and O(3)-ModelNet.\n\n", "questions": "- In line 302, the authors mention as \"For the Max G-Pool model, ReLU is used as the nonlinearity. Given the third-order nonlinearity of the TC, we omit the nonlinearity in the G-Conv block in the TC Model.\" How would the performance be affected if ReLU is applied to G-Conv with G-TC?\n\n- In line 303, \"The *G-Pool* layer increases the dimensionality of the output of the G-Conv block\". It seems that *G-TC* is correct?", "limitations": "Computational cost might be a bottleneck for the proposed G-TC method. This limitation is adequately addresed in the paper.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper has a clear motivation, i.e., replacing the excessive invariance (G-pooling) with more informative and precise approach (G-TC).\n\n* The proposed method is technically sound. The assumptions used in this paper seems to be mild. The useful theoretical properties (G-invariance, completness, uniqueness) and computational complexity of the proposed method are nicely presented.\n\n* The paper is generally well-written and easy-to-follow. Figure 1 is a nice overview for the motivation of this paper.\n\n* The experimental results are convincing, especially for the cases of 3-dimensional groups.", "weaknesses": "- As the authors mentioned, the main bottleneck of the proposed G-TC method is its computational cost.  Although the paper includes a theoretical analysis on the time complexity of G-TC, it would be beneficial if the study also presented experimental comparisons of the training and inference computational costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NbDh, \n\nWe thank you for the thoughtful review, and address your comments and questions below:\n\n**Computational Complexity**. Please see the global response on computational complexity.\n\n**ReLU & Nonlinearity**. For the G-TC model, we remove the ReLU that is standardly performed after the convolution because it *breaks completeness*. By pushing all negative values to zero, the ReLU operation eliminates information that the TC computation needs to retain completeness. Importantly, we find that removing the ReLU for the G-TC model doesn’t reduce classification performance. Rather, it improves it, due to the completeness property. The completeness be observed in the adversarial example experiments. For a G-TC model without ReLU nonlinearity, no out-of-orbit adversarial examples can be found (i.e. Figure 2 of the paper). By contrast, when ReLU is used directly prior to the G-TC layer, adversarial examples can be found – in line with theoretical expectations. \n\n**G-Pool Typo**. Thanks for catching this! We have updated it in the paper.\n"}, {"review_id": "zCJpN7HrFI", "reviewer": "Reviewer_Axj4", "summary": "This paper proposes a $G$-triple-correlation ($G$-TC) pooling layer in order to achieve group-invariance in group-equivariant convlutional neural networks (G-CNNs). Compared to the widely used $G$-pooling layers, the proposed $G$-TC layer is supposed to be \"complete\" in the sense that it removes only the variation due to the actions of the group, while preserving all information about the structure of the signal. In other words, the $G$-TC layer is injective over different group orbits. The authors claim that this property endows the $G$-TC layer with strong robustness, leading to resistance to invariance-based adversarial attacks.", "questions": "Please refer to the previous sectioin.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is relatively well written and well organized. It is easy to read.", "weaknesses": "1. The proposed method is only applicable to \"regular\" G-CNNs, where feature maps are signals over a group. It is hard to see how this can be extended to steerable G-CNNs where feature maps can be arbitrary fields.\n2. One particular disadvantage of \"regular\" G-CNN is its computational and memory burden -- one has to physically store a function over the group $G$, which, after discretization, could be very large. The feature dimension after the proposed $G$-TC layer increases from $|G|$ to $|G|^2$. Even if there are potential ways to reduce the computational cost of the $G$-TC layer as speculated by the authors, this memory cost can not be circumvented.\n3. The authors have claimed that the \"completeness\" of the $G$-TC layer leads to robustness to invariance-based adversarial attacks. However, there is neither theoretical nor empirical evidence to verify this claim.\n4. Also, I am not completely convinced why \"completeness\" is so important. $G$-pooling is typically adopted close to the end of the model, by which time \"coarse-graining\" of the feature is usually intentional.\n5. Also, can completeness be achieved by simple registration of the feature maps (e.g., according to the max-magnitude function value over the group)? In this case, the signal can stay at the length of $|G|$.\n6. The experimental set up of this paper is non-canonical. There are too many adjustment to the baseline (e.g., same filter size as the input, only one $G$-Conv block, etc.) To make the results more convincing, the authors should simply replace the last $G$-max pooling layer of a E(2)-CNN with the proposed $G$-TC layer. I understand the authors are not trying to achieve SOTA, but the results displayed in Table 1 are simply too unconvincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Axj4,\n\nThank you for the time you’ve taken to read and review our paper. We address your comments and questions below:\n\n**Generalization to Steerable G-CNNs**\n\nPlease see the global response on steerable G-CNNs.\n\n**Computational Complexity**  \n\nPlease see the global response on computational complexity.\n\n**Completeness & Robustness**\n\n*Theoretical evidence*. The completeness of the triple-correlation (TC) is well-established in the signal processing literature: It was first demonstrated by Yellott & Iverson (1992) for the case of the translation-invariant TC, and later generalized to the TC on compact groups by Kakarala (2009). We provide a statement of the completeness property in Section 3.2 Proposition 2, and point the reader to the original references for the proof, which is lengthy and requires substantial space and background to establish. As the G-TC Layer directly performs a triple correlation, it follows that the G-TC layer is also complete. Since the G-TC Layer is complete, it follows definitionally that it is robust to invariance-based adversarial attacks. In an invariance-based adversarial attack, the goal is to find inputs that are non-identical (up to group action) that yield identical output representations. Completeness states that the only inputs that yield identical outputs are identical up to group action. Thus completeness directly implies robustness to invariance-based adversarial attacks. Since this line of reasoning was not clear to the reviewer from our paper, we plan to make it much more explicit in the main body of the paper. We will also include an appendix that replicates the proof of completeness.\n\nWe provide *empirical evidence* for the completeness of the G-TC layer in Section 5.2 and Figure 2, where we perform invariance-based adversarial attacks on the models trained in Section 5.1. We find empirically that our G-TC model is complete, whereas the Max G-Pool model is not. That is, when we optimize inputs to yield points identical to a target image in the output space of the G-TC model, every optimized input is identical to the target image up to group action. This analysis was first established by Sanborn, et al (ICLR 2023) and provides empirical demonstration that the theoretically-expected completeness property holds for the trained model. As this empirical demonstration of completeness is presented in the main paper along with a figure illustrating the results, we are not sure why it was missed by the reviewer. Do you have recommendations on how to improve its presentation? \n\n**The Importance of Completeness.**\n\nA key point that we emphasize in our paper is the distinction between *coarse-graining* and *invariance*. Standardly, pooling has been used in CNNs to accomplish both objectives. However, the concepts of coarse-graining and invariance are distinct and serve different purposes. We elaborate on this distinction in paragraphs 1- 4 of the introduction. In our paper, we introduce a layer that focuses on the problem of invariance – *not* coarse-graining. Spatial coarse-graining is still important, and standard pooling approaches can be used to achieve this. However, at the end of a G-CNN, the goal is to achieve invariance to the group action, to remove the G-equivariance preserved throughout the layers of the network and output a representation that is “canonicalized.” It is in this setting that completeness becomes important. \n\nThe standard approach for achieving invariance at the output of a G-CNN is to perform Max G-Pooling, which returns the maximum output value over the group. This operation loses substantial information, as many non-identical inputs may yield identical max outputs. We provide an illustration of this problem in Figure 1 of the paper. Empirically we demonstrate that the lossiness of Max G-Pooling results in lower classification accuracy than the G-TC model when all else remains fixed. Thus, model performance reflects a tangible and substantial benefit of completeness.\n\n**Registration of Feature Maps.**\n\nThe idea of registering the feature maps is an interesting alternative to the TC approach we take here. It sounds like your suggestion is something like: Take the maximum value of the output of the group, use that to perform registration – e.g. shift the outputs so that the max value appears in the first position of the vector (or something like this). This is an interesting idea worth exploring in future work. We foresee a few possible issues that may arise with this approach: First, if there are multiple equivalent (or similar) maximum values, there is an ambiguity as to how to register. In practice, due to variations in data, images from the same category that are not exactly identical up to transformation will yield different feature vectors and so the maximum is likely not a reliable guide to finding the canonical registration. Nevertheless, it would be worth experimenting to see if these issues matter in practice.\n\nGenerally speaking, registration is a heuristic-based approach. Our work takes a mathematically-motivated approach that possesses theoretical guarantees. Future work may seek to test its limits: Are the theoretical gains worth the computational cost in comparison to a good-enough heuristic-based approach?\n\n**Non-Canonical Experimental Setup**\n\nIn this work, we chose to test our new layer in what we see to be the simplest setting: a model possessing a single G-Conv layer with no translational convolution (only convolution on the group $G$). We agree it is a somewhat non-canonical setup. However, since we used an identical setting for both models and changed only the pooling method between them, we can still draw valid conclusions about the impact of that change. Our results show an unambiguous significant increase in accuracy with our pooling layer, across all datasets, averaged over many random seeds.\n"}, {"review_id": "urqkfv1WKk", "reviewer": "Reviewer_aA5W", "summary": "The paper introduces a novel method to enhance robustness and group-invariance in group-equivariant convolutional neural networks (G-CNNs), named the G-triple correlation (G-TC) layer. This approach uses the concept of triple correlation on groups, a polynomial invariant map that is also complete, unlike many other invariant maps like the max. This \"completeness\" only removes variations caused by group actions but retains the signal's structure, contributing to the G-TC layer's strong robustness, particularly against invariance-based adversarial attacks. The authors also state that the G-TC layer results in improved classification accuracy over the standard Max G-Pooling in G-CNN architectures. An efficient implementation method for any discretized group is provided, and benefits are demonstrated on various groups and datasets.\n\nThe paper provides the context, highlighting the central role of the pooling operation in CNNs. The pooling operation has remained unchanged over the years, and while it serves its purpose of local-to-global coarse-graining of structure, it fails to construct robust representations that are invariant only to irrelevant visual changes. The paper then goes on to explain the role of group-equivariant convolutional networks (G-CNNs) and how they exploit group theory to achieve precise generalized group-equivariance. However, the pooling operation in G-CNNs is still susceptible to the lossiness of the max operation. To tackle this, the authors propose uncoupling the invariance and coarse-graining operations and introducing a new method for robust G-invariance through the group-invariant triple correlation. The authors demonstrate the superior performance of this approach through experiments showing improved classification accuracy.", "questions": "1. Please include a plot on runtime analysis for the training of a network of G-TC vs. a regular G-CNN with max pooling.\n", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper tackles the loss of information of the pooling operation in equivariant architectures and how to circumvent it. I really appreciated the fresh take on the importance of pooling and how even a very standard architectural block can still be improved. As a result, the paper brought to my attention the triple correlation operator which to my knowledge has yet to be extensively used in ML applications. Thus, this paper has a novel contribution that has the potential to benefit the community. Furthermore, I found the overall presentation and motivation of this paper exceptionally clear and thus it was an easy-to-read and parse paper. Finally, the empirical results do suggest that the introduced G-TC layer does have some benefits---although this analysis is a bit underdeveloped. ", "weaknesses": "There are a few notable weaknesses of this paper. The first being that the proposed approach can only work on discrete groups due to the pairwise nature of the G-TC layer. Many of the current breeds of equivariant networks have found application domains with continuous groups such as $SO(3)$, $SE(3)$, etc ... but the current setup does not seem easy to extend here. Can the authors maybe comment on this or if this can be even a direction for future work?\n\nFurthermore, the scaling of this is $O(|G|^2)$---I understand the reduction by a factor of 2---but this is still quadratic scaling. This means that many potential groups of potential interest are eliminated. For example, one finite group that is not considered in this paper but could possibly be used is the symmetric group of $N$ elements---i.e. $S_N$. But I believe the computational complexity here makes it not possible to scale this up to permutation networks that also have invariance. Can the authors comment on this?\n\nThe background section on signals should really be in the established language of fiber bundles. This has already been done multiple times in equivariant ML papers see E(2)-CNN, ESCNN, \"A General Theory of Equivariant CNNs on Homogenous Spaces\" Cohen et. al 2018. \n\n\nThe experiments in this paper are very limited to toy datasets in MNIST. I believe there is an avenue for multiple more important datasets that are larger scale. For example, E(2)-CNN has Cifar-10 but even this dataset is missing here. I encourage the authors to try larger-scale image datasets to show the benefits of their proposed approach.\n\n\n**Minor**:\n- Typo: Eqn 4. LHS your $g$ is written as $\\tilde{g}$ but the paragraph below assumes regular $g$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer aA5W,\n\nWe thank you for the thoughtful review. We respond to your comments and questions below:\n\n**Discrete Groups**\n\nWe thank the reviewer for mentioning the very interesting question of how to generalize our method to continuous groups. We first note that many of the papers that make use of continuous groups require choosing a discretization. In our paper, we apply our model to the cases of $SO(3)$ and $O(3)$ using the octahedral discretization of the groups, which is standardly used in many G-CNN papers.\n\nSteerable G-CNNs are the exception, as their theory applies to continuous groups. In the global response on Steerable G-CNNs, we show that the G-TC Layer can be naturally generalized to this setting. However, we note that the pairwise nature of the G-TC still requires us to discretize $G$. We have brainstormed a few ideas on how to overcome this. We emphasize that these ideas could also help reduce the computational cost of the G-TC layer, since the computation would no longer depend on the size of a discrete group.\n\n- Idea 1: Consider the (continuous) Lie group H. The G-TC on steerable functions leverages the action of $H$, we could think of considering instead the action of its Lie algebra $\\mathfrak{h}$. We would then consider the generators of $\\mathfrak{h}$ and show that computing the G-TC on the generators is enough. However, the complicated equation of the G-TC possibly makes this approach intractable.\n\n- Idea 2: We could take inspiration from steerable CNNs, which express any (learnable) equivariant convolutional filter bank as (learnable) coefficients on an equivariant basis computed offline. Similarly, we could think of expressing the G-TC as (fixed) coefficients on a basis of invariant operators. However, the convolution is a linear operator while the G-TC is not. Consequently, the linear algebra used for steerable CNNs do not apply.\n\n- Idea 3: Lastly, we could think about using implicit representations. Instead of computing the values of the G-TC on any possible pair $(g_1, g_2)$ we could instead represent it as a neural network, depending on $f$, that takes $(g_1, g_2)$ as input. However, it is unclear whether it would be a convenient data structure to work with. It is possible that the complexity cost of the TC would only be shifted into a computational cost of the NN.\n\nAlthough the generalization to the continuous case is not immediately obvious, we believe the above ideas could provide fruitful paths for exploration in future work. We believe, nonetheless, that presenting the theory for discrete groups is an important first step. In equivariant deep learning, the paper *Group-Equivariant Convolution Networks* (Cohen & Welling, 2016) introduced the G-CNN for discrete groups, and only later did the paper *Steerable CNNs* (Cohen & Welling 2017) present a theory that applies to continuous groups (while the authors still used the octahedral discretization in their experiments).\n\n**Computational Complexity \\& Runtime Analysis** \n\nPlease see the global response on computational complexity.\n\n**Fiber Bundle Formulation** \n\nWe agree with the suggestion to present the background in the language of fiber bundles, as it is has become standard in the G-CNN literature. We did not do so in the first draft to avoid overcomplication for the reader, but think it will be worth it for consistency, and plan to make these edits in the final draft of the paper. Thank you for the suggestion!\n\n**Toy Datasets** \n\nWe agree that the datasets we use here (MNIST and ModelNet10) are relatively simplistic. We made the decision to use simple datasets in our analyses because we can exactly control their mathematical structure. We generate our synthetic datasets by applying the actions of specific groups ($SO(2), O(2), SO(3), O(3)$) to these data exemplars. Thus, we know exactly the groups that structure the data, and can formulate theoretical expectations for the application of the G-TC layer. The CIFAR image dataset, by contrast, possesses many variations that are not attributable to group actions -- e.g. lighting changes, general differences in appearance across exemplars from the same category. Thus, theoretically, one expects less benefit from the use of group-invariant and group-equivariant structures with this dataset. Our goal in this work was to demonstrate empirically the theoretically expected properties of our proposed layer, which we think we have accomplished with the use of simple, controlled datasets.\n\n\n**Typo**\n\nThanks for catching this! We’ve updated it in the paper.\n"}]

Author rebuttal:
We thank the four reviewers for their time and attention. We have provided responses to each reviewer for unique points made in the review. Here, we highlight a few points that appear across reviews, to avoid redundancy.

**Generalization to Steerable G-CNNs** 

We thank reviewer Axj4 for bringing up the very interesting idea of generalizing the G-TC to steerable G-CNNs, which emerges naturally from our framework -- as described here.

Consider a group $G$ that is the semi-direct product $G = \mathbb{Z}_2 \ltimes H$ of the group of translations $\mathbb{Z}_2$ and a group $H$ of transformations that fixes the origin $0 \in \mathbb{Z}_2$. Consider the feature map $\Theta: \mathbb{Z}_2  → \mathbb{R}^K$ that is the output of a steerable G-CNN. Here, $\Theta$ is a field that transforms according to a representation $\pi$ induced by a representation $\rho$ of $H$ on the fiber $R^K$ (Steerable CNNs. Cohen & Welling, 2017).

The G-TC can be applied to $\Theta$ by replacing the regular representation by the induced representation in Eq.(6). Specifically, replace any $\Theta(g)$, i.e. the scalar value of the feature map at group element g by $\pi(g)(\Theta)(x)$ i.e. the vector value of the feature map at $x$ after a group element $g$ has acted on it via the representation $\pi$:

$$T(\Theta) = \int_G  \pi(g)(\Theta)(x)^\dagger . \pi(g_1g)(\Theta)(x) . \pi(g_2g)(\Theta)(x) dg$$

Instead of computing the TC for each scalar coordinate k of $\Theta$, we directly compute it as a vector. The formulation does not depend on the choice of $x$ by homogeneity of the domain $\mathbb{Z}_2$ for the group $G$. Importantly, this TC is invariant to the action of the induced representation: the proof leverages the same arguments that the ones used in our Appendix. Consider a feature map $f_2 = \pi(g_0)[f]$ obtained from the action of $g_0$ on a feature map $f$, and show that $TC(f_2) = TC(f)$. The key ingredient of the proof is the change of variable within the integral $\int_G$, which follows the semi-direct product structure of G:

$h’ = h h_0$ and $t’ = \phi(h)t_0 + t$

where $\phi(h)$ is a matrix representing h that acts on $\mathbb{Z}_2$ via matrix multiplication: e.g. a rotation matrix $R = \phi(r)$ in the case of $SE(n)$. This concludes the proof.

In short, the definition of the G-TC and the proof of its invariance naturally emerge by replacing the regular representation by the induced representation. We thank the reviewer for this remark and would be happy to highlight this natural extension to steerable G-CNNs as an avenue for future work.

**Computational Cost** 

Indeed, the TC has quadratic complexity that scales with the size of the group: $O(|G|^2)$. We acknowledge and describe this as a limitation in the paper. We note that it is quadratic in the size of the *group* -- not the size of the input. 

Practical use of the TC can be limited to “small” groups and enjoy the benefits provided by our approach. For the groups of rotation, this means limiting to rotations discretized into multiples of $d$ degrees. This is the framework used in many equivariant neural networks (Group Convolution Neural Networks, Cohen & Welling 2016, experiments of Steerable CNNs, Cohen & Welling 2017, etc). For the groups of translations, this means limiting to images defined on a small grid, or to feature maps that have been coarse-grained and are now defined on a small grid. Coarse-graining is typically applied during earlier stages of the network, whereas the group-invariant G-TC layer is applied at the end of the network, at which point the signal is heavily coarse-grained and substantially reduced in size. Many groups of interest can discretized coarsely in G-CNNs without considerable loss of accuracy, and this makes the computational cost of the G-TC in practice less impactful. 

This can be seen in the empirical time required for 1,000 training epochs for the G-TC and standard Max G-Pool model. For this analysis, we use the octahedral (rotational and full) models trained on the ModelNet10 dataset, averaged over all random seeds, and trained on a NVIDIA GeForce GTX 1080 Ti 12GB GPU. 1k epoch training times can be found below:

**Rotational Octahedral |G| = 24**

- Max G-Pool: 22.5 minutes
- G-TC: 37.5 minutes

**Full Octahedral |G| = 48**
- Max G-Pool: 29 minutes
- G-TC: 76 minutes

We note that the octahedral groups are relatively large for discretized groups used in practice. Still, the G-TC models only result in 1.67x and 2.62x increases in training time over the Max G-Pool models. We believe that there are many scenarios in which the strong improvements in classification accuracy (Table 1), and the completeness of the model (Figure 2) will be worth this increased cost. 

One reviewer (aA5W) asked whether our approach could be applied to the symmetric group $S_n$ of permutations of $n$ elements, given its computational complexity. The size of the symmetric group itself is $n!$. This typically limits *any* computation on the full group, except in the regime of small $n$. Indeed, the TC would be faced with $O(n!^2)$ computations. Thus, we acknowledge that any exponentially-sized groups will pose a problem for the TC, except in the regime of small $n$. Nonetheless, many groups used in practice--such as the rotation, translation, and reflection groups--are continuous groups that possess natural discretizations that limit their size and thus the complexity of the TC in practice. 

We would be happy to further emphasize this limitation in the paper, if the reviewers feel that it would help the readers and future users decide on how they can best integrate the G-TC in their applications. Additionally, we refer the reviewers to our answer to aA5W regarding the extension to continuous groups. There, we have included ideas that have the potential to reduce the computational cost in future work.


Meta-review: The paper proposes a novel G-triple-correlation (G-TC) layer to achieve group-invariance in group-equivariant convolutional neural networks (G-CNNs), addressing the issue of information loss associated with traditional pooling operations. The method is technically sound and well-motivated, offering a fresh perspective on an established architectural component. While the reviewers commend the clarity of presentation and the innovative concept, they note limitations in computational cost as well as a need for further empirical validation in more benchmark datasets. Despite these weaknesses, the overall contribution is considered novel and potentially useful. Therefore, the paper is recommended for acceptance.
Decision: Accept (poster)