Bispectral Neural Networks
Authors: Sophia Sanborn
Christian A Shewmake
Bruno Olshausen
Christopher J. Hillar
Venue: ICLR 2023 poster (2023)
Keywords: invariance
group theory
representation theory
geometry
representation learning
symmetry

Abstract: We present a neural network architecture, Bispectral Neural Networks (BNNs) for learning representations that are invariant to the actions of compact commutative groups on the space over which a signal is defined. The model incorporates the ansatz of the bispectrum, an analytically defined group invariant that is complete -- that is, it preserves all signal structure while removing only the variation due to group actions. Here, we demonstrate that BNNs are able to simultaneously learn groups, their irreducible representations, and corresponding equivariant and complete-invariant maps purely from the symmetries implicit in data.  Further, we demonstrate that the completeness property endows these networks with strong invariance-based adversarial robustness. This work establishes Bispectral Neural Networks as a powerful computational primitive for robust invariant representation learning.

Published as a conference paper at ICLR 2023
BISPECTRAL NEURAL NETWORKS
Sophia Sanborn': 1.2
Christian Shewmake shewmake@berkellee edu edu
sanborn@berkeley..o
Bruno Olshausen
Christopher Hillar hi billarmath@gmaa com
baol.siausen@berkeleyyyc edu
Redwood Center for Theoretical Neuroscience Universiit California, Berkeley 3 "Awecom, Inc.
ABSTRACT
We present neural network architecture, Bispectral Neural Networks (BNNs) for learning representations that are invariant to the actions of compact commuta- tive groups on the space over which a signal is defined. The model incorporates the ansatz of the bispectrum, an analytically defined group invariant that is com- plete-that that is, it preserves all signal structure while removing only the variation due to group actions. Here, demonstrate that BNNs are able to simultaneously learn groups, their irreducible representations, and corresponding equivariant and complete-invarian invariant maps purely from the symmetries implicit in data. Further, we demonstrate that the completeness property endows these networks with strong invariance-based adversarial robustness. This work establishes Bispectral Neural Networks as a powerful computational primitive for robust invariant representation learning.
INTRODUCTION
fundamental problem of intelligence model the transformation structure of the natural world. In the context of vision, translation, rotation, and scaling define symmetries of object categorization-tle transformations that leave perceived object identity invariant. audition, pitch and timbre define symmetries of speech recognition. Biological neural systems have learned these symmetries from the statistics of the natural the problem of learning symmetries in artificial neural networks.
through evolution or accumulated experience. Here, we tackle
At the heart of the challenge lie two requirements that are frequently in tension: invariance to transformation structure and selectivity to pattern structure. In deep networks. operations such as max or average are commonly employed to achieve invariance to local transformations. Such operations invariant to many natural transformations; however, they are also invariant unnatural transformations that destroy image structure, such as pixel permutations. This lack of selectivity may contribute failure modes such as susceptibility adversarial perturbations ([], excessive invariance (2). and selectivity for textures rather than objects [3]. Thus, there is a need for computa- tional primitives that selectively parameterize natural transformations and facilitate robust invariant representation learning.
An ideal invariant would be complete. complete invariant preserves all pattern information and is invariant only to specific transformations relevant to task. Transformations in many datasets arise from the geometric structure of the natural world. The mathematics of groups and their associated objects give us the machinery precisely define, represent, and parameterize these transformations, and hence the problem of invariant representation learning. In this work, we present a novel neural network primitive based the bispectrum, complete invariant map rooted in harmonic analysis and group representation theory [4].
Bispectral Neural Networks flexibly parameterize the bispectrum for arbitrary compact commutative groups, enabling both the group and the invariant map to be learned from data. The architecture is remarkably simple. It is comprised of two layers: single learnable linear layer, followed by fixed

---

Published as a conference paper at ICLR 2023
collection of triple products computed from the output of the previous layer. BNNs are trained with an objective function consisting of two terms: one that collapses all transformations of pattern to a single point in the output (invariance), and another that prevents information collapse in the first layer (selectivity).
We demonstrate that BNNs trained separate orbit classes in augmented data learn the group its Fourier transform, and corresponding bispectrum purely from the symmetries implicit in the data (Section 4.1) Because the model has learned the fundamental structure of the group, we show that it generalizes to novel, out-of-distribution classes with the same group structure and facilitates downstream group-invariant classification (Section 4.2). Further, we demonstrate that the trained network inherits the completeness of the analytical model, which endows the network with strong adversarial robustness (Section 4.3). Finally, we demonstrate that the weights of the network can be used to recover the group Cayley table-the the fundamental signature of a group's 's structure (Section 4.4). Thus, an explicit model of the group can be learned and extracted from the network weights. To our knowledge, our work is the first to demonstrate that either bispectrum can be learned from data alone. Our results set the foundation of a new computational primitive for robust and interpretable representation learning.
group Cayley table
1.1 RELATED WORK
The great success and efficiency of convolutional neural networks owe much to built-in equivariance to the group of 2D translations. In recent years, an interest in generalizing convolution to non- Euclidean domains such as graphs and 3D surfaces has led to the incorporation of additional group symmetries into deep learning architectures (5). [6]. In another line of work, Kakarala (7) and Kondor [8] pioneered the of the analytical group-invariant bispectrum in signal processing and machine learning contexts. Both of these approaches require specifying the group of transformations a priori and explicitly building its structure into the network architecture. However, the groups that structure natural data are often either unknown or too complex to specify analytically for example, when the structure arises from the interaction of many groups, or when the group acts on latent features in data.
Rather than building in groups by hand, another line of work has sought to learn underlying group structure solely from symmetries contained in data. The majority of these approaches use structured models that parameterize irreducible representations or Lie algebra generators, which act on the data through the group exponential map 9 [14. In these models, the objective is typically to infer the group element that acts template to generate the observed data, with inference accomplished through Expectation Maximization or other Bayesian approaches. A drawback of these models is that both the exponential map and inference schemes are computationally expensive. Thus, they approach [13] learns distributions
difficult to integrate into large-scale systems. A recent over the group of 2D affine transformations in a deep learning architecture. However, the group not learned in entirety, as it is restricted to a distribution on a pre-specified group.
Here, we present a novel approach for learning groups from data in an efficient, interpretable, and fully feed-forward model that requires no prior knowledge of the group, computation of the exponential map, or Bayesian inference. The key insight in our approach is to harness the generality of the form of the group-in invariant bispectrum, which can be defined for arbitrary compact groups. Here, we focus on the class of compact commutative groups.
2 THE BISPECTRUM
The theory of groups and their representations provides natural framework for constructing compu- tational primitives for robust machine learning systems. We provide one-page introduction to these mathematical foundations in Appendix [A] Extensive treatment of these concepts can be found the textbooks of Hall (16) and Gallier & Quaintance 177.. We now define the concepts essential to this work: invariance, equivariance, orbits, complete invariance, and irreducible representations.
Let G' be group, X space which G acts, and a signal defined on the domain X. The orbit of a signal is the set generated by acting on the domain of the signal with each group element, i.c. {f (gx)) g G). In the context of image transformations this is the set of all transformed versions of a canonical image for example, if G is the group S0(2) of 2D rotations, then the orbit contains all rotated versions of that image.
A function 6 X + Y is G-equivariant if o(gz) = g' ő(x) for all g E G and x E X, with g' E G' homomorphic to G. That is, a group transformation on the input space results in a corresponding
2

---

Published as a conference paper at ICLR 2023
group transformation the output space. A function 6 X - Y is G-invariant if d(x) = d(gx) for all g E G and æ E X. This means that a group action on the imput space has no effect on the output. We say Ø is complete invariant when ((21) = (x22) if and only if 22 gr, for some 9 G and 31, 22 E X. A complete invariant collapses each orbit to a single point while also guaranteeing that different orbits map to different points. In the context of oMation-invarian vision, for example, this would mean that the only information lost through the map Φ is that of the specific "pose" of the object.
A representation of a group G is a map p G - GL(V) that assigns elements of G to elements of the group of linear transformations (e.g. matrices) over vector space V such that p(9192) = p(91) p(())) That is, p is group homomorphism. In our case, V is C". A representation is reducible if there exists a change of basis that decomposes the representation into a direct sum of other representations. An irreducible representation is one that cannot be further reduced in this manner, and the set Trr(G) are often simply called the irreps of G. The irreducible representations of all finite commutative groups are mie-iimeniional (therefore equivalent to the characters of the irreps), and in bijection with the group itself.
2.1 FOURIER ANALYSIS ON GROUPS
powerful approach for constructing invariants leverages the group theoretic generalization of the classical Fourier Transform [18). Let f(g)= G - R be signal group G, and let Irr(G) denote the irreducible representations of the group. We assume that p E Irr(G) are also unitary representations (i.e., p = p), with p the conjugate transpose). The Generalized Fourier Transform (GFT) the linear map + f, in which Fourier frequencies are indexed by p E Irr(G):
Po. = Σ ((g)
(1)
gEG
Readers familiar with the classical Fourier transform should identify p(g) with the exponential term in the usual sum or integral. Here, they arise naturally the representations of Z/nă or St. Intuitively, these p are the "fundamental frequencies" of the group. In particular, for G 1i/nZ-the group of integers modulo n n-they
they are the classical n discrete Fourier frequencies: ps (g) = e 12rkg/" for k 0...., 1 and g Z/nZ. For compact continuous groups G, the sum above is replaced by an integral f p( ff (() (u)du over a Haar measure du for the group
Importantly, the Fourier transform is equivariant to the action of translation the domain of the input-a fundamental, if not defining, property. That is, ft = f(g t) is the translation of by t G, its Fourier transform becomes:
(2) For the classical Fourier transform, this is the familiar Fourier shift property: translation of the input results in phase rotations of the complex Fourier coefficients. This property is exploited in one well-known Fourier invariant- the power spectrum:
=
= f.fo
The power spectrum preserves only the magnitude of the Fourier coefficients, eliminating the phase information entirely by multiplying each coefficient by its conjugate:
df 12nkks fx) (e (2-2kkkt//
ft 9k.
Although this approach eliminates variations due to translations of the input, it also eliminates much of the signal structure, which is largely contained in relative phase relationships [19]. Thus, the power
spectrum is incomplete. As a result, it is easy to create "adversarial examples" demonstrating the incompleteness of this invariant: any two images with identical Fourier magnitudes but non-identical phase have the same power spectra (Appendices) C and D)
2.2 THE BISPECTRUM
The bispectrum is a lesser known Fourier invariant parameterized by pairs of frequencies in the Fourier domain. The transltioo-invariint bispectrum for ID signals E R2 is the complex matrix B E x
B = -i
where f (fo.....f-1) is the ID Fourier transform of f and the sum i i j is taken modulo n. Intuitively, bispectral coefficients B. indicate the strength of couplings between different phase
3

---

Published as a conference paper at ICLR 2023
components in the data, preserving the phase strueture that the power spectrum loses. Again, the equivariance property of the GFT is key to B's invariance. Like the power spectrum, it cancels out phase shifts due to translation. However it does while preserving the relative phase structure of the signal. This remarkable property may be seen as follows:
B B2
12 f fR, -12rkzt/" e ( t22
12nktt/n 12kkt//n j jankst/n fi.. frz botkit Bk
Unlike the power spectrum, the bispectrum (for compact groups) is complete and can be used to reconstruct the original signal up to translation [20]. More generally, the bispectrum is defined for arbitary compact groups:
Br.,p = Sp. f
(5)
with the addition modulo n replaced by the group product. For non-commutative groups, the bispectrum has a similar structure, but is modified to accommodate the fact that the irreps of non- commutative groups map to matrices rather than sealars (Appendix BB)
Historically, the bispectrum emerged from the study of the higher-order statistics of non-Gaussian random processes, and was used as a tool to measure the Gaussiannty signal (23). The work of Yellott and Iverson (20) established that every integrable function with compact support is completely identified-ut up to translation- -by its three-point autocorrelation and thus its bispectrum (Appendix(F) IF). Later work of Kakarala tied the bispectrum to the more general formulation of Fourier analysis and thus defined it for all compact groups [4] (7]. Kakarala went on to prove that it is complete [4 in both commutative and non-commutative formulations (Appendix) D
3 BISPECTRAL NEURAL NETWORKS
We draw inspiration from the group-invariant bispectrum define Bispectral Neural Networks, neural network architecture that parameterizes the ansatz of the bispectrum. The key idea behind this approach constrain the optimization problem by building in the functional form of solution known to exist. Importantly, we build in only the generic functional form and allow the network to learn the appropriate parameters (and thus the group over which it is define) from data. We draw upon two key properties of the bispectrum, which allow it to be naturally parameterized and learned:
1. The bispectrum is computed on a group-cquivarian Fourier transform. The Fourier transform is linear map and can be written as W x. with W a matrix of irreducible representations the Discrete Fourier Transform (DFT) matrix, in the classical case. If the group is unknown, we can treat W as a matrix of parameters be learned through optimization. The bispectrum computation is identical regardless of the structure of the irreps; thus we can use a single network architecture learn finite approximations of arbitrary compact commutative groups.
2. The bispectrum separates orbits. Here, computation as the criterion to optimize the parameters defining the group-equivariant Fourier transform. This requires knowing only which datapoints are in the same orbit (but not the group that generates the orbit), and is similar in spirit to contrastive learning [25
orbit separation at the output of the bispectrum
3.1 MODEL ARCHITECTURE
Concretely, given input vector E R the Bispectral Neural Network module is completely specified by one complex matrix W E Caxn, where each row W; E C plays the role of an irreducible representation. This matrix defines the linear transformation:
2 Wr.
(6)
W defines a Fourier transform on G. then each element of z a coefficient on a different irreducible representation of G. The analytical bispectrum then be computed from triples of coefficients as:
(7) However, if the group is unknown, then we do not know the product structure of the group and thus do not know a priori which element the vector z corresponds to the (ij)th irrep. Conveniently, for commutative groups, all irreducible representations in bijection with the group elements. That is,
B =
4

---

Published as a conference paper at ICLR 2023
each irreducible representation be uniquely identified with a group element. Consequently, the (ij)"h representation can obtained from the element- wise multiplication of the 2th and jth irreducible representations. Thus, the (conjugated) (ij)th coefficient can be obtained as
(W, W
with @ indicating the Hadamard (element-wist product and the complex conjugate. Plugging this into Equation| 7 and expanding the terms, we obtain the Bispectral Network:
(x)( Wax Wz (W; ① W x
Figure [ Illustrates the computation. Each linear term Wa an inner product yielding scalar value, and the three terms are combined with scalar multiplication in C. Note that this equation shows the computation of single scalar output 8t.. In practice, this is computed for all non-redundant pairs.
Weight Matrix
Input
Linear Transform
Triple Product
Output
Wxx
W
x 8d
VM
y= W/r
B.. = 22
W
W C
o
mww W/
20 -M wDtx 2, W0
W
Figure 1: Bispectral Neural Networks. A single linear neural network layer parameterized by W generates the output Wx. Pairs of coefficients and zy and the corresponding weights W; and W, are then used to compute the output of the network 9e =.zjz with æy computed as (W; @ W2)æ. Here, the weights are depicted as the canonical Fourier basis on s for illustrative purposes. In practice, the weights are randomly initialized, but expected to converge the irreps of the group that structures the dataset.
While the bispectrum a third-order polynomial, it results in only n² rather than n products, due to the constraint on the third term. Moreover, the bispectrum has many symmetries and is thus highly redundant. The most obvious symmetry reflection over the diagonal-i.e. 3r,, By.s. However, there are many more (see Appendix G.. Here, we compute only the upper triangular of the matrix.
As the Bispectral Network computes third-order products of the input, the output values can be both very large and very small, which can result in numerical instability during optimization. To combat this, we normalize the output of the network the unit sphere C":
((x)
3(x)=
(10)
[x)
We demonstrate in Appendix D that the analytical bispectrum can be normalized to the unit sphere while preserving completeness up to scalar factor (Appendix [ Theorem E.3). Thus, we similarly expect that this normalization will not impact the robustness of the network.
3.2 ORBIT SEPARATION Loss
Given dataset X = {01, ...; } with latent group structure and labels Y = -V11.... Mm indicating which datapoints are equivalent up to transformation (but without information about the transformation itself, i.c. how they are related), we wish to learn an invariant map that collapses elements of the same group orbit and separates distinet orbits. Critically, we want this map to be both invariant to group actions and selective to pattern structure-thar is, to be complete [24). A network that successfully learns such a map will have implicitly learned the group that structures transformations in the data.
To that end, we introduce the Orbit Separation Loss. This loss encourages the network to map elements of the same orbit to the same point the representation space, while ensuring degenerate solutions (i.e. mapping all inputs to the same point) avoided:
L(x.)
Σ 113(z.) B(zb) + 7||2. W zall2
(11)
The first term measures the Euclidean distance in the output space between points in the same orbit. Thus, minimizing the loss encourages them to be closer. In the second term, the input x, is reconstructed by inverting the linear transform WtWr,.. x., The second term is thus measure of
5

---

Published as a conference paper at ICLR 2023
whether information is preserved in the map. During training, each row vector in the weight matrix W is normalized to unit length after every gradient update, which pushes W towards the manifold of unitary matrices. The coefficient y weights the contribution of the reconstruction error to the loss.
4 EXPERIMENTS AND ANALYSES
We now test the capacity of Bispectral Networks to learn group by learning separate orbit classes. We conduct four experiments to analyze the properties of the architecture: learning Fourier transforms on groups (Section 4.11. group- inoup-invariant classification (Section 4.2p, model completeness (Section and extracting the Cayley table (Section 4.4).
4.3
4.1 LEARNING FOURIER TRANSFORMS ON GROUPS
We first examine whether Bispectral Networks trained to separate orbit classes will learn the Fourier transform and corresponding bispectrum of the group that structures the data. Here, we test the network on two groups, which act on the image grid to generate image orbits. The first is one whose bispectrum is known and well-undee cell-understood:: the group S x S of 2D cyclic translations, i.e. the canonical 2D Fourier transform. The second is the group SO(2). To our knowledge, a bispectrum on SO(2) acting on the grid (or disk) has not been defined the mathematics literature, as the domain is not a homogeneous space for the group-an an assumption fundamental to much of bispectrum theory. However, a Fourier transform for 50(2) acting on the disk has been defined (26) using the orthogonal basis of disk harmonics. We thus expect our weight matrix W' to converge to the canonical Fourier basis for the group Sa St, and hypothesize the emergence of disk harmonics for SO(2).
Models are trained on datasets consisting of 100 randomly sampled (16, 16) natural image patches from the van Hateren dataset (27] that have been transformed by the two groups to generate orbit classes. Further details about the dataset can be found in Appendix H Networks trained to convergence on the Orbit Separation Loss, Equation TT All hyperparameters and other details about the training procedure can be found in Appendix !|]
Remarkably, we find that the relatively loose constraints of the orbit separation loss and the product structure of the architecture is sufficient to encourage the network to learn a group-equivariant Fourier transform and its corresponding bispectrum from these small datasets. This is observed in the structure of the network weights, which converge to the irreducible representations of the group in the case of S x S', and filters resembling the disk harmonics for SO(2)) (Figure 2 Importantly. we observe that the learned linear map (Equation 6) equivariant to the action of the group on the input, and the output of the network (Equation T0p is invariant to the group action on arbitrary input signals. We visualize this in Figure 3 for transformed exemplars from the MNIST dataset, which the model was not trained on, demonstrating the capacity of the model to generalize these properties to out-of -itto---iitribution data.
s x S
SO(2)
Figure 2: Filters Learned in Bispectral Networks. Real components of weights learned in the Bispectral Network trained on the transformed van Hatcren Natural Images datasets, for S x S' 2D Translation (left) and SO(2) 2D Rotation dataset (right). The full set of learned weights can be found in Appendix
4.2 GROUP-INVARIAN CLASSIFICATION
We next examine the utility of the rotation model learned in Section 4.1 for facilitating performance on a downstream image classification task: digit classification for the Rotated MNIST dataset. For this experiment, we append a simple multi-layer fully-connected ReLU network (MLP; the "classification network") to the output of the Bispectral Network and train it on the classification objective. While the Bispectral Network learns to eliminate factors of variation due group actions, the classification network learns to eliminate other variations within digit classes irrelevant to classification, i.e. those due to bandwriting style. We train the classification model using the scheme described in Appendix [I] We compare the performance of our model to four leading approaches to group- group-invariant invariant classification, in addition Linear Support Vector Machine (SVM) model that train as a baseline (Table T.

---

Published as a conference paper at ICLR 2023
s x s!
SO(2)
Model Output Linear Output
Orbit 1
Orhit 2
Ozbit 3
Orbit
Orbic
Orbit 3
Figure 3: Model Outputs on Transformed Data. Real components of the outputs of the linear transform (Wa top row) and full network output (Eq. [T]] bottom row). Data imputs cyclically translated//ottaed in each transformation. Each colored line represents the output of a single neuron, with the activation value on the y-axis. In the top row, only four neuron outputs are visualized, for clarity. The cquivariance of the first layer is observed in the sinuisoidal responses, which reveal equivariant phase shifts each neuron's output as the input transforms. The invariance of the network output observed constant response across translated or rotated images.
direction (X-axis) and layer/network outputs are computed
Our model obtains competitive 98.1% classification accuracy on this task-within 0.2 1.2% of the tate-offtthe-at methods compared to here. It also possesses several notable differences and advantages to the models presented in this table. First, our model learns the group from data, in contrast to Harmonic Networks and E2CNNs-whic build in (0(2))equivariance and Augerino, which learns a distribution over the pre-specified group of 2D affine transformations. Although method is marginally outperformed by these models in terms of classification accuracy. outperforms [13), the only other model that fully learns a group. Second, our model is fully feed- forward, and it does not rely the computationally intensive EM inference scheme used in [13]. or the exponential map as used in 13 and [5). Finally, in the next section, we demonstrate that our model is approximately complete. By contrast, demonstrate, representations within the leading models E2CNN and Augerino exhibit excessive invariance.
Model
Learns Group
Fully Feedforward
Approximately Complete
RoIMNIST Accuracy
Linear SVM (Baseline)
N
NA
54.7%
Cohen & Welling (2014)
13
Y
N
97%
Harmonic Networks (2017)
28
N
Y
98.3%
Augerino (2020)
15
Y
Y
N 98.9%
E2CNN (2019)
29
N
Y
N
99.3%
Bispectral Network + MLP
Y
Y
Y
98. 1 %
Table 1: Model Comparison ROEMNIST. Bispectral Networks achieve 98. 1% classification accuracy on RoMMNIST. The model is marginally outperformed by three models that require the partial complete specification of the group structure in their design. Our model, by contrast learns the group entirely from data. Further, our model is approximately complete, while the leading models E2CNN and Augerino exhibit excessive invariance (Figure 4
This number was estimated from the har plot in Figure 4 of (13) and approved by the first author of the personal communication. The model [15 leans distribution pre specilied group. is differeni kind [13 and model, which learn the group itself.
4.3 MODEL COMPLETENESS
A key property of the analytical bispectrum is its completeness: only signals that are exactly equal up to the group action will map to the same point in bispectrum space. Here, we analyze the completeness of the trained models, using Invariance-basee to identify model metamers pnetamers-inputs that yield identical outputs in the model's 's representation space, but perceptually dissimilar in the input space. For approximately complete model, expect
Adversarial Attacks (IAAs) (2) as proxy. IAAs aim
7

---

Published as a conference paper at ICLR 2023
all inputs that map to the same point in model representation space to belong to the same orbit. That is, the only perturbations in pixel space that will give rise to identical model representations are those due to the group action.
We test the completeness of the trained models as follows. A batch of targets {21, .... NN J. a E R" randomly drawn from the RoMMNIST dataset. A batch of inputs (f1.-..,NN) F E Ih' randomly initialized, with pixels drawn from a standard normal distribution. Both the targets and random inputs are passed through the network to yield (1x;) (target) and ((x;) (adversary). The following objective is minimized with respect to the Jy:
L Σ (1 3
(12)
An optimized input &; is considered a model metamer if yields a model representation within e of 3(x,), is classified equivalently T₁. but is not in the orbit of z;. Here, we aim minimize distance between targets and adversaries in the model representation space-ic. the output the Bispectral Network, prior to the classification model. For the comparison models E2CNN and Augerino, the output of the layer prior to the softmax.
Figure 4 shows the targets and the optimized adversarial inputs for the Bispectral Network trained on rotation in the van Hateren dataset, and the E2CNN and Augerino models trained on Rotated MNIST. For the Bispectral Network, we observe that all random initializations consistently converge to an element within the orbit of the target, indicating that the invariant map learned by the model close to complete. By contrast, many unrelated or noisy metameric images can be found for E2CNN and Augerino. These metamers yield identical model representations (up to e) and identical classifications as the targets, but are not contained in their orbits of the targets. Interestingly, the metamers optimized for Augerino possess structure that at times resembles features in the target image. We roughly estimate the percentage of these "perceptually similar" inputs as 35%. The model nonetheless exhibits excessive invariance and is not complete.
E2CNN
Augerino
Bispectral Networks
8
3
6
7
3
d
6
!
7
Targets
S
O
O
+
3
5
P
O
d
F
b
O
5
X
6
X
3
6
^
3
J
5
5
5
H
Optimized Imputs
O
/
O
9
6
6
Classifted us target: In tergot orbit:
100%
Classifted as target:
100%
Classified as target:
100%
0%
In target arbit:
0% In target orbit:
100%
Percepiually similar:
-35%
Targets and optimized inputs for (left) E2CNN, (middle) Augerino, and (right) Bispectral Networks. Images are randomly initialized and optimized to yield representations identical to target image. If model possesses excessive invariance, there exist images that will yield equivalent representations while being non -uuivvllllnnt up to the group action. For Bispectral Networks, all optimized inputs are within the orbits of the targets, indicating approximate completenes. By contrast, many model metamers are found for E2CNN and Augerino.
Figure 4: Completeness in Bispectral Networks.
4.4 LEARNING THE GROUP: EXTRACTING THE CAYLEY TABLE
Finally, demonstrate that the converged network weights can be used to construct the group lisel--ice the product structure the group. as captured by the group's 's Cayley table. For finite groups, Cayley table provides complete specification of group structure. Here, we demonstrate
8

---

Published as a conference paper at ICLR 2023
novel method for constructing a Cayley table from learned imreps, described in in Appendix K which exploits the bijection between irreps and group elements in commutative groups.
We test this approach on all unique finite commutative groups of order eight: Z/82, Z/4Z/ Z/2Z. and 2/22 x 2/22 x 2 For each group, we generate synthetic dataset consisting of 100 random functions over the group, f G - I,, with values in IR drawn from standard normal distribution, and generate the orbit of each exemplar under the group. Networks are trained to convergence on each dataset using the Orbit Separation Loss, Equation [1] Details about the training procedure can be found in Appendix I.4] After model has converged, we use its learned weights to compute a Cayley table using Algorthm in Appendix K We check the isomorphism of the learned Cayley
table with the ground truth using Algorithm in Appendix
K
5
5
3
5
5
5
G
Z/8Z
Z/4Z X Z/2Z
Z/2Z x /2Z X Z/2Z
Figure 5: Learned Cayley Tables for All Unique Groups of Order Eight. Rows and columns are labeled with integers that index group elements. Each cell of the table contains the index of the group element obtained by composing the group elements indexed in the and column. The Cayley table thus contains all information about the group structure. Bispectral Networks perfectly recover the Cayley tables for all groups tested here.
Strikingly, we are able to recover the exact structure of the groups that generated the data orbits from the converged models' weights, in the form of the Cayley tables displayed in Figure 5 For each dataset, the recovered Cayley tables are identical to the known ground-trute Cayley tables. To our knowledge, this is the first demonstration that an explicit representation of the product structure of an unknown group be learned from data. Our results on all finite commutative groups of order eight suggest that this will generalize more broadly to all finite commutative groups
5 DISCUSSION
In this work, we introduced novel method for learning groups from data in neural network architecture that simultaneously learns a group-equivariant Fourier transform and its corresponding group-invariant bispectrum. The result is an interpretable model that learns the irreducible repre- sentations of the group. which allow us to recover an explicit model of the product structure of the group-i.. i.c. the group itself, in the form of its Cayley table. The resulting trained models are both powerful and robust-capable capable of generalizing to arbitrary unseen datasets that are structured by the same transformation group, while exhibiting approximate completeness. To our knowledge, these results provide the first demonstration that a bispectrum or Cayley table can be learned simply from the symmetries in data alone. We view the work presented here as a starting point for learning the more complex groups needed to recognize 3D objects in natural imagery. In ongoing work, we are extending the model to the non-com non-commutative form of the bispectrum (Appendix B). and developing a localized version amenable stacking hierarchically in deep networks.
The precise convergence of the Bispectral Network parameters to canonical forms from representation theory-under the relatively weak constraints of the third-order model ansatz and the orbit separation loss-suggests the uniqueness of this solution within the model class. Indeed, the Fourier solution likely guaranteed by the uniqueness of the bispectrum as third-order polynomial invariant (30). Future work should explore the bounds of these guarantees in the context of statistical learning. We venture to suggest further that the uniqueness and completeness of the bispectrum make it compelling candidate computational primitive for invariant perception in artificial and biological systems alike. Indeed, results from across areas of visual cortex suggest that group representations may play an important role in the formation of visual representations in the brain (3T). Models like the one presented here may provide concrete hypotheses to test in future mathematically motivated neuroscience research.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
The code to implement all models and experiments in this paper can be found at We provide files containing the trained models, configs containing the hyperparameters used in these experiments, notebooks to generate the visualizations in the paper, and download links for the datasets used in these experiments.
github com/ sophiaas hatiilass/iiipeee wtw.ppppttaa-eeetott......
ACKNOWLEDGMENTS
The authors thank Nina Miolane, Khanh Dao Due, Giovanni Luca Marchetti, David Klindt, Mathilde Papillon, Adele Myers, Taco Cohen, and Ramakrishna Kakarala for helpful suggestions, feedback, and proofreading.
REFERENCES
[1] I. J. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples." arXiv preprint arXiv:1412.6572, 2014.
[2] J.-H. Jacobsen, J. Behrmann, R. 7emel, and M. Bethge, "Fxcessive invariance causes adversar- ial vulnerabilityy" arXiv preprint arXiv: 181 1.00401, 2018.
[3] W. Brendel and M. Bethge, "Approximating cnns with bag-of-local-fetttee models works surprisingly well on imagenet," arXiv preprint arXiv: urXiv:1904.00766 2019.
[4] Kakarala, "A group-theoretic approach the triple correlation," " in (1993 Proceedings] IEEE Signal Processing Workshop on Higher-Order Statistics, IEEE, 1993, pp. 28-32.
[5] T. Cohen and M. Welling, "Group equivariant convolutional networks," in International conference on machine learning, PMLR, 2016, pp. 2990-2999.
[6] M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković, "Geometric deep learning: Grids, groups, graphs, geodesics, and gauges," arXiv preprint arXiv:2104. 13478, 2021.
[7] R. Kakarala, "The bispectrum as a source of phase-sensitive invariants for fourier descriptors:
A group-theorete approach," Journal of Mathematical Imaging and Vision, vol. 44, no. 3, pp. 341-353, 2012. 2012.
[8] R. Kondor, "A novel set of rotationally and translationally invariant features for images based on the commutaaivee bispectrum." arXiv preprint cs/0701 2007.
[9] R. P. Rao and D. L. Ruderman, "Learning lie groups for invariant visual perception," " Advances in neural information processing systems, pp. 810-816. 1999.
[10]
Miao and R. P. Rao, "Learning the lie groups of visual invariance." " Neural computation, vol. 19, no. 10, pp. 2665-2693, 2007.
[11] B. J. Culpepper and B. A. Olshausen, "Learning transport operators for image manifolds." in NIPS, 2009, pp. 423-431.
[12] J. Sohl-Dickstein, C. M. Wang, and B. A. Olshausen, "An unsupervised algorithm for learning lie group transformations." arXiv preprint arXiv:1001.102. 2010.
[13] T. Cohen and M. Welling, "Learning the irreducible representations of commutative lie groups,"
in International Conference on Machine Learning. PMIR, 2014, pp. 1755-1763.
[14] H. Y. Chau, F. Qiu, Y. Chen, and B. Olshausen, "Disentangling images with lie group transfor- mations and sparse coding," arXiv preprini arXiv:2012.1207. 2020.
[15] G. Benton, M. Finzi, P. Izmailov, and A. G. Wilson, "Learning invariances in neural networks from training data," Advances neural information processing systems, vol. 33, pp. 605-
17 616, 2020.
[16] B. C. Hall, "Lie groups, lie algebras, and representations," in Quanium Theory for Mahemati- cians, Springer, 2013, pp. 333-366.
[17] J. Gallier and J. Quaintance, Differential Geometry and Lie Groups: A Computational Perspec- tive. Springer Nature, 2020, vol. 12.
[18]
W. Rudin, Fourier analysis on groups. Courier Dover Publications, 1962.
[19] A. V. Oppenheim and J. S. Lim, "The importance of phase in signals," Proceedings of the IEEE, vol. 69, no. 5, pp. 529-541. 1981.
10

---

Published as a conference paper at ICLR 2023
[20] J. Yellott Jr and G. Iverson, "Uniqueness theorems for generalized autocorrelationns" Journal of the Optical Society of America, vol. 9, pp. 388 0884 1992.
[21] J. Tukey, "The spectral representation and transformation properties of the higher moments of stationary time series," Reprinted in The Collected Works of John W. Tukey, vol. pp. 165-184 1953.
[22] D. Brillinger, "Some history of higher-order statistics and spectra," Stat. Sin., vol. 1, pp. 465- 476, 1991.
[23] C. L. Nikias and J. M. Mendel, "Signal processing with higher-order spectra," IEEE Signal processing magazine, vol. 10, no. 3. pp. 10-37. 1993.
[24] R. Kakarala, "Completeness of bispectrum on compact groups," arXiv preprint
arXiv:0902.0133 vol. 1, 2009.
[25] R. Hadsell, S. Chopra, and Y. LeCun, "Dimensionality reduction by learning invariant mapping." in 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 006), IEEE, vol. 2, 2006, pp. 1735-1742.
[26] S. C. Verrall and R. Kakarala, "Disk-harmonic coefficients for invariant pattern recognition." JOSA A, vol. 15, no. 2. pp. 389-401, 1998.
[27] J. H. Van Hateren and A. van der Schaaf, "Independent component filters of natural images compared with simple cells in primary visual cortex," Proceedings of the Royal Society of London. Series B: Biological Sciences, vol. 265, no. 1394, pp. 359-366, 1998.
[28] F. Worrall, S. J. Garbin, Turmukhambetov, and G. Brostow, "Harmonic networks: Deep translation and rotation equivariance." in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5028-5037.
[29] M. Weiler and G. Cesa, "General e (2)-euuivariant steerable cnns," Advances in Neural Information Processing Systems, vol. 32, 2019.
[30] B. Sturmfels, Algorithms in invariant theory. Springer Science & Business Media, 2008
[31] S. Sanborn, A Group Theoretic Framework for Neural Computation. UC Berkeley, PhD Thesis,
2021.
[32] B. Giannakis, "Signal reconstruction from multiple correlations: Frequency-an time-
domain approaches," JOSA A, vol. 6, no. 5, pp. 682-697, 1989.
[33] R. Kondor, Group theoretical methods in machine learning. Columbia University, PhD Thesis, 2008.
[34] C. A. Haniff, "Least- squares fourier phase estimation from the modulo 2n bispectrum phase," JOSA A. vol. 8, no. 1. pp. 134-140, 1991.
[35] J. H. van Hateren and A. van der Schaaf, "Independent component filters of natural images compared with simple cells in primary visual cortex," Proceedings: Biological Sciences, vol. 265, no. 1394, pp. 359-366, Mar. 1998.
[36] Y. LeCun, C. Cortes, and C. Burges, "The mnist dataset of handwritten digits (images)," " NYU: New York, NY, USA, 1999.
[37] A. Paszke, S. Gross, F. Massa, et al., "Pytorch: An imperative style, high-performance deep learning library." " Advances in neural information processing systems, vol. 32, pp. 8026-8037, 2019.
[38] I. Nazarov and H. Schroter, Cplxmodule, https : :/ / github com / ivannz /
cplxmodule] 2021.
[39] C. Trabelsi, O. Bilaniuk, Y. Zhang, et al., "Deep complex networks," arXiv preprint 7xXiv17755007922 2017.
[40] K. Musgrave, S. Belongie, and S.-N. Lim, "Pytorch metric learning"" arXiv preprint arXiv:2008.0 2XX1:2008.09964, 2020.
[41] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization." arXiv preprint
14X:
2014.
[42] L. N. Smith, "Cyelical learning rates for training neural networks," in 20/7 IEEE winter conference applications of computer vision (WACV), IEEE, 2017, pp. 464-472.
[43] He, X. Zhang, S. Ren, and Sun, "Delving deep into rectifiers: Surpassing human-level performance imagenet classification," in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026-1034.
11

---

Published as a conference paper at ICLR 2023
APPENDIX A MATHEMATICAL BACKGROUND
Groups. A group (G,-) is a set G with a binary operation which we can generically call the product. The notation a b denotes the product of two elements in the set; however, is standard to omit the operator and write simply ab. Concretely, a group G' may define a class of transformations, such as -wo-dimensional translations or rotations in the plane. The elements of the group E G define particular transformations, such rotation by 30 rotation by 90' The binary operation provides a means for combining two particular transformations- for example, first rotating by 30 and then rotating by 90" For set of transformations G to be group under the operation ' the four following axioms must hold:
1. Closure: The product of any two elements of the group is also element of the group, i.e. for all a, E G, ab E G.
2. Associativity: The grouping of elements under the operation does not change the outcome, so long as the order of elements is preserved, i.c. (ab)c = a(bc).
3. Identity: There exists a "do-nothing" identity element e that such that the product of e with any other element g returns g. i.c. ge eg = 9 for all g G.
4. Inverse:
For every element g, there exists an inverse element g such that the product of g and g returns the identity, i.c. 99 = g =
Two groups (G, and (H, +) are homomorphic if there exists correspondence between elements of the groups that respect the group operation. Concretely, homomorphism is a map/ p + И such that p(u v) = p(u) p(v). An isomorphism is bijective homomorphism.
Homomorphisms,
Product Groups. A product group as set is the cartesian product G' x H of two given groups G and Н. The new group product is given by (g1,h)) (g2, (9192, h2
Commutativity. A group (G,+) is commutative or abelian if the order of operations does not matter, i.e. ab ba. If this does not hold for all elements of the group, then the group called non-commutative. The classification of finite commutative groups says that each such group is a product of cyclic groups.
Group Actions. A group action is map T G x X + X that maps (g. x) pairs to elements of X. We say a group G acts on a space X if the following properties of the action T hold:
1. The identity e E G maps an element of x E X to itself, i.e. T(e, 2) = x
2. Two elements 91, 92 E G can be combined before or after the map to yield the same result, i.e. T(g,, T(g2, .r) (g)122,
For simplicity, we will use the shortened notation Ty to denote T(g, x), often expressed by saying that a point maps to gr (= (x)).
Invariance. A function o X ++ Y is G-invariant if d(x) = o(gz) for all E G and æ e X. This means that group actions on the input space have no effect on the output.
Equivariance. A function é X + Y is G-equivariant if ((gr)= = g (zx for all g E G and x e X, with g' E G', a group homomorphic to that acts on the output space. This means that a group action on the input space results in corresponding group action on the output space.
Orbits. Given a point I E X, the orbit Gz of x is the set {gr 9 E G}. In the context of image transformations, the orbit defines the set of all transformed versions of a canonical image- for example, if G is the group of translations, then the orbit contains all translated versions of that image.
Homogeneous Spaces. We say that X is a homogeneous space for group G if G acts transitively on X-that is, if for every pair 11, x2 E X there exists an element of g E G such that gz, = x2. A homogeneous space X equipped with the action of G is called G-space. The concept can be clearly illustrated by considering the surface of a sphere, the space 52. The group SO(3) of orthogonal 3 x 3 matrices with determinant one defines the group of d-dimensional rotations. The sphere S² is homogeneous space for SO(3). For every pair of points on the sphere, one can define a 3D rotation matrix that takes one the other.
12

---

Published as a conference paper at ICLR 2023
Representations. A representation of group G is map G GL(V) that assigns elements of G to elements of the group of linear transformations over a vector space V such that p(9192) = p(9)) That is, p is group homomorphism. In many cases, V is R" C A representation reducible if there exists a change of basis that decomposes the representation into a direct sum of other representations. An irreducible representation is that cannot be further reduced in this manner, and the set of them [rr(G) are often simply called the irreps of G. The irreducible
representations of finite commutative group are all one-dimensional and in bijection with the group itself. In particular, is straightforward to compute [r)(G) for such groups given their classification as products of cyclic groups Z/ nZ.
APPENDIX B THE BISPECTRUM ON NON-COMMUTATIVE GROUPS
The theory of the bispectrum applies also the setting of non-commutative groups and can be expressed in terms of the generalized Fourier transform and the (not necessarily one-dimensional) representations of the group. The more general, non-commutative form the bispectrum is:
3p
pf @ fp,]Cppus prips
+
f3]
Ct
(13)
pEp:00ps
where @ is the tensor product, @ is direct sum over irreps, and the unitary matrix Cpopp defines Clebsch-Gordan decomposition on the tensor product of pair of irreps Pt.
APPENDIX C INCOMPLETENESS OF THE POWER SPECTRUM
Image
Power Spectrum
Phase Spectrum
Pizse
Target
Log Powes
Random Phase
Pizes
Loe Power
Raudom Power
Piasn
Figure 6: The Power Spectrum a Non-Selective Invariant. images with identical power spectra (top and middle) can have very different image structure. However, images with identical phase spectra and non-identical power spectra (top and bottom) share much of the mcaningful image structure- that which defines edges and contours. May require enlargement to see. Note that the sharp horizontal and vertical lines in the power and phase spectra are artifacts due to wrap-around of the image boundaries.
13

---

Published as a conference paper at ICLR 2023
APPENDIX D COMPLETENESS OF THE BISPECTRUM
The bispectrum is a (generically) complete invariant map with respect to given compact group G 24). Here, generic a technical term that means "outside of a of measure zero." " For instance, in the finite commutative group case, this amounts to uniqueness for all elements of the space that have all-nonzerr Fourier coefficients. In practice, this property easily derived from the exactly computable inverse of the bispectrum; see, for instance, algorithms spelled out in [3]] 33 In these
computations, an inverse defined whenever denominators which end up being Fourier coefficients recursively computed appearing in the calculations are nonzero.
APPENDIX E COMPLETENESS OF THE NORMALIZED BISPECTRUM
useful observation is that this generalizes somewhat normalized form of the bispectrum, where we project features to the unit sphere in C (normalization).
Definition E.1. The normalized bispectrum function f is defined
B(f) = B(S)
First, we show that any scalar multiple of the bispectrum preserves completeness up positive scalar multiplication of the input signal f. Using this result, we show that the normalized bispectrum is generically complete up the action of G and scalar multiplication.
Lemma E.1. The bispectrum of scalar 0 times function is:
B(ef) B()).
Proof. By linearity of the Fourier transform, we have: ef of. From the definition, the commutative bispectrum of of therefore satisfies:
B(cf) = =ffeffeftts
(14)
- =
This is easily seen to also work for the non-commutative formulation.
Lemma E.2. If the bispectra of two (generic) functions f and g differ by a constant factor >> 0, then the orbits of the two functions differ by constant factor. i.e.
B() cB(g) = GJ = Ggg.
Proof. Assume B(f) = cg(g) and let h = c3g. By Lemma E. we have: 3(h) = cB(g). and by assumption, ((f) = cg(g) that (() = B(h). By (generic) completeness of the bispectrum, the orbits of f and l are equivalent; that is, Gf Gh.. Because h is scalar multiple of g, the orbits of h and g related by constant factor: Gh = cà Gg. Finally, because the orbit of equals that of h, that the orbit of is related to that of g by the same factor: Gf = ct Gg-
Li
Theorem E.3. Two functionss f and g have equivalent normalized bispectra if and only if their orbits under G are related by scalar factor E Tk.
functions
B( = B(g) + Gf = CcG.
Proof. (=)) By assumption, Gf cGg, which, together with Lemma E.1 implies 3(J) (3(())
14

---

Published as a conference paper at ICLR 2023
Writing out the definition of the normalized bispectrum,
B(S)
B(f)
1130111
c)((g)
(15)
e)((g)
B(g)
3(g)-
||3(g)]1
(+) Given B( B()), expand and see the following relation:
B(f) B(g)
8(F)
3(g)
||(0
113(0)11
(16)
113(11
B()
[B(()]| (())
Letting
|(/II [F
have:
B(f) c3(g),
which by Lemma
implies that Gf cGg.
APPENDIX F POLYSPECTRA AND AUTOCORRELATIONS
Both the power spectrum and bispectrum intimately related to the second- and third-order statisties of signal, as the Fourier transforms of the two- and threc-point autocorrelation functions, respectively. The two-point autocorrelation A₂ of complex-valuee function f on the real line is the integral of that function multiplied by shifted copy of it:
A2,5(5) = f (x)s(x + s)dr,
(17)
where denotes the increment of the shift. Taking its Fourier transform, it becomes:
(s) - faftt.
(18)
which is the power spectrum. The three-poine autocorrelation, also called the triple correlation, is the integral a function multiplied by two independently shifted copies of it:
A3.5(81.82) 1 (x))S(( + + 82)dr.
(19)
Taking its Fourier transform it becomes:
(2, $2) = fra
(20)
which is the bispectrum.
15

---

Published as a conference paper at ICLR 2023
APPENDIX G SYMMETRIES IN THE BISPECTRUM
The bispectrum has n² terms (with n = number of frequencies). However, the object is highly =" terms without loss of information for real-valued signal [23] This redundant and is reducible to 12 due to the following symmetries, depicted Figure For w1
2
Bui w
Bwz ww1
B way m11 = 99-wi -12 -22442
Bur, -ws -w2 B- w2
Bwai-w1- -w22
Thus, the subset of bispectral coefficients for which w2 >= 0, == W2, w1 tw2 <= v is sufficient for complete description of the bispectrum.
Symmetrics of the Triple Correlation and Bispectrum
Triple Conrelation
Bispectrum
R
$1
olly
2 A
R
Figure 7: Symmetries of the Triple Correlation and Bispectrum. For real-valued signal, the six-fold symmetry of the triple product becomes twelve-fold symmetry in which six paris of regions conjugate symmetric (with the complex conjugate indicated grey). Adapted from 23
It is also known that, in fact, the bispectrum can be further reduced. For example, for the bispectrum on ID translation, only n + coefficients are needed to reconstruct the input up translation, and thus define a complete description of the signal structure in the quotient space [33]. The reconstruction algorithm proposed in (33) requires that the signal has Fourier coefficients equal to zero. More robust reconstruction algorithms, such as least squares approaches, use more elements of the bispectrum matrix (34] for stability purposes.
APPENDIX H DATASETS
H.1 VAN HATEREN NATURAL IMAGES
We use natural image patches from the Van Hateren database 27 as the data on which the groups act, and generate two datasets, each using different group action: Cyclic 2D Translation (S' x S²) and 2D Rotation S0(2). For each group, we randomly select 100 image patches as the "exemplars" on which the group acts. Each image patch is augmented by each element of the group to generate the orbit of the image under the transformation. The group S x S' is discretized into integer translations, and all 256 possible discrete translations are used to generate the data. The group SO(2) is discretized into 360 integer degree rotations and 30% of these rotations are randomly drawn for each image to generate its orbit. Following orbit generation, image patches are normalized to have zero mean and unit standard deviation. For the 2D rotation dataset, patterns with diameter equal to the side length of the image, with values outside of the circle set to zero, to maintain perfect rotational symmetry. Appendix H shows examples from the dataset. A random 20% of each dataset is set aside for model validation and used to tune hyperparameters. The remaining 80% is used for training.
cropped to circle
16

---

Published as a conference paper at ICLR 2023
2D Cyclic Transatiion x 8
Van Hateren Image Patches
2D Rotation SOG)
Figure Examples from the Cyelic Translation Rotation Datsets. Image patches are randomly sampled from the Van Hateren natural image database (35 The orbit of each patch under 2D translation (Left) and rotation (Right) are generated.
H.2 ROTATED MNIST
The Rotated MNIST dataset is generated by rotating each digit in the MNIST (36 training and test datasets by random angle. This results in training and test sets with the standard sizes of 60, 000 and 10, 000. random 10% of the training dataset is set aside for model validation and is used to tune hyperparameters. The remaining 90% is used for training. Images additionally downsized with interpolation to 16 x 16 pixels.
S 6
N 3
0
S E
O
6
4
3
3
5
0
8
S
J
S
U
h
7
2
9
5
9
9 2
5
5
4
I
C
3
6
2
8
0
Figure 9: Examples from the Rotated MNIST Dataset.
APPENDIX TRAINING PROCEDURES
All networks were implemented and trained in PyTorc (37] We additionally made use of the open- source CplxModule library (38), which provides implementations of the various complex-valuee operations and initializations used in this work.
I.I LEARNING FOURIER TRANSFORMS ON GROUPS
The weight matrix W square complex-valuse matrix of dimension 256 256, matching the dimension of the (raveled) (16, 16) image patches in the dataset. Weights were initialized using the complex orthogonal weight initialization method proposed in (39). Each parameter vector in W was normalized to unit length after every gradient step. We used batch sampler from the Pytorch Metric Learning library (40] to load batches with M random examples per class, with M 10 and a batch size of 100. Networks were trained with the Adam optimizer [4] until convergence, using initial learning rate of 0.002 and cyclic learning rate scheduler with 0.0001 and 0.005 as the lower and upper bounds, respectively, of the cycle.
17

---

Published as a conference paper at ICLR 2023
1.2 GROUP-INVARIANT CLASSIFICATION
The images in the Rotated MNIST dataset were down-scaled with interpolation to 16 x 16 pixels match the trained model size. The dataset passed through the Bispectral Network yield rotation-invariant representation of the images. The complex-valued output vector reduced to 500 dimensions with PCA. The real and imaginary components of the output vectors are then horizontally concatenated before being passed to the classifier network.
The classifier model consists of a four-layer feedforward network with hidden layer dimension [128, 128, 64, 32], ReLU activation functions between each hidden layer and softmax activation the output that generates distribution over the digit classes. The MLP is trained with the cross-entropy loss on the labeled digits using the Adam optimizer with an initial learning rate of 0.002, which reduced by a factor of 0.95 when the validation loss plateaus for 30 epochs, until minimum learning rate of 0.00005 is reached. Weights in the network are initialized using the Kaiming initialization [43] and biases are initialized to zero. The network is trained until convergence at 1500 epochs. The baseline Linear SVM model is trained with the L2- 12-penalty, squared-hing loss, and regularization parameter of 1.0.
1.3 COMPLETENESS
Inputs {#1...., nN} are optimized convergence using the Adam optimizer (41). An initial learning rate of 0.1 is used for all models, which is reduced by 0.1 when the loss plateaus for 10 epochs.
1.4 RECOVERING CAYLEY TABLES
Models were trained to convergence with the Adam optimizer 41 using cyclic triangular learning rate scheduler [42 base learning rate og 10-5, mximum learning rate of 0.001, and step size up of 5 epochs.
APPENDIX LEARNING FOURIER TRANSFORMS ON GROUPS
We show the model's learned weights W for each experiment: 2D Cyclic Translation and 2D Rotation. Each tile is the real of row of W reshaped into 16 16 image.
Cyclic Translation
x
2D Rotation SO(2)
Figure Learned Filters. Real components of weights learned Bispectral Networks trained on van
Hateren Natural Images dataset under (left) Cyclic 2D Translation and (right) 2D Rotation.
18

---

Published as a conference paper at ICLR 2023
APPENDIX K RECOVERING CAYLEY TABLES FROM IRREPS
Below we list the algorithms used to recover Cayley tables from a group's irreducible representations.
Algorithm TETCAYLEYTROMMRRRSS
Algorithm GETCAYLEYYROMGRRU
Compute the Cayley table from the (unordered)
Compute the Cayley table of group G.
irreducible representations of a commutative
procedure GETCAYLLYYROMGRUUPC)) for 91 E G do
group.
procedure GETCAYIEYYFROMIRRESSII for i 1, ,,.,, n do
for 92 G do
C( 92) + 91
for j 1,... n do
end for end for return C' end procedure
P - W; + W
k* argmax W
((i,j) + k*
end for end for return Ö end procedure
Algorithm 3 IsISOMORPHIC
Check whether the learned group and the provided group are isomorphic by checking equality of their Cayley tables under permutation of group elements.
Let Sn the group of all permutations elements.
procedure ISISOMORPHIC{() W)
for E S, do
G + π o G C f GETCANIEYYROMGGOUUGGG
Permute the elements of according to the permutation element π
C' (ETCANLEYFROMMRREES(()))
C' = Ć' then
return True
end if end for return False end procedure
19


Reviews:
[{"review_id": "yO_v4zyz-E", "reviewer": "Reviewer_pa8G", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a neural network architecture, Bispectral Neural Networks(BNNs), which aims to learn the group-invariant representations of the actions of compact commutative groups. This simple architecture is composed of two layers: a single learnable linear layer, followed by a fixed collection of triple products computed from the output of the previous layer. Then, some comparison experiments are conducted to show the effectiveness of the proposed method. ", "summary_of_the_review": "From the manuscript, the authors propose BNNs, which could simultaneously learn groups, their irreducible representations, and corresponding complete invariant maps purely from symmetries implicit in data. I think it is good. ", "strength_and_weaknesses": "strengths:\n1. This paper has comprehensive and rigorous mathematics proof. \n2. The network is simple yet effective, and the competitive results show the effectiveness. \n3. This paper has demonstrated that the completeness property endows these networks with strong adversarial robustness. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O8JerSSGaE8", "reviewer": "Reviewer_1MdX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents the Bispectral Neural Network (BNN), a network primitive which learns to be invariant to transformations present in the data. The model is defined in terms of two main components. Firstly, the model treats the irreducible representations of a group of transformations as weights in the network, thus essentially learning a Fourier transform on the group, which is equivariant to the group action. The Fourier representation of the input is then mapped to the bispectrum, which is invariant to the transformation but retains the information needed to uniquely restore the image (up to the transformation), by the network. Secondly, a novel loss function encourages the bispectrum for inputs of the same class to be identical, effectively making the trained network invariant to the transformation. A crucial point here is that the group of transformations need not be defined a priori; the network discovers the transformations automatically.\n\nThe authors test the model in four different experiments, demonstrating that it can construct the transformation group from data alone, that the learnt irreducible representations are transferable to other datasets, and that the model is robust to adversarial attacks.", "summary_of_the_review": "The paper presents a novel method for learning representations of data that are invariant to transformations of the data. Not only is the method novel, the ideas and concepts presented in the paper are intriguing and should be of significant interest to the ICLR community. The experiments are somewhat weak, though, and the paper seems to be mostly a proof-of-concept, yet the strengths far outweigh these weaknesses. I view the paper as a very significant and impactful contribution to the fields of invariance and equivariance in machine learning.", "strength_and_weaknesses": "### Strengths\nOverall, I find this paper very strong. While the bispectrum is extensively used in other fields, it does not seem to be the case in machine learning (as far as my searching abilities go). Simply introducing this incredibly useful concept to the machine learning community has value in itself, and then further parametrising it in terms of a neural network, which, given a novel loss function, learns the group of transformations directly from data, makes this paper very strong. The simplicity and elegance of the proposed method are intriguing, and I could see the ideas introduced here forming the basis of many future papers on invariant and equivariant models.\n\nIn summary, the paper\n- presents an incredibly useful concept (the bispectrum),\n- introduces a novel and interesting loss function, and\n- demonstrates how to learn irreducible representations of the transformation group using a neural network.\n\n\n### Weaknesses\nThe paper appears to be mostly a proof-of-concept. The datasets used in the experiments are quite simple, which the authors acknowledge themselves. It is also unclear to me if the method is usable in practice in terms of the computations that are required to compute the bispectrum. Since the complex weight matrix needs to be of size $n \\times n$, $n$ being the number of input dimensions, does the method scale to inputs of even moderately large dimensionality? A study of this, e.g., in terms of the training time and memory cost for increasing image sizes, would have been interesting. The authors say that they are currently working on a localised version of the model, such that it can be used in a convolutional neural network, which would address this issue. Still (and without knowing exactly how much work this entails), one might have expected such an extension to be included in the current paper.\n\nI see these weaknesses as minor, though, given the significance of the model and the ideas and concepts the paper introduces.\n\nIn summary, I see the main weaknesses as being that\n- the model is only tested on simple datasets, and that\n- it is unclear if the model scale to inputs of moderately large dimensionality.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JQ6I1HOCsRQ", "reviewer": "Reviewer_QqWn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The idea is to learn inverariance to commutative groups. The groups are learned from data through the proposed block, which relies on the bispectrum idea. The proposed block (depicted in Fig. 2) resorts to a third-order polynomial that can be implemented on standard frameworks, as it requires an affine transformation of the input and a Hadamard product. In addition to this, an orbit separation loss is proposed. ", "summary_of_the_review": "The proposed bispectral block is new to me, and the idea of including invariances to groups in the network is also an interesting research direction. I would say one of the weaknesses in my mind is the experimental validation that is currently weak, given the empirical nature of this work. \n\n_____________\nAfter the rebuttal: See the discussion below, but the core idea is that some of the concerns have been addressed below. ", "strength_and_weaknesses": " * [+] Including symmetries in neural networks is an interesting research direction. \n\n * [+] The writing is mostly clear, while I like the summary of the group notation in the appendix. \n\n * [+] The proposed block for the bispectrum is simple. \n\n * [-] The experiments are weak at the moment. To be specific, I do not find much the application of this in real-world data in the experimental section. \n\n * [-] The claim on the adversarial robustness seems exaggerated (see below). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster