Learning symmetries via weight-sharing with doubly stochastic tensors
Authors: Putri A Van der Linden
Alejandro García Castellanos
Sharvaree Vadgama
Thijs P. Kuipers
Erik J Bekkers
Venue: NeurIPS 2024 poster (2024)
Keywords: symmetry discovery
weight-sharing
Primary area: deep_learning_architectures
TL;DR: We learn a weight-sharing scheme by defining a collection of learnable doubly stochastic matrices that act as permutation matrices on canonical weight tensors.

Abstract: Group equivariance has emerged as a valuable inductive bias in deep learning, enhancing generalization, data efficiency, and robustness. Classically, group equivariant methods require the groups of interest to be known beforehand, which may not be realistic for real-world data. Additionally, baking in fixed group equivariance may impose overly restrictive constraints on model architecture. This highlights the need for methods that can dynamically discover and apply symmetries as soft constraints. For neural network architectures, equivariance is commonly achieved through group transformations of a canonical weight tensor, resulting in weight sharing over a given group $G$. In this work, we propose to *learn* such a weight-sharing scheme by defining a collection of learnable doubly stochastic matrices that act as soft permutation matrices on canonical weight tensors, which can take regular group representations as a special case. This yields learnable kernel transformations that are jointly optimized with downstream tasks. We show that when the dataset exhibits strong symmetries, the permutation matrices will converge to regular group representations and our weight-sharing networks effectively become regular group convolutions. Additionally, the flexibility of the method enables it to effectively pick up on partial symmetries.

Learning symmetries via weight-sharing with doubly stochastic tensors
Putri A. van der Linden
Alejandro Carcía-Castellan Thijs P. Kuipers2.3 Erik J. Bekkers!
Sharvaree Vadgama
Amsterdam Machine Learning Lab, University of Amsterdam Department of Biomedical Engineering and Physics, Amsterdam UMC, the Netherlands Department of Radiology and Nuclear Medicine, Amsterdam UMC, the Netherlands
Abstract
Group equivariance has emerged as valuable inductive bias in deep learning, enhancing generalization, data efficiency, and robustness. Classically, group equiv- ariant methods require the groups of interest to be known beforehand, which may not be realistic for real-world data. Additionally, baking in fixed group equivariance may impose overly restrictive constraints on model architecture. This highlights the need for methods that can dynamically discover and apply symmetries as soft constraints. For neural network architectures, equivariance is commonly achieved through group transformations of canonical weight tensor, resulting in weight sharing over a given group G. In this work, we propose to learn such a weight- sharing scheme by defining a collection of learnable doubly stochastic matrices that act as soft permutation matrices on canonical weight tensors, which can take regular group representations as special case. This yields learnable kernel trans- formations that are jointly optimized with downstream tasks. We show that when the dataset exhibits strong symmetries, the permutation matrices will converge regular group representations and our weight-sharing networks effectively become regular group convolutions. Additionally, the flexibility of the method enables it to effectively pick up on partial symmetries.
Introduction
Equivariance has emerged as a beneficial inductive bias in deep learning, enhancing performance across a variety of tasks. By constraining the function space to adhere to specific symmetries, models not only generalize better but also achieve greater parameter efficiency [ [.. For instance, integrating group symmetry principles into generative models has enhanced sample generation and efficient learning of data distributions, particularly benefiting areas such vision and molecular generation
12
The most well-known and transformative models in equivariant deep learning are convolutional neural networks (CNNs) [8] which achieve translation equivariance by translating learnable kernels to every position in the input. This design ensures that the weights defining the kernels are shared across all translations, so that if the input is translated, the output features are correspondingly translated: in other words, equivariance is achieved through weight sharing. In the seminal work by Cohen and Welling [9) this concept was extended to generalize weight-sharing under any discrete group of CNN (G-CNN). G-CNNs enable G-cquivariance to thereby expanding
symmetries, resulting in the group-cquivartn broader range of symmetries, such as rotation, reflection, and scale 99 35 the applicability of CNNs to more complex data transformations.
"Corresponding author: <p.a. a. wwnderl indennuva.nn. ml>
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, the impact of G-CNNs is closely tied to the presence of specific inductive biases in the data. When exact symmetries, such as E(3) group symmetries in molecular point cloud data, are known to exist, G-CNNs excel. Yet, for many types of data, including natural images and sequence data, these exact symmetries are not present, leading to overly constrained models that can suffer in performance [3 24 31. In scenarios with limited data and for certain critical downstream tasks, having appropriate inductive biases becomes even more crucial.
To avoid overly constraining models, symmetries must be chosen carefully to match those in the input data. This requires prior knowledge of these symmetries, which may not always be available. Furthermore, different symmetries at different scales can coexist, making manual determination of these symmetries highly impractical. To address this, multiple works have proposed partial or relaxed G-CNNs [24 6 2 32]. Such models are initialized to be fully equivariant to some groups and learn from the data to partially break equivariance on a per-layer basis where necessary. However, these methods still require specifying which symmetries to include and can only achieve equivariance to subsets of these symmetries.
In this work, we tackle the challenge of specifying group symmetries upfront by introducing a general weight-sharing scheme. Our method can represent G-CNNs special case but not limited to exact equivariance constraints, offering greater flexibility in handling various symmetries in the data, Inspired by the idea that group equivariance for finite groups can be achieved through weight-sharing patterns on a set of base weights [23), we propose learning the symmetries directly from the data on per-layer basis, requiring no prior knowledge of the possible symmetries.
We leverage the fact that regular group representations act as permutations and that the expectation doubly stochastic matrix 77]. This
of random variables defined over this set of permutations implies that regular partial group transformation can be approximated by a stack of doubly stochastic matrices which essentially act (soft) permutation matrices. Consequently, we learn set of doubly stochastic matrices through the Sinkhorn operator (28). resulting in weight-sharing under learnable group transformations.
We summarize our contributions as follows:
We propose novel weight-sharing scheme that can adapt to group actions when certain symmetry transformations are present in the data, enhancing model flexibility and performance.
We present empirical results image benchmarks, demonstrating the effectiveness of our approach in learning relevant
schemes when there are clear symmetries.
The proposed method outpaces models configured with known symmetries in environments where they are only partially present. Moreover, in the absence of predetined symmetries, adeptly identifies effective weight-sharing patterns, matching the performance of fully flexible, models.
We provide analyses of the learned symmetries on some controlled toy settings.
2
Related work
Partial or relaxed equivariance Methods such as [24] 6 2 learn partial equivariance by learning distributions over transformations, and thereby aim to learn partial or relaxed equivariances from data by sampling some group elements more often than others.
relax equivariance by 32
33
introducing learnable qquivariance-brelle by parameterizing layers as (linear) combinations of fully flexible, non-constrained components and constrained equivariant components. Finally, several works model soft invariances through learning the amount of data augmentation in the data or model relevant for a given task, either through learned distributions on the group or (automatic) hyperparameter selection 6 22 However, these methods require pre-specified sets of symmetry transformations and/or group structure to be known beforehand. In contrast, we aim to pick up the relevant symmetry transformations during training.
components. [T 31 Relax equivariance constraints
Symmetry discovery methods Several methods have been developed for symmetry discovery by leveraging the infinitesimal generators of symmetry groups, as articulated through the Lie algebra 10 36 37 21. Works such as [10 36 21 focus discovering symmetries associated with

---

general linear actions. In contrast, [37 extends these concepts to encompass non-linear symmetry transformations, offering capabilities for discovering partial symmetries within the data.
Learn the group structure via (irreducible) group representations. [26] Proposed to learn the Fourier transform of finite compact commutative groups and their corresponding bispectrum by learning to separate orbits on our dataset. This approach can be extended to non-commutative finite groups leveraging advanced unitary representation theory [20]. However, these methods are constrained to finite-dimensional groups and require specific orbit-predicting datasets. In contrast, our approach learns relaxation of regular group representations as opposed to irreducible repre- sentations. Moreover, our approach is not merely capable of learning symmetries, it subsequently utilizes them in a regular soouu--onvolution-yee architecture.
126 20
Weight-sharing methods Previous studies have demonstrated that equivariance to finite groups can be achieved through weight-sharing schemes applied to model parameters. Notably, the works in [23] [39] and [38] provide foundational insights into this approach. In [9], weight-sharing patterns are learned by using matrix that operates on flattened canonical weight tensors, effectively inducing weight sharing. They additionally prove that for finite groups, there are
matrices capable of implementing the corresponding group convolution. However, their approach requires learning these patterns through meta-learning and modeling the weight-sharing matrix as an unconstrained tensor. In contrast, our method learns weight sharing directly in conjunction with the downstream task and enforces the matrix to be doubly stochastic, thereby representing soft permutations by design.
38 Presents an approach closely aligned with ours, where a weight-sharing scheme is learned that is characterized by row-stochastic entries. Their method involves both inner- and outer-loop opti- mization and demonstrates the ability uncover relevant weight-sharing patterns straightforward scenarios. However, their approach does not support joint optimization of the canonical weights and weight-sharing pattern, and they acknowledge difficulties in extending their method to higher input dimensionalities. Unlike (38), we enforce both row and column stochasticity. Additionally, we can optimize for the sharing pattern and weight tensors jointly, and successfully apply our approach to more interesting data domains such as image processing.
In 33), group actions are integrated directly into the learning process of the downstream task. This method involves learning set of generator matrices that operate via matrix multiplication on flattened input vectors. However, this approach constrains the operators members of finite cyclic groups, which inherently limits their ability to represent more complex group structures. Furthermore, this restriction precludes the possibility of modeling partial equivariances, reducing the flexibility and applicability of the model to more diverse or complex scenarios.
Background
We begin by revisiting group convolutional methods in the context of image processing, followed by their relation weight-sharing schemes. We then proceed to briefly cover the Sinkhorn operator, which is the main mechanism through which we acquire weight-sharing schemes. Some familiarity with group theory is assumed, and essential concepts will be outlined in the following.
Groups We are interested (symmetry) groups, which are algebraic constructs that consist of set G and a group product-which we denote as a juxtaposition that satisfies certain axioms, such as the existence of an identity element E G such that for all E G we have eg = ge = g. closure such that for all g, h E G we have gh E G. the existence of an inverse g-1 for each such that g and associativity such that for all g, h, E we have (gh)i = g(hi).
Representations In the context of geometric deep learning g is most useful to think of groups as transformation groups. and the group structure describes how transformations relate to each other. Specifically,, group representations : G > GL(V) are concrete operators that transform elements in vector space V in way that adheres to the group structure (they are group homomorphisms). That is, to each group element g. we can associate a linear transformation p(g) E GL((), with GV) the set of linear invertible transformations vector space V.
3

---

Group convolution Concretely, such representations can be used to define group convolutions. R over some domain on which group action defined, i.e., over G-space. E.g., for images (signals over X = R we could consider the group G = (IR2, + of translations, which acts on X via gz + y, with G merely defines how two transformations g, h G applied one after the other correspond to a net translation gh E G. a representation concretely describes how data is transformed. E.g., signals f: X R can be transformed via the left-regular representation (x) == f(g which in the case of images and the translation group is given by (((g)
Consider feature maps f
(y) a translation by y E R While the group
y).
In general, group convolution is defined as transforming a base kernel under every possible group action, and for every transformation taking the inner product with the underlying data via
G-convolution, inner product form:
(k G f) fp
(1)
with ( denoting the inner product. For images, which are essentially functions over the group G = (R+,)), +), and taking (k: f) == fx fx)dr the standard inner product, Eq. (1) boils down to the standard cross-correlation operator
()()) J( (h))
G-convolution, integral form:
(2)
*k* [)() = [ k(dl - I
Standard (IR?, +) Convolution:
Semi-direct product groups When equivariance to larger symmetry groups is desired, e.g. in the case of G (2) roto-translation equivariance for images with domain X = R2, lifting convolution can be used to generate signals over the group G. In essence is still of the form of 2 however integration is over X instead of over G:
f()) = J kig x(g)))
G-lifting Convolution:
(4)
(k (x. = / k(R- =z))
-E(2)-littnng Convolution:
(5)
R22
with g (x, R) (R2,+) SO(2). The roto-translation group an instance of a semi-direct product group (denoted with x) between the translation and rotation group, which has the practical benefit that stack of rotated kernels can be precomputed [9], and the translation part efficiently be taken care of via optimized Conv2D operators. Namely via (k fx, R.) Conv2D (k;, f), with k; = k(R,122 2). This trick can also be applied for full group convolutions
Method
Our objective is to uncover the underlying symmetries of datasets whose exact symmetries may not be known, in a manner that is both parameter-efficient and free from rigid group constraints. To achieve this, we re-examine regular representations and their critical role in generating various instantiations of the fundamental group convolution equation (T. Moving from the continuous setting conerete instantiations, we derive weight-sharing from anite-dimensional vector of base weights by interpreting regular representations as permutations. We further analyze the characteristics of this beight-sharing approach, proposing the use of doubly stochastic matrices. This analysis forms the foundation for developing weight-sharing layers that adaptively learn dataset symmetries.
4.1 Weight-sharing through permutations
In the current section, we first establish the connection between representations, weight-sharing and permutations. We then proceed to provide practical instantiations as ingredients for the proposed weight-sharing convolutional layers.
*Due to the equivalence between convolution and correlation via kernel reflection, we henceforth simply refer operators of the type of (2) convolution though technically they cross-correlations.

---

Weight-sharing through learnable representations To achieve weight-sharing over finite set of symmetries, we define learnable representations p G GV(V). Specifically, we assign learnable transformation (permutation matrix) to each element in G. It is important to note that we refer to G and p as a "group" and "representation" in a loose sense, as we relax the homomorphism property and do not initially endow G with a group product. Consequently, the collection of linear transformations does not form a group representation priori. However, our proposed method is capable of modeling this structure in principle.
Regular representations as permutations Consider the case of continuous linear Lie group G, e.g. of rotations S0(2), and a real signal f: G R. over it. This signal is to be considered an infinite dimensional vector with continuous "indices" g E G that index vector elements f [g). To emphasize the resemblance of the regular representation to permutation matrices we write it as
( -sa Poli, ()()))) (6)
Regular representation integral form:
with Po h) 6, (1))) kernel that for each g maps h new "index" E G, where the Dirac delta essentially codes for the group product (2) is non-zero only g h + g h
The discrete counterpart of such an integral transform is matrix -ector multiplication with matrix p(g) P. with entries Pahi 1 if gh = and zero otherwise. As concrete example, consider signal/ f G - over the discrete group G = C4 of cyclic permutations size 4, i.e., a periodic signal of length four, then we could vectorize it as - v with entries Ug f(g) and the regular representation becomes a simple cyclic permutation matrix with
1
Po
P
P2
P3
0
We refer the collection of permutation matrices P [0, || as the permutation tensor.
Recall that the regular convolution operator (2) not only defined for signals over groups G but for G-spaces X, in general, as in Eq. 4 It requires representation that acts signals (convolution kernels) over X. However, since the group acts by automorphisms (bijections) on the space X, we can define the regular representation as before using permutation integral with (z z') = 65-12' that effectively sends old "indices" " x' to their new location æ, or concretely via permutation matrices P of shape |G| x |X| [X). where X is the domain of the signal that is transformed-whee can also be G.
In practice, it is common that we perform a discretization of continuous signals f. e.g. when we work with images, we discretize the continuous signal f R + RC to : Z² + R.C Therefore, the group action over the discretized signal should approximate the group action over the continuous signal. We can see that some cases, the group representation of the action on the discrete signal can still be implemented using permutation matrices. E.g.. 902 rotations (in C4) applied to images merely permute the pixels. However, for finer discretizations, e.g. using 45 rotations, interpolation can be used form of approximate permutations [17 Fig. 2].
Weight sharing In discrete group setting, we then see that convolution |I is obtained by multiplications with matrices obtained by stacking permuted base kernel weights E RIX|
(Pow) (Piw)
Discrete G-convolution:
pout
Wpin
with W = Pw =
E ER
GIX|X]
Mio
10
M2
E.g., a group convolution over C4 is implemented with matrix of the form W
103
10g
W1
10g
10g
16g
101
W11
102
124
Regular representations allow for element-wise activations An important practical element of using regular representations to define group convolutions is that permutations commute with element-wise activations, namely, (P(g)a()))( (1)) = -1)) = a (()) In contrast, stecrable methods- baseddaaaddd on irreducible representations (cf. App. A.I require specialized
5

---

R1 vec(k)
R2 vec(k)
reshape
R3 vec(k)
R4 vec(k)
k: H
Figure 1: Kernel stacks acquired through learned weight-sharing scheme applied set of flattened base kernels.
activation functions so as not to break group equivariance. Such activations in practice are not as effective the classic clement-wise activations such ReLU [B4] they may introduce discretization artifacts that disrupt exact equivariance, ultimately constraining the method's 's expressivity (5). Hence, when learning weight-sharing schemes- is our objective-it it is preferred to achieve weight-sharing using regular representations without the risk of breaking equivariance by using standard activation functions.
4.2 Learnable doubly stochastic tensors
Having established the link between regular representations and soft permutations, we now motivate the use of doubly stochastic matrices as natural candidates for their implementation. Specifically, we utilize the fact that the expected value of random variables over this set of permutations yields doubly stochastic matrix 77.
Doubly stochastic matrices Let Soc (S,,) denote respectively the system of infinite (n x n) doubly stochastic matrices, i.e. matrices S = {sij E [0, 1] i,j = 1.,2,...(,1) such that >, Sij = 1, and
Σ, 8ú 1. Let Poo (Pn) denote respectively the system of infinite (n x n) permutation matrices, Le. matrices p {Py E {0, i, 1,2, (2....).)) n)) such that E, Ptj and Σ, Ply Note that for any permutation tensor P E IR [[||||X|I|X| where X is the domain of the signal that is transformed by the group G, then P, Pix for every G.
Then, by Birkhoff's Theorem 7 and its extension to infinite dimensional matrices, commonly called Birkhoff's Problem 111 [14 25 15 3), have that any convex combination of permutation matrices will be equal to a doubly stochastic matrix, i.e,
Σ
APP) S Sn, with PEP
Σ xPP)= =1
(Vn E NU
PEP
where X(P) gives probability measure supported on a finite subset of the set of permutation matrices P. Therefore we may state that
Using doubly stochastic matrices, we can model approximate equivariance as defined in 24
Le., let S be random variable over (P, Pix g e G} with a finitely supported probability measure P[S p.l x(P,) for every g G, then S E[S] = = P[S P.IP, is doubly stochastic matrix. We want to note that S can be seen as generalization of the convolution matrix presented in [9]
Sinkhorn operator The Sinkhorn operator 28 transforms an arbitrary matrix doubly stochastic one through iterative row and column normalization, provided that the number of iterations

---

R1
R₂
R2
R4
C1
Cs
Figure 2: Learned kernels from the lifting layer of WSCNN, applied to rotated MNIST and re- shaped to [No.clem., Cout]. Since R2 is set as the identity operator, the first column displays the raw kernels.
Figure 3: Comparison of C₁ representations and the representation stack learned by the lift- ing layer on the rotated MNIST dataset. Top: learned representations. Bottom: permutations for C4 on 25.
is large enough. That is, initialize tensor X E NNN, then it will converge to doubly stochastic tensor via the following algorithm:
5 x)) exp(X),
('(X) (TTTT
SN S lim S'
(8)
with T. and T, the normalization operators over the rows and columns, respectively, defined as Te X @ 1x1 and T, X X1 where @ denotes elementwise division, sume(-).
(X)) sum, (-) perform column- wise and row-wise summation, respectively.
Weight Sharing Convolutional Neural Networks Having established the foun- dational elements, we now define our Weight Sharing Convolutional Neural Networks (WSCNNs). We let O E R x be a collection of learnable parameters that parametrize the representation stack of the layer / as R = (S* (0)), S rameterize this tensor wherein stochasticity is enforced via K applications the Sinkhorn operator. We also define a set of
Our proposal:
S ()) E [o, 1]|[|x[X/x|X] i.c., we pa- stack of |G' approximate doubly stochastic XX -dimensiional matrices,
learnable base weights g E x X XCin, The WSCNN layer is then simply given by 7 with P and w respectively replaced by R' and 0².
We further note that on image data X can be large, making the discrete matrix form implementation computationally demanding. Hence, we consider semi-direct product group parametrizations for G, in which we let G be of the form (RT +) H. with H/ learnable (approximate) group. Then, the representation stacks will merely be shape |H] XX'] x [X'), with |H| < GG the size sub-grou groug and XX the number of pixels that support the convolution kernel. A WSCNN layer is then efficiently implemented via Conv2D f. R' For group convolutions. (after the lifting layer) the representation stacks will be of shape x ([X") x H x ([X') x [H). Computational scaling requirements can be found in Appendix C.4
5 Experiments
We first demonstrate that the proposed weight sharing method can effectively pick on useful weight sharing patterns when trained image datasets with different equivariance priors. We then proceed to show the method can effectively handle settings where partial symmetries are present in the data, and further analyze the learned weight-sharing structures on a suite of toy datasets. Model architectures, regularizers (norm, ent) and design choices can be found in Appendix C.4 and respectively. analysis of computational requirements can found Appendix
5.1 Image datasets and equivariance priors
We assess the efficacy of our proposed weight-sharing scheme in recognizing data symmetries through experiments on datasets subjected to various data augmentations. Specifically, we evaluate our model MNIST images that have been rotated (with full SO(2) rotations) and scaled (with scaling
7

---

Table I: Test accuracy on MNIST for both rotation and scaling transformations. Additional parameters induced by weight-sharing are marked (+). Parameter counts denoted in millions (M) or thousands (K). Best-performing models (equivalent within < 1% marked bold.
Model
# Params
Sharing Scheme
Accuracy
Rotations
Scaling
CNN
412 K
7/ Zz X C4
98.48 + .08
99.30 + .01
GCNN
103 K
98.96 + .14
97.50 + .15
WSCNN + norm
410 (+ 122 K)
Learned Learned
97.56 + .07
99. 99.27 + .04
WSCNN + norm + ent
410 K (+ 122 K)
98.04 + .11
99.24 + .01
Table 2:
Test accuracy on CIFAR-10. Number of elements denotes the number of group elements used in group convolutional models. Additional parameters induced by weight-sharing are marked (+). Parameter counts denoted in millions (M) or thousands (K). Best Best-performing models (cquivalent within 1%) marked bold.
Model
# Params
# Elements
Accuracy
CNN-32
428 K
70.50 + 0.62
CNN-64
1.66 M
76.29 0.57
CNN-128
6.5 M
78.83 0.01
GCNN
1.63
4
76.72 + 0.26
WSCNN norm
1.63 M (+ 468 K)
4
78.80 + 0.46
WSCNN norm + ent
1.63 M 468 K)
4
76.80 1.40
factors between [0.3, 1.0).. We categorize these datasets based their data symmetry characteristies: MNIST with rotation and scaling as datasets with known symmetries, and CIFAR-10 with flips as dataset with unknown symmetries.
For RotatedMNISST we regard a C4-group convolutional model as benchmark since it has been equipped with subgroup of the underlying data symmetries priori. Additionally, contrast our results with a non-constrained CNN model which has double the number of channels, allowing for free optimization without symmetry constraints. As such, evaluations benchmarked against two distinct models: 1) group convolutional model that equivariant to discrete rotations, embodying fixed equivariance constraints, and standard CNN that adheres only translational equivariance, without additional constraints.
This experimental setup positions the standard CNN the most flexible model lacking predefined inductive biases. In contrast, the group convolutional neural network (GCNN) is characterized by fixed weight sharing, while our proposed weight-sharing CNN (WSCNN) introduces semi-flexible, learnable weight-sharing mechanism. Note that we explicitly distinguish between the number free model and kernel parameters and the additional parameters introduced by weight sharing scheme throughout results (marked by +).
Results can be found in Tab. When there is clear misalignment between the model and data symmetries, the constraints imposed by the model hinder performance, as demonstrated by the C4-GCNN on the scaled MNIST dataset. Notably, our proposed method consistently achieves high performance across all datasets without requiring fixed group specifications. Furthermore, visual inspection of the learned kernels indicates that WSCNN adapts to the underlying data symmetries by effectively rotating kernels, as shown Figure 2 Additionally, analysis of the learned representation stack reveals that closely resembles elements of Cs permutations, further demonstrating the model's capability to internalize and replicate data transformations (see Figure and Appendix B.3)
Additionally, we test the model on CIFAR-10, 10, representing a dataset with possibly more com- plex/unknown symmetry structures, and CIFAR- 10 with horizontal flips (which cannot represented by CN transformations). Results can be found in Tab. 2 with detailed training and model specifica- tions available Appendix C.4 We compare against the (possibly misspecified) C4-GCNN, and several unconstrained CNNs models with varying number of channels: 1) CNN-32 matched in free

---

Table Test accuracy on MNIST with partial rotations.
Table 4: Test accuracy on CIFAR-10 with horizontal flips.
Model
Rot. Range
Accuracy
Model
# Params
# Elements
Accuracy
GCNN
0., 90%
98.84 + .002
CNN-64
1.7 M
79.81 + .001
(0, 180%
98.72 + .001
CNN-128
6,5 M
82.60 + .001
GCNN
1.6
4
76.05 + .004
WSCNN
0, 90º
98.87 .001
0, 180%
99.25 .001
WSCNN
1.6 M (+ 468
4 82.38 .003
kernel size, 2) CNN-64 matched in parameter budget, and 3) CNN-128 matched in effective kernel size (calculated as |G| x channels 4 x 32 128). Despite the kernel constraints in WSCNN, it achieves performance comparable to that of the unconstrained 128-channel CNN (within < 1% accuracy), significantly smaller parameter budget (2.1 M vs. 6.5 M).
5.2 Learning partial equivariances
We show our method is able to pick up on partial symmetries by testing it on MNIST with rotations sampled from subset of SO(2) and compare it to the C4-GCNN. Additionally, show results on CIFAR-10 with horizontal flips, which is a commonly used train augmentation. Results can be found in Tab. and Tab. 4
Additionally, We proceed to test the model's capability to detect data symmetries by applying to suite of toy problems, wherein the datasets comprise noisy G-transformed samples. Details on the data generation framework are provided in Appendix B Our testing employs single-layer setup aimed at learning collection of kemnlls that ideally match each data sample, considering inherent noise. This involves training the model to identify a set of base kernels and their pertinent transformations, effectively adapting to the variations presented by the toy problems.
Discussion and Future Work
We demonstrated method that can effectively identifies underlying symmetries in data, even without strict group constraints. Our approach uniquely capable of learning both partial and approximate symmetries, which more closely mirrors the complexity found in real-world datasets. Utilizing doubly stochastic matrices to adapt kernel weights for convolutions, our method offers a flexible means of learning representation stacks, accommodating both known and unknown structures within the data. This adaptability makes possible detect useful patterns, although these may not always be interpretable in traditional group-theoretie terms due to the absence of predefined structures in the representation stack.
Limitations include computational requirements, which scale quadratically with the size of the group and the kernel size, posing challenges in scenarios with large groups or high-reoolttion data. As such, in this work we have designed the representation stack to be uniform across the channel dimension. However, this prevents learning of other commonly used image transformations such as color jitter. Furthermore, the need for task-specific regularization manage entropy scaling during learning of representations introduces complexity in hyperparameter tuning, which can be barrier in some applications. Additionally, we observed that representations in later layers may show minimal diversity, suggesting that further innovation in regularization strategies might be necessary to enhance the distinctiveness of learned features actoss different layers.
For future work, we aim enhance our method by implementing hierarchical weight-sharing across layers and promoting group equivariance more systematically. One promising direction is to leverage the concept of a Cayley tensor, akin to (20), to identify and reuse learned group structures across different layers of the network. This approach would not only impose more unified and coherent group structure within the model but also potentially reduce the computational overhead associated with learning separate representations for each layer. By encouraging shared group structure throughout the network, we anticipate improvements in both performance and interpretability, paving the way for more robust and efficient symmetry-awe aware learning systems.

---

Acknowledgements
SV and AGC are supported by the Hybrid Intelligence Center, a 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research. The computational results presented have been achieved in part using the Snellius cluster at SURFsara. Additionally, we thank Alexander Timans for helpful discussions.
10

---

References
[1] Ryan Prescott Adams and Richard S. Zemel. Ranking via sinkhorn propagation, 2011.
[2] James Urquhart Allingham, Bruno Kacper Mlodozeniec, Shreyas Padhy, Javier Antorán, David Krueger, Richard E. Turner, Eric Nalisnick, and José Miguel Hemández-Lobato. A generative model of symmetry transformations, 2024.
[3] Daiki Asakura. An infinite dimensional birkhoff's theorem and locc-convertibility. IEICE Technical Report; IEICE Tech. Rep., 2016.
[4] Erik Bekkers, Maxime W. Lafarge, Mitko Veta, Koen A. J. Eppenhof, Josien W. Pluim, and Remco Duits. totoo-raasllttoon covariant convolutional networks for medical image analysis. CoRR, Abs//1804003399 2018. URL htt.htttt//axxii.orr.g./ abs/ 1804. 03393
[5] Erik Bekkers, Sharvaree Vadgama, Rob Hesselink, Putri Van der Linden, and David W. Romero. Fast, expressive se(n) equivariant networks through weight-sharing in position- orientation space. In The Tweifth International Conference on Learning Representations, 2024.
URL http:////ppppueeeeee.eee////rrrp htt.:/oouummndddaPHLLUaUUoe
[6] Gregory W. Benton, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. Learning invariances in neural networks. CoRR, abs/2010.11892 2020. URL https: //arxiv org/ abs/2010.
[7] Garrett Birkhoff' Three observations on linear algebra. Univ. Nac. Tacuman, Rev. Ser. A, 5: 147-151, 1946. httpss/////i att.:/iiii.aa.......ii.... ptt.......pp/////////////6663383
[8] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. CoRR, abs/2104. 13478, 2021. URI. https
//arxiv.org/ /0op///2144.0 13478
[9] Taco S. Cohen and Max Welling. Group equivariant convolutional networks. CoRR, abs/1602.07576, 2016. http:///://arxiv abs/
[10] Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, and Rose Yu. Automatic symmetry discovery with lie algebra convolutional network. CoRR, abs/2109.07103, 2021.
URL htttttttt///aiiiii org/
abs/ wtwg/bs/////eoooooooggoooeeeeera
[11] Mare Finzi, Gregory W. Benton, and Andrew Gordon Wilson. Residual pathway priors for soft equivariance constraints. CoRR, abs/2112.01188,, 2021. /tttpp://aaxxv.orgg//
2112. 01388
[12] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d, 2022.
[13] Alexander Immer, Tycho A. van der Ouderaa, Gunnar Rätsch, Vincent Fortuin, and Mark der Wilk. Invariance learning in deep neural networks with differentiable laplace approxi- mations, 2022. BtLhhttps:/ //axxiv org/
10638
[14] J.R. Isbell. Infinite Doubly Stochastic Matrices.
Canadian
Mathematical Bul-
letin. 5(1):1-4, January 1962. ISSN 0008-4395, 1496-4287.
doi: 10.4153/ CMB-1962 1188-192-00111 001-44. URL htttps://wwt htttp::/oww..camridgg.. cawbridge org/ ottg://ore/prooutt//ieeeettteart
[15) David G. Kendall. On Infinite Doubly-Stochastie Matrices and Birkhoff's Problem 111. Journal
of the London Matherattial Society, -1-35 1960. ISSN 1469-7750. doi: 10. jlms/s 1-35.1335.... 1.81. URL
wwwey. wwt.:y...ooooee//eee/eeeee donp /ww./////aa///ooeeeo 1112 lms// eeprint: http:///oaiilbbaaaaayy.y///////p
81 35 1. 1.81
[16] David M. Knigge, David W Romero, and Erik Bekkers. Exploiting redundancy: Separable group convolutional networks on lie groups. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Muchine Learning Research, pages 11359-1 1136. PMLR, 17-23 Jul 2022. URL https: https:s///rrccedinggs mlr, mml...ress/ press/ v162 htt2//miige22.. html

---

[17] Maxime W Lafarge, Erik J Bekkers, Josien PW Pluim, Remco Duits, and Mitko Veta. Roto- translation equivariant convolutional networks: Application to histopathology image analysis. Medical Image Analysis, 68: 68:101849, 2021.
[18) Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11):2278-23222 1998. doi: 10. 10.1109/5.72679)
[19] Yue Liu. Homomorphism of independent random variable convolution and matrix multiplication, 2023.
[20] Giovanni Luca Marchetti, Christopher Hillar, Danica Kragic, and Sophia Sanborn. Har- monics of learning: Universal fourier features emerge invariant networks. arXiv preprint arXiv:2312.08550, 2023.
[21] Artem Moskalev, Anna Sepliarskaia, Ivan Sosnovik, and Arnold Smeulders. Liegg: Studying learned lie group generators. In Advances in Neural Information Processing Systems, 2022.
[22] Eric Nalisnick and Padhraic Smyth. Learning priors for invariance. In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 366-375. PMLR, 09-11 Apr 2018. URL http:tss////ccccediiggeooo/// mmleeeeeddinnnmm.r.....m. httpeeeedddnasm...rreess//////ee ppess/v84. nalismi ck18a.
[23] Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parameter- sharing, 2017.
[24] David W. Romero and Suhas Lohit. Learning equivariances and partial equivariances from data. CoRR, abs/21 10. 10211, 2021. URL
hatp//aab///....11/ 10211
[25] Révész. A probabilistic solution of problem 111. of G. BirkhofT. Acta Mathematica Academiae Scientiarum Hungarica, 13(1):187-198, March 1962. ISSN 1588-2632. doi: 10.1007/BB002033637.
[26] Sophia Sanborn, Christian Shewmake, Bruno Olshausen, and Christopher Hillar. Bispectral neural networks. arXiv preprint arXiv:2209.03416. 2022.
[27] Jean-Pierre Serre. Linear representations of finite groups. Springer-Verlag, New York, 1977. ISBN 0-387-90190-6. Translated from the second French edition by Leonard L. Scott, Graduate Texts in Mathematics, Vol. 42.
[28] Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochas- tic matrices. Annals of Mathematical Statistics, 35:876-877 879, 1964. URL hhtpss://ppi semanticscholar arg/CorpusID) 120846714
[29] Ivan Sosnovik, Artem Moskalev, and Amold W. M. Smeulders. DISCO: accurate discrete scale convolutions. CoRR, abs/2106.02753 2021. URL http::tttt///aitttt wrw...rg///aa//a a20/2 027333
[30] Emmanouil Theodosis, Karim Helwani, and Demba Ba. Learning linear groups in neural
networks, 2023.
[31] Tycho F. A. van der Ouderaa, Alexander Immer, and Mark van der Wilk. Learning layer-wise equivariances automatically using gradients, 2023.
[32] Rui Wang, Robin Walters, and Rose Yu. Approximately equivariant networks for imperfectly symmetric dynamies. CoRR, abs/2201.11169) 2022. URL http:/t:://axxii.or///////////// 11969
[33] Rui Wang, Elyssa Hofgard, Han Gao, Robin Walters, and Tess E. Smidt. Discovering symmetry breaking in physical systems with relaxed group convolution, 2024. URL httth://////riiv org/ hrggabs//20 abs/23is010 2229992299999)
[34] Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. Advances neural information processing systems, 32, 2019.
12

---

[35] Daniel E. Worrall and Max Welling, Deep scale-spaces: Equivariance over scale. CoRR, 0bbs/195.11199) 2019. URL http:///arrx org/ abs/ 1905. 11697
[36] Jianke Yang. Robin Walters, Nima Dehmamy, and Rose Generative adversarial symmetry discovery, 2023, URL http:pps///ax..oooor/a////////// 00236
[37] Jianke Yang, Nima Dehmamy, Robin Walters, and Rose Yu. Latent space symmetry discovery, 2024. wtt.ltttss//ari..oooo aaapg//abb//// /ww..///.....11...
(38] Raymond A. Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, and Alexander Schwing. Equivari- ance discovery by learned parameter-sharing In Gustau Camps-Valll Valls, Francisco R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelli- gence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 1527- 1545. PMLR, 28 30 Mar 2022. URL httpp::/pproceeddngs.. mlr http:/sw..........se///eeo/ /ress//155/yeeh2b html
[39] Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterizzaion. CoRR, abs/2007.02933, 2020. URL htt.s: /eatps://arxiv. org//os///////////ooo/o /ab0a///.009933
13

---

A Preliminaries
A.1 Irreducible representations
In this section, we closely follow the mathematical preliminaries outlined [34. For a comprehensive reference on Representation Theory, see [27]
Equivalent representations if there exists similarity transform such that:
Two representations p and p' of a group are considered equivalent
Ag E G p(g) = Qp'()\0
where Q represents a change of basis matrix.
Irreps A matrix representation called reducible if can be decomposed as:
p(g) Q0 e (=()))
(i((g) (
p2 p2(9) (o(o))
0-
where Q is change of basis matrix. If the sub-representations P and P2 cannot be further decom- posed, they are termed irreducible representations (irreps). The all irreducible representations of group G is denoted as G. Additionally, any representation p: P.G GV( of a compact group G' can be expressed as:
PGG) = Q
Q-1
jEI
where Z is an index set (possibly with repetitions) over G.
representation p G Rdxd can be viewed collection of d2 functions over G. The Peter-Weyl theorem asserts that the collection of functions formed by the matrix entries of all irreps in spans the space of all square-integrable functions over G. For most groups, these entries form orthogonal basis, allowing any function - R be written as:
Similarly to what is showed in Section 4
f(g) = Σ Σ W.mm.n (d)lp,(9))mx
PGE m.nddd
where d, the dimension of the irrep Pj. while m, n index the entries Pj. Note that this expression corresponds to the inverse Fourier transform and that the coefficients Wjmm. can be obtained by the Fourier transform ff with respect to the basis functions {lp, (g)]mn
Connection with regular representations can be shown that the regular representations can be decomposed using the corresponding irreps as follows:
Preg (g) Q-1
Pj
Ps
where Q performs the Fourier transform, while Q performs the inverse Fourier transform. This implies that when functions f: G - R are considered as vectors in R Gl with a basis where each axis corresponds to a group element, then, as we have seen in Section 4 the group action results in permutation of these axes. However, applying the Fourier transform changes the basis so that G acts independently on different subsets of the axes, resulting the action being represented by block-diagonal matrix, which is the direct sum of irreps.
B
Toy problems
B.1 Data generation processes
For the construction of the toy problems, we look at two types of data-generating processes:
14

---

Equivariant data. Assume a canonical vector & and corresponding label y. We assume these vectors transform under known group such that the data- generatinn process follows:
Sample group action: 9 ~~ p(G),
Apply group action to feature with noise: x = /² (g) 110)+++ +
+e
Apply group action label: y (9))
with p², pD the representations of acting on the feature and label space, respectively.
Co
ww
CO
C2
C2
C0
C2
C1 mww
C2
C0
C2
Data
Labels
Data
Labels
(a) Equivariant task. The labels transform with the data.
(b) Invariant task. The labels are fixed.
Figure 4: Samples the two tasks.
Evaluation metrics To assess whether our method effectively captures the underlying data symme- tries, we analyze the learned weight-sharing structure by comparing known ground truth patterns. Since our model does not impose associativity or any strict group constraints on the representation stack, may learn to represent mixtures interpolations group elements. Hence, in general, we will not observe relevant algebraic structure if we produce Cayley table based on the learned representations as done 226 20
Therefore, we will use an approach that allows us to capture the flexibility of our representations. To this end, we will examine each representation matrix to determine how closely it resembles convolution matrix [19] associated with a random variable defined over some specific group G. We
Jk==1 employ the set of group actions from G, represented doubly stochastic tensors ) reference framework to quantify the fit and alignment of our model's 's representations with these predefined group actions. As such, for each learned weight sharing tensor R calculate the fit
P: = Σ e& P and acquire coefficients ej > O, such that Σ/e in a constrained linear regression setup by minimizing |P, R/Il2-
B.2 Additional results: toy problems
conducted experiments on various signals subjected to different transformations, including: 1D signal with cyclic shifts (exemplary samples shown in Fig. 4a). 2D signal with Cs rotations (illustrated in Fig. 77). and 3D voxel grid enhanced by cube symmetries. each scenario, the learned kernel stack accurately matched the data samples, achieving perfect accuracy. Figure
15

---

0006600
P
P
Shift-dataset with uniformly sampled
(b) Shift-dataset with partially sampled group elements.
group elements.
Figure 5: Coefficient responses of learned representations and their base transformations.
displays the learned representations for localized shifts in the ID signal, while Figure presents the learned kernel stack for the 2D signal dataset.
Furthermore, assess whether our model can identify partial group structures, we evaluate it using two distinct datasets: ID signal enhanced with cyclical shifts, and a 3 x x 3 voxel grid subjected to rotations from C4 x C4. For the shift dataset, we utilize the complete set cyclical shifts as the ground truth representations. Figure a, b) displays the coefficients for the shift dataset. Specifically, part (a) shows coefficients when trained uniformly across group elements, while part (b) illustrates coefficients using only the first half of the group elements, where the group longer retains cyclic properties or satisfies closure. Given that our method does not assume cyclic groups or group closure, it effectively captures the relevant group transformations even for partial transformations. App. B.4 shows the coefficients for the base representations of C4 x C4 x C4 cube symmetries. Since the data augmentation only applied C4 C2 transformations, the method predominantly identifies elements corresponding to these transformations, as highlighted by the red line.
Figure Representations learned for ID equivariant shift task
t
Figure 7: Samples of the 2D-signal dataset.
16

---

*
Figure 8: Equivariant task for flattened rotated 2D signals. Learned kernel stack
B.3 Visualization of G-Cony layers
Figure 9 displays the ground truth permutation matrices that implement a shift-twist operator, which the group transformation that underlies regular group convolution operator for rotations. Figure 10 illustrates the corresponding matrices learned for each learnable weight sharing layer on the rotated MNIST dataset. The learned matrices closely show similar patterns as the shift-twist operator, suggesting the model's ability to capture such transformations from training data.
17

---

Figure 9: Ground-truth permutation matrices for C₁ rotations for 5 x spatial kernel, implementing shift-twist operator.
Figure 10: Learned representations for the weight-sharing G-cony layers. Top to bottom: first to last layer learned representation stacks.
B.4 Additional results: learning partial equivariance
shows the coefficients for the base representations of C4 C4 Ca cube symmetries. Fig. 11
The x-axis quantifies the permutation representation for each element consisting of the number of 90-degree flips around each x, y or axis.
18

---

14
13
12
10
9
Ri
8
4
3
2
(0. 0,
[1.
(2,
0,
(0, 0)
(1,1,
(2, 0)
(3
2, 0)
21 0)
22, 0)
(3,2, 0)
10,3, 0)
21. 3,
(2, 0)
3, 0)
(0, 0, 1)
(1. 0, 1)
(2. 0, 1)
(3, 0, 1)
(0, 10,2, 1)
(1,2, 1)
(2, 1)
2, 1)
P
Figure 11: Cube dataset with C4 x C4 rotations with C4 X C4 X C4 base elements.
B.5 Convolution matrix decomposition for Fig. 3
3
2
R
1
0
90
180
270
gt
R
Figure 12: Coefficients for C4 representations for 25 and learned representations the lifting layer of the rotatedMNIST experiment.
C Architectural details
In the current section, we outline some design choices used across all experiments. Code is available sharing Firstly, when defining the learnable representation stack R, we have found that anchoring an identity element aids distin- guishing the base kernel from its potential transformations. This approach is based on the intuition of starting with learnable base kernel and subsequently learning its transformations and we have found that it aids optimization.
at httph:: //tpthu.// com/ computri/ ceppputrr/leeaaalll
19

---

C.1 Regularizers
We test two regularizers on the representations R. which are similar to those used by 38
Entropy Regularizer: The primary motivation for using the entropy regularizer is to encourage sparsity our bright-sharrnn schemes, which helps the matrices approximate actual permutation matrices rather than simply being doubly stochastic. This approach stems from the intuition for some transformations, the weight-sharing schemes should mimic soft-permutation matrices. The effectiveness of this sparsity depends on the specific group transformations relevant to the task for example, C4 rotations are typically represented by exact permutation matrices. In contrast, CN rotations scale transformations might require interpolation, thus aligning more closely with soft permutation matrices. Our experimental results indicate that the utility of this regularizer varies with the underlying transformations in the data; for instance, it not beneficial for scale transformations the MNIST dataset, anticipated. The entropy regularizer is of the following form:
N.D.D Σ
ent (R)
Rijs (og(R))))
Normalization Regularizer: Empirically, have found the normalization regularizer essential for reducing the number of iterations needed by the Sinkhorn operator to ensure the matrices to achieve double stochasticity or require an excessively high number of Sinkhorn iterations to do so. The normalization regularizer is of the following form:
and column-normalin
Without this regularizer, the tensors either fail
1
norm R
sum, (R) + sum. (R)
D
Where sum,, sum, are defined as in 4.2
C.2 MNIST
Model architecture For all MNIST experiments, simple 5-block CNN used. Each block uses a kernel size of 5 and is succeeded by instance norm and ReLU activation, respectively. After the final convolution block, any spatial and group dimensions are reduced through global average pooling operation, and a single linear layer is used as a classification head. For the group convolutional model and weight sharing model, the default hidden channel dimension in the blocks set 32 unless otherwise stated, and 64 in the regular CNN models.
Training details The models used learning rate of le-2 and were trained for 100 epochs, All the experiments were done on a single GPU with 24GB memory under six hours.
C.3 CIFAR10
Model architecture We used the ResNet architecture in [16] Appendix B.I, except that swapped the final global max pooling operator with global mean pooling. However, in contrast to [6], regular discrete kernels instead of continuous kernel parameterizations.
Training details Following [16,, we trained the models for 200 epochs using learning rate of le-4. All the experiments were done single GPU with 24GB memory under six hours.
C.4 Computational Demands
Fig. 14 show the computational scaling analysis of our weight sharing layer, comparing it to group convolutional model of the same dimensions. We highlight that regular group convolutions can be implemented via weight-sharing schemes, resulting in equal computational demands for both approaches. Since the weight-sharing approach applies the group transformation in parallel across all elements (as a result of matrix multiplication and reshape operations), our method can prove quite efficient. Regarding memory allocation, group convolutions often implemented using for-loops
20

---

over the group actions, and this sequential method imposes less heavy memory burden since the operation is applied in series per group element. However, although scaling is quadratic w.r.t. the number of group elements and the domain size for weight-sharing, we mitigate this issue by use of the typically lower-dimensional support of convolutional filters (i.e., and ), rendering our approach practical for wider range of settings.
362
8
Memory (C8
16
Memory (G8)
16
kernel size
kamel sizc
(a) C4-GCNN layer
(b) WSCNN layer
Figure 13: Memory allocation inference time for C4-GCNN layer and a weight sharing layer, for different number of group elements |G] and different kernel sizes. The input is 32 x 3 x 100 100 (batch channels x height x width)
XI
Time (5)
16
Time (5)
16
kennel
kennel
(a) C4-GCNN layer
(b) WSCNN layer
Figure 14: Execution time at inference time for a C4-GCNN layer and a weight sharing layer, for different number of group elements GG] and different kernel sizes. The input is 32 3 x 100 100 (batchx channels x height x width)
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification: We do not give new theoretical results per see, the proposed method has theoretical formulation, that is give in detail the paper and appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipp/.cc/pullc//uuuii wwwiii/gguie//eeeeeeeeeic CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: haapas:///uripps/ http///uris.ccpppupppp/eee http:///.ccc.uccccttttctttttteea
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
We do not use proprictary datasets or code
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppsiipteeeee www.reeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
26

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "gNI5au91E8", "reviewer": "Reviewer_1myY", "summary": "This paper introduces a parameterisation that contains the ability to represent weight tying corresponding to arbitrary (?) group equivariances. In practice it can represent interpolations of weight tying, but it is argued that this is a feature (not a bug!) since strict equivariance is often too strong a constraint to place on a model for it to still fit the training data.\n\nStrict equivariance is known to be represented by a permutation matrix (correct me if I'm wrong), which justifies the need to use doubly stochastic matrices to interpolate. This makes the edges of the parametisation strict equivariances. (For equivariances of discretised continuous signals this is not quite true, as the permutation in continuous space would need to be approximated by some interpolation in the discrete space)\n\nThis leads to a new parameterisation of a weight structure in a layer that can be trained in the usual way. The doubly-stochastic matrices can then be investigated to see if equivariance is actually learned.\n\nThe experiments implement the method, and run on benchmark datasets and synthetic datasets, showing good performance, and somewhat interpretable group structure appearing. It is unclear what the actual point of the experiments is, since there are many reasons to use equivariance, but the experiments are not phrased in terms of this (see discussion).", "questions": "- Can this parameterisation represent arbitrary group equivariances? Is the representation of every strict equivariance a permutation matrix? It would be helpful to be explicit about how general this really is.\n- Am I right in understanding that this method ultimately just parameterises a low-rank weight matrix over different feature channels?\n- Can the benefit that equivariance promises in the low-data regime still be provided by this method when the invariance is learned? Since effectively, you're just parameterising weights in a different way (low rank?). In rotationally equivariant settings in low-data, could these weights not just overfit, rather than learning to rotate filters? Would this not lose an important benefit of equivariance?\n- How important is the doubly-stochastic nature of things? Could you just run an experiment without the Sinkhorn component at all?\n- Can you give a very short (ideally 1 sentence, or a 2-3 sentences) summary of the quantitative claims that are made about the method, that are verified in the experiment section?", "limitations": "See above.\n\nOverall, this is a really interesting idea. I do have some concerns about the evaluation. These concerns are large in the scheme of determining how well this method really works relative to clearly formulated claims, but small relative to typical approaches in the ML community.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The problem of learning equivariances is very important, as it would remove a significant difficulty in designing networks with the correct inductive biases. The solution is flexible, as any (?) group structure can be represented by the parameterisation. There is also an elegant solution to the problem of needing a large number of parameters, that will work in practice for image data: Assuming translational equivariance, and only parameterising additional equivariances on the filters that are much smaller than the image.", "weaknesses": "Overall, the paper presents a well-reasoned method to an important problem, and I do believe that it meets the standard for publication at NeurIPS.\n\n**Method & Presentation**\nThe method is well-justified. However, a final summary of what a forward pass through a layer looks like was not given, and would be really helpful. In addition, it would be helpful to have a clearer discussion of how many additional parameters are added (beyond lines 220 onwards), with the architectures that are discussed given as an explicit example.\n\nEssentially, one thing which seems to be the case, but is not explicitly acknowledged, is that this method collapses to just a special parameterisation of weight matrices, where the weights have low-rank combined with doubly-stochastic structure. The low-rank-ness is shown in eq 6. While this is a simplistic way of looking at the method, it does give a helpful alternative view. Making this explicit would help the paper.\n\n**What is the claim of the experiments?**\nThe experiments are the main weakness of the paper. Some qualitative results about the structure of the learned weights are given, which are helpful. But it is not clear what the quantitative claim of the experiments is. Equivariance can help in several ways, e.g. better out-of-distribution prediction, better prediction at low data, or smaller/compacter models. So is the claim that the equivariance inductive bias helps, and it can be discovered automatically? But in this case this is not disentangled from the model capacity. Perhaps a normal CNN would perform better if it were just made larger! This is additionally indicated that the baseline 70% accuracy on CIFAR 10 is low compared to what other non-group-equivariant methods can achieve. This unclarity also exists in the synthetic experiments, where the size of the dataset is not discussed. Low-data experiments could help here, since it's easier to make the model large enough that size doesn't help any longer, which isolates inductive bias only.\n\nThe CIFAR experiments show that making the model larger improves performance. How can we be sure that this is really the benefit of learning equivariances, rather than adding more capacity? Another experiment that is necessary here, is a comparison to a weight structure that does not have the doubly-stochastic constraint enforced on it. This would allow the effect of simply adding capacity to be tested.\n\nAlternatively, a low-data experiment would allow the generalisation capabilities of the model to be tested (e.g. [Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations](fig 3 in https://proceedings.neurips.cc/paper_files/paper/2022/file/50d005f92a6c5c9646db4b761da676ba-Paper-Conference.pdf)). This could be done on MNIST variants, where currently the differences are so small, it is hard to draw conclusions.\n\nThis same issue pops up in the synthetic experiments. What is the dataset size? If the dataset is so large that all transformed signals are in the dataset, even a fully-connected network would learn the correct function. A low-data experiment is needed to really show that truly an equivariance has been learned that can help to _generalise_. Alternatively, you need to argue a benefit on the basis of parameter count.\n\nIn summary: The claims that the experiment section support are not clear. The field of equivariances is mature enough that the potential benefits have been clearly described, and these need to be clearly evaluated in experimental sections.\n\n**Related Work**\nThe idea of relaxing equivariance by placing a distribution over transformations is older than the papers currently cited. E.g.:\n- [Local Group Invariant Representations via Orbit Embeddings](http://proceedings.mlr.press/v54/raj17a/raj17a.pdf)\n\nThe discussion of \"symmetry discovery methods\" is unclear. What data do these methods need, or what kind of training signal do they use? What kind of predictive improvements do they obtain? Is the goal of these papers the same as those in the previous paragraph? Or, is the way that these methods learn group structure different from those in the previous section? If so, how?\n\nMethods that learn a degree of equivariance on a layer-by-layer basis are relatively new, and it would be good to discuss this explicitly. E.g.:\n- [Residual Pathway Priors for Soft Equivariance Constraints](https://proceedings.neurips.cc/paper/2021/file/fc394e9935fbd62c8aedc372464e1965-Paper.pdf) Finzi et al is not mentioned at all, but does allow partial equivariance in a layer-by-layer way.\n- Reference [26] \"Learning layer-wise equivariances automatically using gradients\" is cited, but only in the context of arguing that overly constrained models suffer poor performance, but not in the context that this paper also discusses how to learn the right equivariance to use. This author has more papers on learning invariance/equivariances that may be relevant.\n\nOverall, the literature review misses a lot of relevant work. The suggestions I gave are off the top of my head and I definitely missed some important papers too. However, it is the responsibility of the authors to put the time and effort into going beyond this to give a more thorough overview.\n\n**Minor**\n- _\"Requiring no prior knowledge of the possible symmetries.\"_ (line 52) It is true that earlier methods (with the exception of [31]) could only pick between groups that were completely specified a-priori. While this paper _in principle_ does provide a parameterisation that can _search_ over a much wider space, this space needs to be limited for scalability reasons, and it was not demonstrated that the method would work reliably without this \"prior\" being added!\n- It would be really helpful to have a full discussion of the impact on the number of parameters, for these specific experiments.Needs a discussion of the total number of parameters\n- Is there a typo in params in table 1? Under \"Params\" should \"103 + 265K\" be \"103K + 256K\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response:**\nDear reviewer, we thank you for your constructive feedback and engaged questions, which we address in the following.\n\n- We thank the reviewer for pointing out the connection of our proposed method to low-rank approaches. It is possible to make our doubly stochastic matrices converge to permutation matrices with an appropriate regularization technique. In those cases we obtain sparse matrices that could be represented via low-rank approximation methods. However, we note that the degree of regularization is task- and domain-specific.\n- Regarding experimental claims, our claim is indeed that the inductive bias introduced by equivariance is beneficial, and can furthermore be discovered automatically [Ref Fig 1 a,b and Fig 4 in the PDF] For natural images, the bias of translation equivariance may be sufficient in many cases, in particular when enough channels are present. This is not the case for other datasets, where the data itself may contain group symmetries beyond translation, as shown by [20, 27]. Our main claim is the ability to uncover such symmetry groups and employ them in different downstream tasks. While a look-up table may match the existence of an *a priori* specified symmetry group, this is not the case for partial symmetries or incomplete ones. In such cases, we demonstrate that we can discover the symmetry in the sense of a useable shared representation without requiring to point out the exact subgroup (if such a subgroup even exists).\n- We acknowledge that conventional CNNs with an equivalent channel count inherently possess a higher expressive capacity due to a lack of kernel constraints. Therefore, we have opted to compare overall parameter counts rather than the number of kernels, a more comprehensive assessment in our opinion. For completeness, we have additionally measured the performance for an unconstrained 128-channel CNN (please consider point 4 in the general rebuttal).\n- We thank the reviewer for the suggestion on experimenting with the low-data regime. We agree that the size of the dataset plays an important role, and we have added the suggested experiments for both cases of full and partial symmetry discovery (please consider point 2 in the general rebuttal and Fig. 2 in the attached PDF).\n- Regarding additional literature review, we will expand the related works section and further clarify the position of our work within the literature for the camera-ready version. We would like to stress that we do *not* explicitly impose any group structure in our representation stack, since that would defeat our goal of symmetry *learning* and enables us to potentially learn a weight sharing scheme that violates some of the group axioms if the group symmetry is not present in the data. Therefore, we refrain from making any theoretical assumptions that make use of more complex statistical group theory results. We agree that we could have been more precise regarding these claims in the paper, and will clarify this. Our method primarily aims to learn symmetries from data explicitly through a weight sharing scheme which *may or may not* be an exact group symmetric structure (if the data does not possess a full group structure). Then, we implicitly assign several probability measures of finite support over this learned symmetry scheme. We emphasize that our method is therefore more flexible than a strict group equivariant network due to its ability to learn partial symmetries.\n\nRegarding your explicit questions:\n\n- (Question 1) Yes, this parameterization can represent arbitrary group equivariances for compact Lie groups. For strict or partial equivariance, we employ a permutation matrix.\n- (Question 2) We hope that our first point in the response above addressed this question.\n- (Question 3) In the low-data regime, the model might overfit to a limited set of group transformations. This can be mitigated with a proper regularizer. We would argue that this is an advantage for learning partial or incomplete symmetries, as we make no assumptions on completeness or uniform distribution over group transformation in the data. In those cases, strict equivariance can hurt performance and may also be an overconfident bias lacking the data evidence to support it. On the other hand, our method allows us to address this issue by learning directly from any symmetries that are present in the data, without the need of such assumptions on, e.g., completeness of a group structure or uniform distribution over group transformation.\n- (Question 4) Zhou et al. [31] essentially learn a parameter sharing scheme without constraints on the sharing pattern except norm regularization. However, they report that they require meta-learning to pick up on symmetries, while our method does not. We presume that the added constraint on double stochasticity is beneficial in that regard.\n- (Question 5) To briefly summarize in two points: (1) The model can recover group convolutional structures when clear symmetries, including partial symmetries, are present in the data (see our experiments on Rotated MNIST); and (2) The model can pick up on useful weight sharing patterns when there are no *a priori* known symmetry structures. This is shown in our CIFAR-10 experiments, where we match the CNN’s model performance and outperform a model with pre-specified group equivariance. This is additionally underlined in Point 1, 2 and 4 in the general response.\n\nWe thank you for your insightful comments and would appreciate you considering raising your score if you believe they have been addressed sufficiently, or follow up with any questions we may help clarify further.\n\n**References**\n\n[20] Learning equivariances and partial equivariances from data. David W. Romero and Suhas Lohit. NeurIPS 2022\n\n[27] Approximately equivariant networks for imperfectly symmetric dynamics. Rui Wang, Robin Walters, Rose Yu. ICML 2022\n\n[31] Meta-Learning Symmetries by Reparameterization. Allan Zhou, Tom Knowles and Chelsea Finn. ICLR 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2LHcgxk9Ij", "reviewer": "Reviewer_3Y5V", "summary": "The paper proposes a symmetry discovery through learning parameter-sharing in weight matrices. The parameterization relies on relaxing underlying permutation matrices by transforming them as doubly stochastic matrices. In combination with additional regularization, the parameterization can be used to successfully discovery symmetries in data, improving generalization performance.", "questions": "-Regularization\nFor experiments, it would be helpful if it is clear how the strength of this regularizer is chosen. Cross-validation?\n-Analysis of learned symmetries\nIt would be interesting to better understand what weight-sharing is being learned for CIFAR-10, apart from merely measuring improved test accuracy. Is there an analysis on the learned symmetry?\n-Regularization\nEntropy regularization seems to hurt on CIFAR-10, but improve on MNIST experiments. Do authors have an intuition on why this is the case?", "limitations": "The method proposes an elegant and novel symmetry discovery method. I expect the proposed parameterization to be beneficial to the community. The paper could improve a bit on the objective function side of things (seems to rely on directly encouraging symmetry, like some prior work). Apart from the remaining questions, the paper makes for a strong contribution.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The method appears to be very elegant in offering a natural relaxation of underlying weight-sharing scheme through Sinkhorn operator. The paper is well-written and provides nice illustrations which guide the reader in understanding the proposed method. Then, the paper provides thorough empirical validation demonstrating usefulness of the approach in practice.", "weaknesses": "-Regularization.\nThe paper proposes a novel parameterization that can be used for symmetry discovery. In terms of objective, the work relies on direct regularization to avoid equivariants solutions, similar to some other symmetry discovery works. It has been shown in prior work that this strategy can have issues, since it introduces an additional hyperparameter which may need additional tuning (thereby is less ‘automatic’). In terms of explaining the methodology, the paper would benefit from some discussion on the role of the used regularizer.\n\n-Scalability.\nThe paper seems to scale quadratically in |X|. Is this not an issue? How does this compare against scaling of alternative methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we thank you for your time and valuable comments, which we address in the following.\n\nRegarding employed regularizers, we acknowledge that the motivation could have been more explicitly mentioned, and we provide some added details:\n\n- Entropy Regularizer: The primary motivation for using an entropy regularizer is to encourage sparsity in our weight-sharing schemes, which helps the matrices approximate true permutation matrices rather than “simply” satisfying double stochasticity. This approach stems from our initial intuition that weight-sharing schemes should mimic soft permutation matrices. The effectiveness of the induced sparsity depends on the specific group transformations relevant to the task. For example, $C_4$ rotations are typically represented by exact permutation matrices. In contrast, $C_N$ rotations or scale transformations might require interpolation, thus aligning more closely with soft permutation matrices. Our experimental results indicate that the utility of this regularizer varies with the underlying transformations in the data. For instance, as anticipated it is of lesser use for scale transformations in the MNIST dataset.\n- Normalization Regularizer: Empirically, we have found the normalization regularizer essential for reducing the number of iterations needed by the Sinkhorn operator to ensure the matrices are row and column-normalized. Without this regularizer, the tensors either fail to achieve double stochasticity or require an excessively high number of Sinkhorn iterations to do so.\n\nWe will outline our motivation behind these two regularizers and the impact of varying $\\lambda$ values for the entropy regularizer more explicitly in the camera-ready version.\n\n- Regarding concerns about scalability: while we agree that quadratic scaling in $|X|$ can be impractical, potentially leading to polynomial $O(s^2)$ scaling in terms of image width $s$, our method exploits the fact that convolutional kernels have limited localized supports in practice. Thus, for a kernel size $k$ we scale with $|X| = k^2$ rather than the entire signal domain. This approach significantly reduces computational demands and makes our approach practically scalable. Please also consider point 5 in the general rebuttal for this aspect.\n- Regarding the interpretation of the weight-sharing patterns learned on CIFAR-10, it is important to note that our model does not impose algebraic or fixed group constraints on the learned representations. This design choice enhances the flexibility and abstraction of the weight-sharing schemes. As a result, the weight-sharing patterns may not be easily interpreted using conventional group-theoretic approaches, which could affect the direct visualization clarity of the learned weight-sharing.\n\nWe thank you for your insightful comments and would appreciate you considering raising your score if you believe they have been addressed sufficiently, or follow up with any questions we may help clarify further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CSKlvPxwxS", "reviewer": "Reviewer_uLBU", "summary": "Overview:\nThis paper claims to build upon a long line prior work on symmetry detection and learning equivariant kernels. [18,19,20,22, 25-31. Eespecially reference [30] which is an identical weight and parameter sharing scheme that learns  and discovers equivariances  solely by row-stochastic entries.  This paper's idea is to enforce both row and column stochasticity through  use of the Sinkhorn operator  to achieve equivariance discovery and applicable to  more interesting data domains such as images.\n\nAdvantages:\n* The paper addresses an important problem of  Kernel equivariance via symmetry detection and weight  in CNN . However its marginal progress over prior work. \n\nWeaknesses:\n* Comparisons with prior work are missing.\n* The quantitative advantage of  double stochasticity from row and column has advantages for metric spaces (images),. However this is not explained nor quantified through comparison.\n* Moreover while ground truth tests are done for some toy problems, the quantitative generalizability of the double stochasticity over single stochasticity are never delineated, nor proved.\n\nQuestions:\nThis paper is identical to  this ICML 2024 workshop paper.   https://scholar.google.com/citations?view_op=view_citation&hl=en&user=gfRkDXEAAAAJ&citation_for_view=gfRkDXEAAAAJ:ufrVoPGSRksC ?\nAlso why no comparisons to even reference [30] ?\n\nMissing References:\n* Several references are missing, including the entire suite of probabilistic symmetry detection and equivariant NN is not discussed. See for e.g.   @misc{bloemreddy2020probabilisticsymmetriesinvariantneural,\n      title={Probabilistic symmetries and invariant neural networks}, \n      author={Benjamin Bloem-Reddy and Yee Whye Teh},\n      year={2020},\n      eprint={1901.06082},\n      archivePrefix={arXiv},\n      primaryClass={stat.ML},\n      url={https://arxiv.org/abs/1901.06082}, \n}", "questions": "Questions:\nThis paper is identical to  this ICML 2024 workshop paper.   https://scholar.google.com/citations?view_op=view_citation&hl=en&user=gfRkDXEAAAAJ&citation_for_view=gfRkDXEAAAAJ:ufrVoPGSRksC ? Is this ok ??\n\nAlso why no comparisons to even reference [30] ?\n\nMissing References:\n* Several references are missing, including the entire suite of probabilistic symmetry detection and equivariant NN is not discussed. See for e.g.   @misc{bloemreddy2020probabilisticsymmetriesinvariantneural,\n      title={Probabilistic symmetries and invariant neural networks}, \n      author={Benjamin Bloem-Reddy and Yee Whye Teh},\n      year={2020},\n      eprint={1901.06082},\n      archivePrefix={arXiv},\n      primaryClass={stat.ML},\n      url={https://arxiv.org/abs/1901.06082}, \n}", "limitations": "There are no attempts on analyzing the generalizability of the operator to varying initial and boundary conditions or Reynold numbers.\n\nOther potentially relevant paper to consider:\nhttps://www.sciencedirect.com/science/article/pii/S0021999123001997\nThis paper uses error-correction in neural operators based on the residual.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper enforces both row and column stochasticity through  use of the Sinkhorn operator  to achieve equivariance discovery.  However it's not clear what quantitative benefits this paper's method makes.", "weaknesses": "Weaknesses:\n* Comparisons with prior work are missing.\n* The quantitative advantage of  double stochasticity from row and column has advantages for metric spaces (images),. However this is not explained nor quantified through comparison.\n* Moreover while ground truth tests are done for some toy problems, the quantitative generalizability of the double stochasticity over single stochasticity are never delineated, nor proved.\n\nQuestions:\nThis paper is identical to  this ICML 2024 workshop paper.   https://scholar.google.com/citations?view_op=view_citation&hl=en&user=gfRkDXEAAAAJ&citation_for_view=gfRkDXEAAAAJ:ufrVoPGSRksC ?\nAlso why no comparisons to even reference [30] ?\n\nMissing References:\n* Several references are missing, including the entire suite of probabilistic symmetry detection and equivariant NN is not discussed. See for e.g.   @misc{bloemreddy2020probabilisticsymmetriesinvariantneural,\n      title={Probabilistic symmetries and invariant neural networks}, \n      author={Benjamin Bloem-Reddy and Yee Whye Teh},\n      year={2020},\n      eprint={1901.06082},\n      archivePrefix={arXiv},\n      primaryClass={stat.ML},\n      url={https://arxiv.org/abs/1901.06082}, \n}", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we thank you for your time and comments, which we address in the following.\n\nRegarding the motivation for double stochasticity and its comparison to prior works, we point out that there are key benefits about employing double stochasticity over single stochasticity:\n\n- Theoretical motivation: We establish a link between regular representations which forms the basis of group convolution actions, and their manifestation as soft permutation matrices. Using double stochastic matrices allows for an analysis of learned representations through the lens of partial equivariance, a connection we elaborate on in Section 4.2. We are happy to motivate this in more detail in the camera-ready version.\n- Practical Implications: As outlined in Section 4.2, our methodology employs doubly stochastic matrices for weight-sharing, which facilitates direct calculation of expectations from the implicitly learned distribution. This setup avoids the need for Monte Carlo approximations and is adaptable to unrecognized symmetries that may not conform to standard group structures. This approach to handling partial equivariance offers a novel angle compared to traditional symmetry discovery methods, such as those proposed by Yeh et al. [30] and Zhou et al. [31].\n\nHence, we propose that there is a well-grounded preference for employing double stochasticity over single stochasticity in the context of group convolutions. Following your comments, we have added the row-normalized baseline from [30] on rotated MNIST in the additional results (please consider point 3 in the general rebuttal, and Table 2 in the attached PDF).\n\n- As per author guidelines on dual submissions (section on Dual Submission, [**https://neurips.cc/Conferences/2024/CallForPapers**](https://neurips.cc/Conferences/2024/CallForPapers)) it states that *“Papers previously presented at workshops are permitted, so long as they did not appear in a conference proceedings (e.g., CVPR proceedings), a journal or a book.”* The work referenced in your review was submitted as an extended abstract at the ICML workshop “Geometry-grounded Representation Learning and Generative Modeling” (https://gram-workshop.github.io/cfp.html) which is **non-archival, and hence does not violate dual submission policy.**\n- Regarding the stated limitation, we appreciate the remark but do not focus on initial and boundary conditions as we do not use PDEs as a use case for our method in this work. If we were to extend our proposed method to PDE solving, we would require updating our architecture to accommodate boundary conditions and initial conditions to solve any PDEs efficiently. As far as the scope of this current work is concerned, we *learn* the symmetries or partial symmetries directly from the data through weight-sharing schemes, which do not require explicitly defining any boundaries.\n\nWe thank you for your insightful comments and would appreciate you considering raising your score if you believe they have been addressed sufficiently, or follow up with any questions we may help clarify further.\n\n**References**\n\n[30] Equivariance Discovery by Learned Parameter-Sharing. Raymond A. Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, Alexander Schwing. AISTATS 2022\n\n[31] Meta-Learning Symmetries by Reparameterization. Allan Zhou, Tom Knowles and Chelsea Finn. ICLR 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bew6qi3YXC", "reviewer": "Reviewer_xU9K", "summary": "In contrast to many works that impose strict architectural constraints to parameterize neural networks that are exactly equivariant to a known group, this work considers learning approximate equivariances to unknown groups. This is done using soft weight sharing schemes, where doubly stochastic tensors are learned and applied on canonical weights. The approach generalizes GCNNs, and is shown to learn natural symmetries in experiments.", "questions": "1. How do the learned filters of WSCNN look on CIFAR10?\n2. Do learned representations ever look the same across different \"group\" elements?", "limitations": "Some discussion in section 6", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Nice background section. In my opinion, it is written much better than other related papers in the area.\n2. The method is a natural generalization of group convolutions, which is developed via the neat perspective of group convolutions as applications of transformations of a canonical weight tensor. It is nice that you can choose the number of \"group elements\", so that e.g. non-group-symmetries can be captured.", "weaknesses": "1. Section on \"Regular representaitons allow for element-wise activations\" is a little out of place and imcompletely justified. The claim that non-elementwise activations \"in practice are not as effective as the classic element-wise activations\" is a strong statement that needs more specific justification.\n2. Direct parameterization of the kernels for each group element is expensive. Empirical runtime and memory analysis would be appreciated, to see the effect of this.\n3. Empirical results are rather limited and weak. Few baselines are considered (see below as well), the baselines already perform well on these tasks, and WSCNN does not improve over the baselines in Table 1. Given that the method does not appear to be too scalable (see above), it is unclear where it would be useful in improving performance.\n4. Other baselines (besides standard CNNs and GCNNs) and ablations are missing. For instance, one can imagine removing the sinkhorn projection onto the doubly stochastic matrices. If we initialize the $\\Theta_i^l$ to be permutation matrices (your fixing of one representation to be identity is related to this), then this may also perform similarly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response:** \n\nDear reviewer, we thank you for your time and valuable comments, which we address in the following.\n\n- On the use of element-wise activations: Regular group convolutions offer the benefit of having scalar feature field co-domains, permitting the use of any point-wise activation function without compromising equivariance [1, 2]. In contrast, steerable methods employ irreducible representations, which do not necessitate domain discretization [2]. However, these methods restrict the use of general point-wise activation functions, requiring specialized ones instead, which can be computationally expensive. This may additionally introduce discretization artifacts that disrupt exact equivariance, ultimately constraining the method’s expressivity [2].\n- Regarding computational analysis, we have included such an analysis comparing our weight-sharing layer to a conventional group convolution of comparable size in the additional results, please consider point 5 in the general rebuttal, and Fig. 2 and 3 in the attached PDF.\n- Regarding our empirical results and the usefulness of our method: we present additional experiments that demonstrate our method matches the performance of an unconstrained CNN with the same amount of effective but free kernels (128-channels) at a lower parameter budget (please also consider point 4 in the general rebuttal). In addition, we can outperform group convolutional methods for the discovery of partial and unknown symmetries. We also address this in points 1 and 2 of the general rebuttal, and Fig. 1a, 1b and 4 in the attached PDF. We hope this sufficiently addresses your concerns about experimental validation.\n- If we initialize the weight-sharing schemes as permutation matrices implementing a specific group transformation, we indeed recover a standard group convolution. A fair comparison of baselines would be comparing to methods that use weight-sharing for images, such as reference [31]. However, they report difficulties with end-to-end training and require meta-learning, making a direct comparison with our approach less straightforward. Following your comments, we have added the row-normalized baseline from [30] on rotated MNIST in the additional results (please consider point 3 in the general rebuttal, and Table 2 in the attached PDF). Visual inspection of the weight-sharing schemes seem to imply it is less effective in picking up meaningful group structures.\n\nRegarding your explicit questions:\n\n- (Question 1) Regarding the learned representations in the CIFAR-10 experiments, we highlight that our model operates *without* applying specific algebraic or group constraints to the learned representations, resulting in a more abstract and adaptable weight-sharing scheme. As a result, the patterns in weight-sharing are not as readily interpretable through traditional group-theoretic approaches, and it also makes direct visual interpretation of these patterns more nuanced.\n- (Question 2) While we do not put any explicit regularization on the diversity of the learned representations, we observed in our experiments that our method learns distinct weight-sharing patterns.\n\nWe thank you for your insightful comments and would appreciate you considering raising your score if you believe they have been addressed sufficiently, or follow up with any questions we may help clarify further.\n\n**References**\n\n[1] General E(2)-Equivariant Steerable CNNs. Weiler et al. NeurIPS 2019\n\n[2] Fast, Expressive SE(n) Equivariant Networks through Weight-Sharing in Position-Orientation Space. Bekkers et al. ICLR 2024\n\n[30] Equivariance Discovery by Learned Parameter-Sharing. Raymond A. Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, Alexander Schwing. AISTATS 2022\n\n[31] Meta-Learning Symmetries by Reparameterization. Allan Zhou, Tom Knowles and Chelsea Finn. ICLR 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General response

We are grateful for the time, effort and invaluable feedback provided by the reviewers. We next address general points raised jointly amongst reviewers, and proceed to respond to specific comments in the individual author rebuttals below.

Firstly, we are glad that you **found the work interesting and elegant** (1myY: "an elegant solution to the problem”; “Overall, this is a really interesting idea”; 3Y5V: “The method appears to be very elegant in offering a natural relaxation of underlying weight-sharing scheme”). Furthermore, reviewers appreciated that the **theory was presented in a clear manner** (3Y5V: ”The paper is well-written”; xU9K: ”Nice background section. In my opinion, it is written much better than other related papers in the area.”). 

The reviewers also identify valuable points for improvement which we will gladly incorporate via minor edits in the camera-ready version of the paper. The main additions and experiments in response to reviewers’ concerns include:

1. We have added two experiments that show the methods effectiveness in the case of partial and misspecified symmetries, namely: MNIST with partial group structure (rotations sampled from a subset of $SO(2)$ (**Fig. 1a** in the attached PDF) and CIFAR10 with horizontal flips. 
2. We have added a comparison of models in the low-data regime for fully rotated and partially augmented MNIST (**Fig. 4** in the attached PDF). In the presence of partial symmetries (half of the rotation angles) and symmetry misspecification (flips) our weight sharing method can find useful weightsharing patterns.
3. Reviewers have pointed out a comparison to reference [30] and the benefits of employing double stochasticity over single stochasticity. We have attached results for this baseline on the rotatedMNIST dataset trained for 100 epochs. Visual inspection of the weight sharing schemes suggest the method is not as effective in picking up the group structure.
4. We have incorporated results for CIFAR-10 for a standard CNN model that matches the effective kernel size used in our GCNN/WSCNN models, namely 128 channels (calculated as $|G| * \text{channels} = 4*32 = 128$) (**Table 1** in the attached PDF). We have also added this model as a baseline in the flipped CIFAR-10 experiment (**Fig. 1b** in the attached PDF). Despite the kernel constraints in our model, it achieves performance comparable to that of the unconstrained 128-channel CNN, but with significantly less learnable parameters (our 2.1 M vs. the 6.5 M of the CNN).
5. We have added a computational scaling analysis of our weight sharing layer, comparing it to a group convolutional model of the same dimensions (**Fig. 2 and 3** in the attached PDF). We highlight that regular group convolutions can be implemented via weight-sharing schemes, resulting in equal computational demands for both approaches. Since the weight-sharing approach applies the group transformation in parallel across all elements (as a result of matrix multiplication and reshape operations), our method can prove quite efficient. Regarding memory allocation, group convolutions are often implemented using *for-loops* over the group actions, and this sequential method imposes a less heavy memory burden since the operation is applied in series per group element. However, although scaling is quadratic w.r.t. the number of group elements and the domain size for weight-sharing, we mitigate this issue by use of the typically lower-dimensional support of convolutional filters (i.e., $|G| <= 16$ and $\text{kernel size} <= 7$), rendering our approach practical for a wider range of settings.
6. We thank the reviewers (1myY, uLBU) for pointing out relevant related literature and will update our related works section accordingly with your suggested references. 

We hope that our additional experiments and clarifications were able to resolve any commonly raised points, and would be happy to respond to remaining concerns, if any, during the discussion period. 

We further address each reviewers' concerns and questions in our individual responses, and thank once again for the valuable feedback that we believe will increase the quality of our contributions.

**References**

[30] Equivariance Discovery by Learned Parameter-Sharing. Raymond A. Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, Alexander Schwing. Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:1527-1545, 2022.

[31] Meta-Learning Symmetries by Reparameterization. Allan Zhou, Tom Knowles and Chelsea Finn. ICLR 2021

Meta-review: This paper received mixed reviews. The reviewers generally acknowledge the technical contribution, considering the proposed method an interesting solution to an important problem. However, the negative evaluations focus primarily on the experiments: Reviewer 1myY noted that the current experiments are not the most effective for demonstrating the strength of the proposed method, and Reviewer xU9K raised concerns about the scalability and practicality of the approach. While these concerns are valid, I still believe the technical contribution of the paper meets the standard for acceptance.
Decision: Accept (poster)