Learning symmetries via weight-sharing with doubly stochastic tensors

Authors: putri a van der linden
alejandro  garcía-castellanos
sharvaree  vadgama
thijs p kuipers
erik j bekkers
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:56:52

Abstract: Group equivariance has emerged as a valuable inductive bias in deep learning, enhancing generalization, data efficiency, and robustness. Classically, group equivariant methods require the groups of interest to be known beforehand, which may not be realistic for real-world data. Additionally, baking in fixed group equivariance may impose overly restrictive constraints on model architecture. This highlights the need for methods that can dynamically discover and apply symmetries as soft constraints. For neural network architectures, equivariance is commonly achieved through group transformations of a canonical weight tensor, resulting in weight sharing over a given group $G$. In this work, we propose to *learn* such a weight-sharing scheme by defining a collection of learnable doubly stochastic matrices that act as soft permutation matrices on canonical weight tensors, which can take regular group representations as a special case. This yields learnable kernel transformations that are jointly optimized with downstream tasks. We show that when the dataset exhibits strong symmetries, the permutation matrices will converge to regular group representations and our weight-sharing networks effectively become regular group convolutions. Additionally, the flexibility of the method enables it to effectively pick up on partial symmetries.

Main paper: 
 introduction: 
 Equivariance has emerged as a beneficial inductive bias in deep learning, enhancing performance across a variety of tasks. By constraining the function space to adhere to specific symmetries, models not only generalize better but also achieve greater parameter efficiency [8, 9] . For instance, integrating group symmetry principles into generative models has enhanced sample generation and efficient learning of data distributions, particularly benefiting areas such as vision and molecular generation [11, 5] .The most well-known and transformative models in equivariant deep learning are convolutional neural networks (CNNs) [17] , which achieve translation equivariance by translating learnable kernels to every position in the input. This design ensures that the weights defining the kernels are shared across all translations, so that if the input is translated, the output features are correspondingly translated; in other words, equivariance is achieved through weight sharing. In the seminal work by Cohen and Welling [9] , this concept was extended to generalize weight-sharing under any discrete group of symmetries, resulting in the group-equivariant CNN (G-CNN). G-CNNs enable G-equivariance to a broader range of symmetries, such as rotation, reflection, and scale [9, 4, 32, 27] , thereby expanding the applicability of CNNs to more complex data transformations.However, the impact of G-CNNs is closely tied to the presence of specific inductive biases in the data. When exact symmetries, such as E(3) group symmetries in molecular point cloud data, are known to exist, G-CNNs excel. Yet, for many types of data, including natural images and sequence data, these exact symmetries are not present, leading to overly constrained models that can suffer in performance [30, 22, 2, 29] . In scenarios with limited data and for certain critical downstream tasks, having appropriate inductive biases becomes even more crucial.To avoid overly constraining models, symmetries must be chosen carefully to match those in the input data. This requires prior knowledge of these symmetries, which may not always be available. Furthermore, different symmetries at different scales can coexist, making manual determination of these symmetries highly impractical. To address this, multiple works have proposed partial or relaxed G-CNNs [22, 6, 2, 30] . Such models are initialized to be fully equivariant to some groups and learn from the data to partially break equivariance on a per-layer basis where necessary. However, these methods still require specifying which symmetries to include and can only achieve equivariance to subsets of these symmetries.In this work, we tackle the challenge of specifying group symmetries upfront by introducing a general weight-sharing scheme. Our method can represent G-CNNs as a special case but is not limited to exact equivariance constraints, offering greater flexibility in handling various symmetries in the data. Inspired by the idea that group equivariance for finite groups can be achieved through weight-sharing patterns on a set of base weights [21] , we propose learning the symmetries directly from the data on a per-layer basis, requiring no prior knowledge of the possible symmetries.We leverage the fact that regular group representations act as permutations and that the expectation of random variables defined over this set of permutations is a doubly stochastic matrix [7] . This implies that regular partial group transformation can be approximated by a stack of doubly stochastic matrices which essentially act as (soft) permutation matrices. Consequently, we learn a set of doubly stochastic matrices through the Sinkhorn operator [26] , resulting in weight-sharing under learnable group transformations.We summarize our contributions as follows:• We propose a novel weight-sharing scheme that can adapt to group actions when certain symmetry transformations are present in the data, enhancing model flexibility and performance. • We present empirical results on image benchmarks, demonstrating the effectiveness of our approach in learning relevant weight-sharing schemes when there are clear symmetries. • The proposed method outpaces models configured with known symmetries in environments where they are only partially present. Moreover, in the absence of predefined symmetries, it adeptly identifies effective weight-sharing patterns, matching the performance of fully flexible, non-weight-sharing models. • We provide analyses of the learned symmetries on some controlled toy settings. 
related work: 
 Partial or relaxed equivariance Methods such as [22, 6, 2] learn partial equivariance by learning distributions over transformations, and thereby aim to learn partial or relaxed equivariances from data by sampling some group elements more often than others. [30] tries to relax equivariance by introducing learnable equivariance-breaking components. [10, 29] Relax equivariance constraints by parameterizing layers as (linear) combinations of fully flexible, non-constrained components and constrained equivariant components. Finally, several works model soft invariances through learning the amount of data augmentation in the data or model relevant for a given task, either through learned distributions on the group or (automatic) hyperparameter selection [6, 12, 20] . However, these methods require pre-specified sets of symmetry transformations and/or group structure to be known beforehand. In contrast, we aim to pick up the relevant symmetry transformations during training.Symmetry discovery methods [24, 19] learn the group structure via (irreducible) group representations. [24] proposed to learn the Fourier transform of finite compact commutative groups and their corresponding bispectrum by learning to separate orbits on our dataset. This approach can be extended to non-commutative finite groups leveraging advanced unitary representation theory [19] . However, these methods are constrained to finite-dimensional groups and require specific orbit-predicting datasets. In contrast, our approach learns a relaxation of regular group representations-as opposed to irreducible representations. Moreover, our approach is not merely capable of learning symmetries, it subsequently utilizes them in a regular group-convolution-type architecture.Weight-sharing methods Previous studies have demonstrated that equivariance to finite groups can be achieved through weight-sharing schemes applied to model parameters. Notably, the works in [21] , [34] , and [33] provide foundational insights into this approach. In [34] , weight-sharing patterns are learned by using a matrix that operates on flattened canonical weight tensors, effectively inducing weight sharing. They additionally prove that for finite groups, there are weight-sharing matrices capable of implementing the corresponding group convolution. However, their approach requires learning these patterns through meta-learning and modeling the weight-sharing matrix as an unconstrained tensor. In contrast, our method learns weight sharing directly in conjunction with the downstream task and enforces the matrix to be doubly stochastic, thereby representing soft permutations by design. [33] presents an approach closely aligned with ours, where a weight-sharing scheme is learned that is characterized by row-stochastic entries. Their method involves both inner-and outer-loop optimization and demonstrates the ability to uncover relevant weight-sharing patterns in straightforward scenarios. However, their approach does not support joint optimization of the canonical weights and weight-sharing pattern, and they acknowledge difficulties in extending their method to higher input dimensionalities. Unlike [33] , we enforce both row and column stochasticity. Additionally, we can optimize for the sharing pattern and weight tensors jointly, and successfully apply our approach to more interesting data domains such as image processing.In [28] , group actions are integrated directly into the learning process of the downstream task. This method involves learning a set of generator matrices that operate via matrix multiplication on flattened input vectors. However, this approach constrains the operators to members of finite cyclic groups, which inherently limits their ability to represent more complex group structures. Furthermore, this restriction precludes the possibility of modeling partial equivariances, reducing the flexibility and applicability of the model to more diverse or complex scenarios.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
background: 
 We begin by revisiting group convolutional methods in the context of image processing, followed by their relation to weight-sharing schemes. We then proceed to briefly cover the Sinkhorn operator, which is the main mechanism through which we acquire weight-sharing schemes. Some familiarity with group theory is assumed, and essential concepts will be outlined in the following.Groups We are interested in (symmetry) groups, which are algebraic constructs that consist of a set G and a group product-which we denote as a juxtaposition-that satisfies certain axioms, such as the existence of an identity element e ∈ G such that for all g ∈ G we have eg = ge = g, closure such that for all g, h ∈ G we have gh ∈ G, the existence of an inverse g -1 for each g such that g -1 g = e, and associativity such that for all g, h, i ∈ G we have (gh)i = g(hi).In the context of geometric deep learning [8] , it is most useful to think of groups as transformation groups, and the group structure describes how transformations relate to each other. Specifically, group representations ρ : G → GL(V ) are concrete operators that transform elements in a vector space V in a way that adheres to the group structure (they are group homomorphisms). That is, to each group element g, we can associate a linear transformation ρ(g) ∈ GL(V ), with GL(V ) the set of linear invertible transformations on vector space V .Group convolution Concretely, such representations can be used to define group convolutions. Consider feature maps f : X → R D over some domain on which a group action is defined, i.e., over a G-space. E.g., for images (signals over X = R2 ) we could consider the group G = (R 2 , +) of translations, which acts on X via gx = x + y, with g = (y) a translation by y ∈ R 2 . While the group G merely defines how two transformations g, h ∈ G applied one after the other correspond to a net translation gh ∈ G, a representation ρ concretely describes how data is transformed. E.g., signals f : X → R can be transformed via the left-regular representation [ρ(g)f ](x) := f (g -1 x), which in the case of images and the translation group is given by [ρ(g)f ](x) = f (x -y).In general, group convolution is defined as transforming a base kernel under every possible group action, and for every transformation taking the inner product with the underlying data via G-convolution, inner product form:EQUATIONwith ⟨•, •⟩ denoting the inner product. For images, which are essentially functions over the group G = (R 2 , +), and taking ⟨k, f ⟩ := X k(x)f (x)dx the standard inner product, Eq. ( 1) boils down to the standard cross-correlation operator 2 :G-convolution, integral form:EQUATIONStandard (R 2 , +) Convolution:EQUATIONSemi-direct product groups When equivariance to larger symmetry groups is desired, e.g. in the case of G = SE(2) roto-translation equivariance for images with domain X = R 2 , a lifting convolution can be used to generate signals over the group G. In essence it is still of the form of (2), however integration is over X instead of over G:G-lifting Convolution:EQUATIONwith g = (x, R) ∈ (R 2 , +) ⋊ SO(2). The roto-translation group is an instance of a semi-direct product group (denoted with ⋊) between the translation and rotation group, which has the practical benefit that a stack of rotated kernels can be precomputed [9] , and the translation part efficiently be taken care of via optimized Conv2D operators. Namely via(k ⋆ f )(x, R i ) = Conv2D[k i , f ], with k i := k(R -1 i x). This trick can also be applied for full group convolutions (2).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
method: 
 Our objective is to uncover the underlying symmetries of datasets whose exact symmetries may not be known, in a manner that is both parameter-efficient and free from rigid group constraints.To achieve this, we re-examine regular representations and their critical role in generating various instantiations of the fundamental group convolution equation (1) . Moving from the continuous setting to concrete instantiations, we derive weight-sharing from a finite-dimensional vector of base weights by interpreting regular representations as permutations. We further analyze the characteristics of this weight-sharing approach, proposing the use of doubly stochastic matrices. This analysis forms the foundation for developing weight-sharing layers that adaptively learn dataset symmetries.In the current section, we first establish the connection between representations, weight-sharing and permutations. We then proceed to provide practical instantiations as ingredients for the proposed weight-sharing convolutional layers.Weight-sharing through learnable representations To achieve weight-sharing over a finite set of symmetries, we define learnable representations ρ : G → GL(V ). Specifically, we assign a learnable transformation (permutation matrix) to each element in G. It is important to note that we refer to G and ρ as a "group" and "representation" in a loose sense, as we relax the homomorphism property and do not initially endow G with a group product. Consequently, the collection of linear transformations does not form a group representation a priori. However, our proposed method is capable of modeling this structure in principle.Regular representations as permutations Consider the case of a continuous linear Lie group G, e.g. of rotations SO(2), and a real signal f : G → R over it. This signal is to be considered an infinite dimensional vector with continuous "indices" g ∈ G that index the vector elements f (g). To emphasize the resemblance of the regular representation to permutation matrices we write it as Regular representation in integral form:EQUATIONwith P g (i, h) = δ g -1 i (h) a kernel that for each g maps h to a new "index" i ∈ G, where the Dirac delta essentially codes for the group product asδ g -1 h (i) is non-zero only if g -1 i = h ⇔ g • h = i.The discrete counterpart of such an integral transform is matrix-vector multiplication with a matrix ρ(g) = P g with entries P ghi = 1 if gh = i and zero otherwise. As a concrete example, consider a signal f : G → R over the discrete group G = C 4 of cyclic permutations of size 4, i.e., a periodic signal of length four, then we could vectorize it as f → v with entries v g = f (g) and the regular representation becomes a simple cyclic permutation matrix withP 0 = 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 , P 1 = 0 0 0 1 1 0 0 0 0 1 0 0 0 0 1 0 , P 2 = 0 0 1 0 0 0 0 1 1 0 0 0 0 1 0 0 , P 3 = 0 1 0 0 0 0 1 0 0 0 0 1 1 0 0 0 .We refer to the collection of permutation matrices P ∈ [0, 1] |G|×|G|×|G| as the permutation tensor.Recall that the regular convolution operator ( 2) is not only defined for signals over groups G but for G-spaces X, in general, as in Eq. ( 4). It requires a representation that acts on signals (convolution kernels) over X. However, since the group G acts by automorphisms (bijections) on the space X, we can define the regular representation-as before-using permutation integral with P g (x, x ′ ) = δ g -1 x ′ that effectively sends old "indices" x ′ to their new location x, or concretely via permutation matrices P of shape |G| × |X| × |X|, where X is the domain of the signal that is transformed-which can also be G.In practice, it is common that we perform a discretization of continuous signals f , e.g., when we work with images, we discretize the continuous signal f : R 2 → R C to f : Z 2 → R C . Therefore, the group action over the discretized signal should approximate the group action over the continuous signal. We can see that in some cases, the group representation of the action on the discrete signal can still be implemented using permutation matrices. E.g., 90 • rotations (in C 4 ) applied to images merely permute the pixels. However, for finer discretizations, e.g. using 45 • rotations, interpolation can be used as a form of approximate permutations [16, Fig. 2 ].Weight sharing In a discrete group setting, we then see that convolution (1) is obtained by multiplications with matrices obtained by stacking permuted base kernel weights w ∈ R |X| Discrete G-convolution:EQUATIONE.g., a group convolution over C 4 is implemented with a matrix of the form W = w0 w1 w2 w3 w3 w0 w1 w2 w2 w3 w0 w1 w1 w2 w3 w4.Regular representations allow for element-wise activations An important practical element of using regular representations to define group convolutions is that permutations commute with element-wise activations, namely,[ρ(g)σ(f )](i) = σ(f )(g -1 i) = σ(f (g -1 i)) = σ([ρ(g)f ](i)).In contrast, steerable methods-based on irreducible representations (cf. App. A.1)-require specialized activation functions so as not to break group equivariance. Such activations in practice are not as effective as the classic element-wise activations such as ReLU [31] : they may introduce discretization artifacts that disrupt exact equivariance, ultimately constraining the method's expressivity [5] . Hence, when learning weight-sharing schemes-as is our objective-it is preferred to achieve weight-sharing using regular representations without the risk of breaking equivariance by using standard activation functions.Figure 1 : Kernel stacks are acquired through a learned weight-sharing scheme applied to a set of flattened base kernels.Having established the link between regular representations and soft permutations, we now motivate the use of doubly stochastic matrices as natural candidates for their implementation. Specifically, we utilize the fact that the expected value of random variables over this set of permutations yields a doubly stochastic matrix [7] .Doubly stochastic matrices Let S ∞ (S n ) denote respectively the system of infinite (n × n) doubly stochastic matrices, i.e. matrices S ≡ {s ij ∈ [0, 1] : i, j = 1, 2, ...(, n)} such that j s ij = 1, and i s ij = 1.Let P ∞ (P n ) denote respectively the system of infinite (n × n) permutation matrices, i.e. matrices P ≡ {p ij ∈ {0, 1} : i, j = 1, 2, ...(, n)} such that j p ij = 1, and i p ij = 1. Note that for any permutation tensor P ∈ R |G|×|X|×|X| , where X is the domain of the signal that is transformed by the group G, then P g ∈ P |X| for every g ∈ G.Then, by Birkhoff's Theorem [7] , and its extension to infinite dimensional matrices, commonly called Birkhoff's Problem 111 [13, 23, 14, 3] , we have that any convex combination of permutation matrices will be equal to a doubly stochastic matrix, i.e, P ∈Pn λ(P )P = S ∈ S n , withP ∈Pn λ(P ) = 1 (∀n ∈ N ∪ {+∞})where λ(P ) gives a probability measure supported on a finite subset of the set of permutation matrices P. Therefore we may state that Using doubly stochastic matrices, we can model approximate equivariance as defined in [22] .I.e., let S be a random variable over {P g ∈ P |X| | g ∈ G} with a finitely supported probability measure P[S = P g ] = λ(P g ) for every g ∈ G, then S = E[S] = g∈G P[S = P g ]P g is a doubly stochastic matrix. We want to note that S can be seen as a generalization of the convolution matrix presented in [18] .The Sinkhorn operator [26, 1] transforms an arbitrary matrix to a doubly stochastic one through iterative row and column normalization, provided that the number of iterations is large enough. That is, initialize a tensor X ∈ R N ×N , then it will converge to a doubly stochastic tensor via the following algorithm:EQUATIONwith T c and T r the normalization operators over the rows and columns, respectively, defined asT c = X ⊘ 1 N 1 T N X sumc(X) and T r = X ⊘ X 1 N 1 T N sumr(X), where ⊘ denotes elementwise division, sum c (•), sum r (•) perform column-wise and row-wise summation, respectively. We let Θ l i ∈ R |X|×|X| be a collection of learnable parameters that parametrize the representation stack of the layer l as R l = (S K (Θ l 0 ), S K (Θ l 1 ), ..., S K (Θ l N )) T ∈ [0, 1] |G|×|X|×|X| . i.e., we parameterize this tensor as a stack of |G| approximate doubly stochastic |X|-dimensional matrices, wherein stochasticity is enforced via K applications of the Sinkhorn operator. We also define a set of learnable base weights θ l ∈ R |X|×Cout×Cin . The WSCNN layer is then simply given by ( 7) with P and w respectively replaced by R l and θ l .We further note that on image data |X| can be large, making the discrete matrix form implementation computationally demanding. Hence, we consider semi-direct product group parametrizations for G, in which we let G be of the form (R n , +) ⋊ H, with H a learnable (approximate) group. Then, the representation stacks will merely be of shape |H| × |X ′ | × |X ′ |, 
experiments: 
 We first demonstrate that the proposed weight sharing method can effectively pick up on useful weight sharing patterns when trained on image datasets with different equivariance priors. We then proceed to show the method can effectively handle settings where partial symmetries are present in the data, and further analyze the learned weight-sharing structures on a suite of toy datasets. Model architectures, regularizers (norm, ent) and design choices can be found in Appendix C.4 and C.1, respectively. An analysis of computational requirements can be found in Appendix C.4We assess the efficacy of our proposed weight-sharing scheme in recognizing data symmetries through experiments on datasets subjected to various data augmentations. Specifically, we evaluate our model on MNIST images that have been rotated (with full SO(2) rotations) and scaled (with scaling factors between [0.3, 1.0]). We categorize these datasets based on their data symmetry characteristics: MNIST with rotation and scaling as datasets with known symmetries, and CIFAR-10 with flips as a dataset with unknown symmetries.For RotatedMNIST, we regard a C4-group convolutional model as a benchmark since it has been equipped with a subgroup of the underlying data symmetries a priori. Additionally, we contrast our results with a non-constrained CNN model which has double the number of channels, allowing for free optimization without symmetry constraints. As such, our evaluations are benchmarked against two distinct models: 1) a group convolutional model that is equivariant to discrete rotations, embodying fixed equivariance constraints, and 2) a standard CNN that adheres only to translational equivariance, without additional constraints. This experimental setup positions the standard CNN as the most flexible model lacking predefined inductive biases. In contrast, the group convolutional neural network (GCNN) is characterized by fixed weight sharing, while our proposed weight-sharing CNN (WSCNN) introduces a semi-flexible, learnable weight-sharing mechanism. Note that we explicitly distinguish between the number of free model and kernel parameters and the additional parameters introduced by our weight sharing scheme throughout results (marked by +).Results can be found in Tab. 1. When there is a clear misalignment between the model and data symmetries, the constraints imposed by the model hinder performance, as demonstrated by the C4-GCNN on the scaled MNIST dataset. Notably, our proposed method consistently achieves high performance across all datasets without requiring fixed group specifications. Furthermore, visual inspection of the learned kernels indicates that WSCNN adapts to the underlying data symmetries by effectively rotating kernels, as shown in Figure 2 . Additionally, analysis of the learned representation stack reveals that it closely resembles elements of C 4 permutations, further demonstrating the model's capability to internalize and replicate data transformations (see Figure 3 and Appendix B.3).Additionally, we test the model on CIFAR-10, representing a dataset with possibly more complex/unknown symmetry structures, and CIFAR-10 with horizontal flips (which cannot be represented by C N transformations). Results can be found in Tab. 2, with detailed training and model specifications available in Appendix C.4. We compare against the (possibly misspecified) C4-GCNN, and several unconstrained CNNs models with varying number of channels: 1) CNN-32 matched in free kernel size, 2) CNN-64 matched in parameter budget, and 3) CNN-128 matched in effective kernel size (calculated as |G| × channels = 4 × 32 = 128). Despite the kernel constraints in WSCNN, it achieves performance comparable to that of the unconstrained 128-channel CNN (within < 1% accuracy), at a significantly smaller parameter budget (2.1 M vs. 6.5 M).We show our method is able to pick up on partial symmetries by testing it on MNIST with rotations sampled from a subset of SO (2) and compare it to the C4-GCNN. Additionally, we show results on CIFAR-10 with horizontal flips, which is a commonly used train augmentation. Results can be found in Tab. 3 and Tab. 4. Additionally, We proceed to test the model's capability to detect data symmetries by applying it to a suite of toy problems, wherein the datasets comprise noisy G-transformed samples. Details on the data generation framework are provided in Appendix B. Our testing employs a single-layer setup aimed at learning a collection of kernels that ideally match each data sample, considering inherent noise. This involves training the model to identify a set of base kernels and their pertinent transformations, effectively adapting to the variations presented by the toy problems. 
discussion and future work: 
 We demonstrated a method that can effectively identifies underlying symmetries in data, even without strict group constraints. Our approach is uniquely capable of learning both partial and approximate symmetries, which more closely mirrors the complexity found in real-world datasets. Utilizing doubly stochastic matrices to adapt kernel weights for convolutions, our method offers a flexible means of learning representation stacks, accommodating both known and unknown structures within the data. This adaptability makes it possible to detect useful patterns, although these may not always be interpretable in traditional group-theoretic terms due to the absence of predefined structures in the representation stack.Limitations include computational requirements, which scale quadratically with the size of the group and the kernel size, posing challenges in scenarios with large groups or high-resolution data. As such, in this work we have designed the representation stack to be uniform across the channel dimension. However, this prevents learning of other commonly used image transformations such as color jitter. Furthermore, the need for task-specific regularization to manage entropy scaling during the learning of representations introduces complexity in hyperparameter tuning, which can be a barrier in some applications. Additionally, we observed that representations in later layers may show minimal diversity, suggesting that further innovation in regularization strategies might be necessary to enhance the distinctiveness of learned features across different layers.For future work, we aim to enhance our method by implementing hierarchical weight-sharing across layers and promoting group equivariance more systematically. One promising direction is to leverage the concept of a Cayley tensor, akin to [19] , to identify and reuse learned group structures across different layers of the network. This approach would not only impose a more unified and coherent group structure within the model but also potentially reduce the computational overhead associated with learning separate representations for each layer. By encouraging a shared group structure throughout the network, we anticipate improvements in both performance and interpretability, paving the way for more robust and efficient symmetry-aware learning systems.In this section, we closely follow the mathematical preliminaries outlined in [31] . For a comprehensive reference on Representation Theory, see [25] .Equivalent representations Two representations ρ and ρ ′ of a group G are considered equivalent if there exists a similarity transform such that:∀g ∈ G ρ(g) = Qρ ′ (g)Q -1where Q represents a change of basis matrix.Irreps A matrix representation is called reducible if it can be decomposed as:ρ(g) = Q -1 (ρ 1 (g) ⊕ ρ 2 (g))Q -1 = Q ρ 1 (g) 0 0 ρ 2 (g) Q -1where Q is a change of basis matrix. If the sub-representations ρ 1 and ρ 2 cannot be further decomposed, they are termed irreducible representations (irreps). The set of all irreducible representations of a group G is denoted as Ĝ.Additionally, any representation ρ : G → GL(V ) of a compact group G can be expressed as:ρ(G) = Q   j∈I ρ j   Q -1where I is an index set (possibly with repetitions) over Ĝ.Similarly to what is showed in Section 4, a representation ρ : G → R d×d can be viewed as a collection of d 2 functions over G. The Peter-Weyl theorem asserts that the collection of functions formed by the matrix entries of all irreps in Ĝ spans the space of all square-integrable functions over G. For most groups, these entries form an orthogonal basis, allowing any function f : G → R to be written as:f (g) = ρj ∈ Ĝ m,n<dj w j,m,n • d j [ρ j (g)] mnwhere d j is the dimension of the irrep ρ j , while m, n index the entries of ρ j . Note that this expression corresponds to the inverse Fourier transform and that the coefficients w j,m,n can be obtained by the Fourier transform of f with respect to the basis functions {[ρ j (g)] mn } j∈I .Connection with regular representations It can be shown that the regular representations can be decomposed using the corresponding irreps as follows:ρ reg (g) = Q -1   pj dj ρ j   Qwhere Q performs the Fourier transform, while Q -1 performs the inverse Fourier transform. This implies that when functions f : G → R are considered as vectors in R |G| , with a basis where each axis corresponds to a group element, then, as we have seen in Section 4, the group action results in a permutation of these axes. However, applying the Fourier transform changes the basis so that G acts independently on different subsets of the axes, resulting in the action being represented by a block-diagonal matrix, which is the direct sum of irreps.For the construction of the toy problems, we look at two types of data-generating processes:Equivariant data. Assume a canonical vector x and corresponding label ŷ. We assume these vectors transform under a known group G, such that the data-generating process is as follows:Sample group action: g ∼ µ(G), Apply group action to feature with noise: x = ρ x (g)x + ϵApply group action to label: y = ρ y (g)ŷ with ρ x , ρ y the representations of G acting on the feature and label space, respectively. Evaluation metrics To assess whether our method effectively captures the underlying data symmetries, we analyze the learned weight-sharing structure by comparing it to known ground truth patterns. Since our model does not impose associativity or any strict group constraints on the representation stack, it may learn to represent mixtures or interpolations of group elements. Hence, in general, we will not observe a relevant algebraic structure if we produce a Cayley table based on the learned representations as done in [24, 19] .Therefore, we will use an approach that allows us to capture the flexibility of our representations.To this end, we will examine each representation matrix to determine how closely it resembles a convolution matrix [18] associated with a random variable defined over some specific group G. We employ the set of group actions from G, represented as doubly stochastic tensors {P gt k } |G| k=1 , as a reference framework to quantify the fit and alignment of our model's representations with these predefined group actions. As such, for each learned weight sharing tensor R l i , we calculate the fit Pi = We conducted experiments on various signals subjected to different transformations, including: a 1D signal with cyclic shifts (exemplary samples shown in Fig. 4a ), a 2D signal with C 8 rotations (illustrated in Fig. 7 ), and a 3D voxel grid enhanced by 24 cube symmetries. In each scenario, the learned kernel stack accurately matched the data samples, achieving perfect accuracy. displays the learned representations for localized shifts in the 1D signal, while Figure 8 presents the learned kernel stack for the 2D signal dataset.Figure 6Furthermore, to assess whether our model can identify partial group structures, we evaluate it using two distinct datasets: a 1D signal enhanced with cyclical shifts, and a 3 × 3 × 3 voxel grid subjected to rotations from C4 × C4. For the shift dataset, we utilize the complete set of cyclical shifts as the ground truth representations. In the current section, we outline some design choices used across all experiments. Code is available at https://github.com/computri/learnable-weight-sharing. Firstly, when defining the learnable representation stack R, we have found that anchoring an identity element aids in distinguishing the base kernel from its potential transformations. This approach is based on the intuition of starting with a learnable base kernel and subsequently learning its transformations and we have found that it aids optimization.We test two regularizers on the representations R, which are similar to those used by [33] :• Entropy Regularizer: The primary motivation for using the entropy regularizer is to encourage sparsity in our weight-sharing schemes, which helps the matrices approximate actual permutation matrices rather than simply being doubly stochastic. This approach stems from the intuition for some transformations, the weight-sharing schemes should mimic soft-permutation matrices. The effectiveness of this sparsity depends on the specific group transformations relevant to the task-for example, C4 rotations are typically represented by exact permutation matrices. In contrast, C N rotations or scale transformations might require interpolation, thus aligning more closely with soft permutation matrices. Our experimental results indicate that the utility of this regularizer varies with the underlying transformations in the data; for instance, it is not beneficial for scale transformations in the MNIST dataset, as anticipated. The entropy regularizer is of the following form:ent(R) = - N,D,D ijk R ijk • log(R ijk )• Normalization Regularizer: Empirically, we have found the normalization regularizer essential for reducing the number of iterations needed by the Sinkhorn operator to ensure the matrices are row and column-normalized. Without this regularizer, the tensors either fail to achieve double stochasticity or require an excessively high number of Sinkhorn iterations to do so. The normalization regularizer is of the following form:norm(R) = 1 D D i sum r (R) 2 i + sum c (R) 2 iWhere sum r , sum c are defined as in 4.2.Model architecture For all MNIST experiments, a simple 5-block CNN was used. Each block uses a kernel size of 5 and is succeeded by instance norm and ReLU activation, respectively. After the final convolution block, any spatial and group dimensions are reduced through a global average pooling operation, and a single linear layer is used as a classification head. For the group convolutional model and our weight sharing model, the default hidden channel dimension in the blocks was set to 32 unless otherwise stated, and 64 in the regular CNN models.The models used a learning rate of 1e-2 and were trained for 100 epochs. All the experiments were done on a single GPU with 24GB memory under six hours.Model architecture We used the ResNet architecture as in [15] Appendix B.1, except that we swapped the final global max pooling operator with a global mean pooling. However, in contrast to [15] , we use regular discrete kernels instead of continuous kernel parameterizations.Training details Following [15] , we trained the models for 200 epochs using a learning rate of 1e-4. All the experiments were done on a single GPU with 24GB memory under six hours.Fig. 13 14 show the computational scaling analysis of our weight sharing layer, comparing it to a group convolutional model of the same dimensions. We highlight that regular group convolutions can be implemented via weight-sharing schemes, resulting in equal computational demands for both approaches. Since the weight-sharing approach applies the group transformation in parallel across all elements (as a result of matrix multiplication and reshape operations), our method can prove quite efficient. Regarding memory allocation, group convolutions are often implemented using for-loops over the group actions, and this sequential method imposes a less heavy memory burden since the operation is applied in series per group element. However, although scaling is quadratic w.r.t. the number of group elements and the domain size for weight-sharing, we mitigate this issue by use of the typically lower-dimensional support of convolutional filters (i.e., and ), rendering our approach practical for a wider range of settings. Justification:Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: We do not use proprietary datasets or code Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Due to the equivalence between convolution and correlation via kernel reflection, we henceforth simply refer to operators of the type of (2) as convolution even though technically they are cross-correlations. 
 
 Appendix:  


Decision: This paper received mixed reviews. The reviewers generally acknowledge the technical contribution, considering the proposed method an interesting solution to an important problem. However, the negative evaluations focus primarily on the experiments: Reviewer 1myY noted that the current experiments are not the most effective for demonstrating the strength of the proposed method, and Reviewer xU9K raised concerns about the scalability and practicality of the approach. While these concerns are valid, I still believe the technical contribution of the paper meets the standard for acceptance.

Reviews: review: paper_summary: paper_summary: This paper introduces a parameterisation that contains the ability to represent weight tying corresponding to arbitrary (?) group equivariances. In practice it can represent interpolations of weight tying, but it is argued that this is a feature (not a bug!) since strict equivariance is often too strong a constraint to place on a model for it to still fit the training data.

Strict equivariance is known to be represented by a permutation matrix (correct me if I'm wrong), which justifies the need to use doubly stochastic matrices to interpolate. This makes the edges of the parametisation strict equivariances. (For equivariances of discretised continuous signals this is not quite true, as the permutation in continuous space would need to be approximated by some interpolation in the discrete space)

This leads to a new parameterisation of a weight structure in a layer that can be trained in the usual way. The doubly-stochastic matrices can then be investigated to see if equivariance is actually learned.

The experiments implement the method, and run on benchmark datasets and synthetic datasets, showing good performance, and somewhat interpretable group structure appearing. It is unclear what the actual point of the experiments is, since there are many reasons to use equivariance, but the experiments are not phrased in terms of this (see discussion).
strength_weakness: strength_weakness: Overall, the paper presents a well-reasoned method to an important problem, and I do believe that it meets the standard for publication at NeurIPS.

**Method & Presentation**
The method is well-justified. However, a final summary of what a forward pass through a layer looks like was not given, and would be really helpful. In addition, it would be helpful to have a clearer discussion of how many additional parameters are added (beyond lines 220 onwards), with the architectures that are discussed given as an explicit example.

Essentially, one thing which seems to be the case, but is not explicitly acknowledged, is that this method collapses to just a special parameterisation of weight matrices, where the weights have low-rank combined with doubly-stochastic structure. The low-rank-ness is shown in eq 6. While this is a simplistic way of looking at the method, it does give a helpful alternative view. Making this explicit would help the paper.

**What is the claim of the experiments?**
The experiments are the main weakness of the paper. Some qualitative results about the structure of the learned weights are given, which are helpful. But it is not clear what the quantitative claim of the experiments is. Equivariance can help in several ways, e.g. better out-of-distribution prediction, better prediction at low data, or smaller/compacter models. So is the claim that the equivariance inductive bias helps, and it can be discovered automatically? But in this case this is not disentangled from the model capacity. Perhaps a normal CNN would perform better if it were just made larger! This is additionally indicated that the baseline 70% accuracy on CIFAR 10 is low compared to what other non-group-equivariant methods can achieve. This unclarity also exists in the synthetic experiments, where the size of the dataset is not discussed. Low-data experiments could help here, since it's easier to make the model large enough that size doesn't help any longer, which isolates inductive bias only.

The CIFAR experiments show that making the model larger improves performance. How can we be sure that this is really the benefit of learning equivariances, rather than adding more capacity? Another experiment that is necessary here, is a comparison to a weight structure that does not have the doubly-stochastic constraint enforced on it. This would allow the effect of simply adding capacity to be tested.

Alternatively, a low-data experiment would allow the generalisation capabilities of the model to be tested (e.g. [Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations](fig 3 in https://proceedings.neurips.cc/paper_files/paper/2022/file/50d005f92a6c5c9646db4b761da676ba-Paper-Conference.pdf)). This could be done on MNIST variants, where currently the differences are so small, it is hard to draw conclusions.

This same issue pops up in the synthetic experiments. What is the dataset size? If the dataset is so large that all transformed signals are in the dataset, even a fully-connected network would learn the correct function. A low-data experiment is needed to really show that truly an equivariance has been learned that can help to _generalise_. Alternatively, you need to argue a benefit on the basis of parameter count.

In summary: The claims that the experiment section support are not clear. The field of equivariances is mature enough that the potential benefits have been clearly described, and these need to be clearly evaluated in experimental sections.

**Related Work**
The idea of relaxing equivariance by placing a distribution over transformations is older than the papers currently cited. E.g.:
- [Local Group Invariant Representations via Orbit Embeddings](http://proceedings.mlr.press/v54/raj17a/raj17a.pdf)

The discussion of "symmetry discovery methods" is unclear. What data do these methods need, or what kind of training signal do they use? What kind of predictive improvements do they obtain? Is the goal of these papers the same as those in the previous paragraph? Or, is the way that these methods learn group structure different from those in the previous section? If so, how?

Methods that learn a degree of equivariance on a layer-by-layer basis are relatively new, and it would be good to discuss this explicitly. E.g.:
- [Residual Pathway Priors for Soft Equivariance Constraints](https://proceedings.neurips.cc/paper/2021/file/fc394e9935fbd62c8aedc372464e1965-Paper.pdf) Finzi et al is not mentioned at all, but does allow partial equivariance in a layer-by-layer way.
- Reference [26] "Learning layer-wise equivariances automatically using gradients" is cited, but only in the context of arguing that overly constrained models suffer poor performance, but not in the context that this paper also discusses how to learn the right equivariance to use. This author has more papers on learning invariance/equivariances that may be relevant.

Overall, the literature review misses a lot of relevant work. The suggestions I gave are off the top of my head and I definitely missed some important papers too. However, it is the responsibility of the authors to put the time and effort into going beyond this to give a more thorough overview.

**Minor**
- _"Requiring no prior knowledge of the possible symmetries."_ (line 52) It is true that earlier methods (with the exception of [31]) could only pick between groups that were completely specified a-priori. While this paper _in principle_ does provide a parameterisation that can _search_ over a much wider space, this space needs to be limited for scalability reasons, and it was not demonstrated that the method would work reliably without this "prior" being added!
- It would be really helpful to have a full discussion of the impact on the number of parameters, for these specific experiments.Needs a discussion of the total number of parameters
- Is there a typo in params in table 1? Under "Params" should "103 + 265K" be "103K + 256K"?
questions: questions: - Can this parameterisation represent arbitrary group equivariances? Is the representation of every strict equivariance a permutation matrix? It would be helpful to be explicit about how general this really is.
- Am I right in understanding that this method ultimately just parameterises a low-rank weight matrix over different feature channels?
- Can the benefit that equivariance promises in the low-data regime still be provided by this method when the invariance is learned? Since effectively, you're just parameterising weights in a different way (low rank?). In rotationally equivariant settings in low-data, could these weights not just overfit, rather than learning to rotate filters? Would this not lose an important benefit of equivariance?
- How important is the doubly-stochastic nature of things? Could you just run an experiment without the Sinkhorn component at all?
- Can you give a very short (ideally 1 sentence, or a 2-3 sentences) summary of the quantitative claims that are made about the method, that are verified in the experiment section?
limitations: limitations: See above.

Overall, this is a really interesting idea. I do have some concerns about the evaluation. These concerns are large in the scheme of determining how well this method really works relative to clearly formulated claims, but small relative to typical approaches in the ML community.
paper_summary: paper_summary: The paper proposes a symmetry discovery through learning parameter-sharing in weight matrices. The parameterization relies on relaxing underlying permutation matrices by transforming them as doubly stochastic matrices. In combination with additional regularization, the parameterization can be used to successfully discovery symmetries in data, improving generalization performance.
strength_weakness: strength_weakness: -Regularization.
The paper proposes a novel parameterization that can be used for symmetry discovery. In terms of objective, the work relies on direct regularization to avoid equivariants solutions, similar to some other symmetry discovery works. It has been shown in prior work that this strategy can have issues, since it introduces an additional hyperparameter which may need additional tuning (thereby is less ‘automatic’). In terms of explaining the methodology, the paper would benefit from some discussion on the role of the used regularizer.

-Scalability.
The paper seems to scale quadratically in |X|. Is this not an issue? How does this compare against scaling of alternative methods?
questions: questions: -Regularization
For experiments, it would be helpful if it is clear how the strength of this regularizer is chosen. Cross-validation?
-Analysis of learned symmetries
It would be interesting to better understand what weight-sharing is being learned for CIFAR-10, apart from merely measuring improved test accuracy. Is there an analysis on the learned symmetry?
-Regularization
Entropy regularization seems to hurt on CIFAR-10, but improve on MNIST experiments. Do authors have an intuition on why this is the case?
limitations: limitations: The method proposes an elegant and novel symmetry discovery method. I expect the proposed parameterization to be beneficial to the community. The paper could improve a bit on the objective function side of things (seems to rely on directly encouraging symmetry, like some prior work). Apart from the remaining questions, the paper makes for a strong contribution.
paper_summary: paper_summary: Overview:
This paper claims to build upon a long line prior work on symmetry detection and learning equivariant kernels. [18,19,20,22, 25-31. Eespecially reference [30] which is an identical weight and parameter sharing scheme that learns  and discovers equivariances  solely by row-stochastic entries.  This paper's idea is to enforce both row and column stochasticity through  use of the Sinkhorn operator  to achieve equivariance discovery and applicable to  more interesting data domains such as images.

Advantages:
* The paper addresses an important problem of  Kernel equivariance via symmetry detection and weight  in CNN . However its marginal progress over prior work. 

Weaknesses:
* Comparisons with prior work are missing.
* The quantitative advantage of  double stochasticity from row and column has advantages for metric spaces (images),. However this is not explained nor quantified through comparison.
* Moreover while ground truth tests are done for some toy problems, the quantitative generalizability of the double stochasticity over single stochasticity are never delineated, nor proved.

Questions:
This paper is identical to  this ICML 2024 workshop paper.   https://scholar.google.com/citations?view_op=view_citation&hl=en&user=gfRkDXEAAAAJ&citation_for_view=gfRkDXEAAAAJ:ufrVoPGSRksC ?
Also why no comparisons to even reference [30] ?

Missing References:
* Several references are missing, including the entire suite of probabilistic symmetry detection and equivariant NN is not discussed. See for e.g.   @misc{bloemreddy2020probabilisticsymmetriesinvariantneural,
      title={Probabilistic symmetries and invariant neural networks}, 
      author={Benjamin Bloem-Reddy and Yee Whye Teh},
      year={2020},
      eprint={1901.06082},
      archivePrefix={arXiv},
      primaryClass={stat.ML},
      url={https://arxiv.org/abs/1901.06082}, 
}
strength_weakness: strength_weakness: Weaknesses:
* Comparisons with prior work are missing.
* The quantitative advantage of  double stochasticity from row and column has advantages for metric spaces (images),. However this is not explained nor quantified through comparison.
* Moreover while ground truth tests are done for some toy problems, the quantitative generalizability of the double stochasticity over single stochasticity are never delineated, nor proved.

Questions:
This paper is identical to  this ICML 2024 workshop paper.   https://scholar.google.com/citations?view_op=view_citation&hl=en&user=gfRkDXEAAAAJ&citation_for_view=gfRkDXEAAAAJ:ufrVoPGSRksC ?
Also why no comparisons to even reference [30] ?

Missing References:
* Several references are missing, including the entire suite of probabilistic symmetry detection and equivariant NN is not discussed. See for e.g.   @misc{bloemreddy2020probabilisticsymmetriesinvariantneural,
      title={Probabilistic symmetries and invariant neural networks}, 
      author={Benjamin Bloem-Reddy and Yee Whye Teh},
      year={2020},
      eprint={1901.06082},
      archivePrefix={arXiv},
      primaryClass={stat.ML},
      url={https://arxiv.org/abs/1901.06082}, 
}
questions: questions: Questions:
This paper is identical to  this ICML 2024 workshop paper.   https://scholar.google.com/citations?view_op=view_citation&hl=en&user=gfRkDXEAAAAJ&citation_for_view=gfRkDXEAAAAJ:ufrVoPGSRksC ? Is this ok ??

Also why no comparisons to even reference [30] ?

Missing References:
* Several references are missing, including the entire suite of probabilistic symmetry detection and equivariant NN is not discussed. See for e.g.   @misc{bloemreddy2020probabilisticsymmetriesinvariantneural,
      title={Probabilistic symmetries and invariant neural networks}, 
      author={Benjamin Bloem-Reddy and Yee Whye Teh},
      year={2020},
      eprint={1901.06082},
      archivePrefix={arXiv},
      primaryClass={stat.ML},
      url={https://arxiv.org/abs/1901.06082}, 
}
limitations: limitations: There are no attempts on analyzing the generalizability of the operator to varying initial and boundary conditions or Reynold numbers.

Other potentially relevant paper to consider:
https://www.sciencedirect.com/science/article/pii/S0021999123001997
This paper uses error-correction in neural operators based on the residual.
paper_summary: paper_summary: In contrast to many works that impose strict architectural constraints to parameterize neural networks that are exactly equivariant to a known group, this work considers learning approximate equivariances to unknown groups. This is done using soft weight sharing schemes, where doubly stochastic tensors are learned and applied on canonical weights. The approach generalizes GCNNs, and is shown to learn natural symmetries in experiments.
strength_weakness: strength_weakness: 1. Section on "Regular representaitons allow for element-wise activations" is a little out of place and imcompletely justified. The claim that non-elementwise activations "in practice are not as effective as the classic element-wise activations" is a strong statement that needs more specific justification.
2. Direct parameterization of the kernels for each group element is expensive. Empirical runtime and memory analysis would be appreciated, to see the effect of this.
3. Empirical results are rather limited and weak. Few baselines are considered (see below as well), the baselines already perform well on these tasks, and WSCNN does not improve over the baselines in Table 1. Given that the method does not appear to be too scalable (see above), it is unclear where it would be useful in improving performance.
4. Other baselines (besides standard CNNs and GCNNs) and ablations are missing. For instance, one can imagine removing the sinkhorn projection onto the doubly stochastic matrices. If we initialize the $\Theta_i^l$ to be permutation matrices (your fixing of one representation to be identity is related to this), then this may also perform similarly.
questions: questions: 1. How do the learned filters of WSCNN look on CIFAR10?
2. Do learned representations ever look the same across different "group" elements?
limitations: limitations: Some discussion in section 6

Comments: title: Reviewer-Author discussion period
comment: > We acknowledge that conventional CNNs with an equivalent channel count inherently possess a higher expressive capacity due to a lack of kernel constraints. Therefore, we have opted to compare overall parameter counts rather than the number of kernels, a more comprehensive assessment in our opinion. For completeness, we have additionally measured the performance for an unconstrained 128-channel CNN (please consider point 4 in the general rebuttal).

This is a really helpful addition. Interesting, so the main claim now is that for a parameter-constrained model, you obtain better performance? Size of the model clearly does come into this, as the CNN also gets better as more filters are added. This is fine, and a clear claim. However this does mean that the current experiments only show that the invariance learning properties work in the _underfitting_ regime! This is different from what is wanted from learning equivariances, where we may want to show the ability to obtain improved performance, once other simpler methods like making the model larger stop working.

This is why other methods e.g. [31, 26] consider second order information (meta-learning, marginal likelihood approximations): to distinguish between inductive biases even when the training loss cannot. This was also discussed in [*] and back in 2018 [**]. This is what the low data experiment could have shown, if it was verified that the model was large enough to sufficiently fit the training data. Also the low-data experiment is of limited usefulness, since it does not contain a baseline of a non-invariant model. The interesting question here is to see how much of an improvement you can get by learning invariances over a model that cannot do this. (Also relevant to your answer to my question 4.)

> We would like to stress that we do not explicitly impose any group structure in our representation stack

This was clear from the paper, and is certainly valuable! However, I don't see how this makes the literature I pointed to any less relevant?


## Overall
Either way, while I don't think this is the end of the question of how to learn equivariance automatically, I do believe it is an interesting paper, and I will continue to argue for acceptance.

[*] [Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations](https://proceedings.neurips.cc/paper_files/paper/2022/file/50d005f92a6c5c9646db4b761da676ba-Paper-Conference.pdf) 2022 (not cited in the paper)

[**] [Learning Invariances using the Marginal Likelihood](https://arxiv.org/abs/1808.05563) 2018 (not cited in the paper)
We thank you for your response. We are glad you appreciated the work and will incorporate the necessary alterations.
Thank you for your continued engagement and feedback on our submission. We appreciate your concerns and take them seriously as they help us refine our approach and clarify our contributions.

We understand that the empirical practice has evolved since the introduction of GCNNs, among other models with known/built-in equivariance structures. However, the field of automatic (partial) symmetry discovery is fairly new, and the ability to detect and utilize equivariances automatically and the state of current research (i.e. [1, 2, 3, 4, 5]) operates in controlled and well-understood environments in order to validate the fundamental aspects of the proposed method.

While we recognize the desire for experiments in application areas that could directly benefit from equivariance, our current focus is on establishing a strong theoretical and empirical foundation for our method. Such foundational work is essential for understanding the potential and limitations of new techniques before they can be successfully applied to more demanding domains.

Your point on scalability and applicability in more practical scenarios is well-taken. We see this as an important direction for future research, where the scalability of our method can be further developed and tested in more complex environments. We plan to explore this in future work. 

**References**

[1] Learning Equivariances and Partial Equivariances from Data. David W Romero, Suhas Lohit. NeurIPS 2022

[2] Learning Invariances in Neural Networks. Gregory Benton, Marc Finzi, Pavel Izmailov, Andrew Gordon Wilson. NeurIPS 2020 

[3] Learning Layer-wise Equivariances Automatically using Gradients. Tycho F. A. van der Ouderaa, Alexander Immer, Mark van der Wilk. 2023

[4] Relaxing Equivariance Constraints with Non-stationary Continuous Filters. Tycho FA van der Ouderaa, David W Romero, Mark van der Wilk. NeurIPS 2022

[5] Bispectral Neural Networks, Sophia Sanborn, Christian Shewmake, Bruno Olshausen, Christopher Hillar. ICLR 2023
We thank the authors for sharing their thoughts on the scope and context of the work within the literature. My worries still hold, so I maintain my score. GCNNs came out 8 years ago, and basic unconstrained CNNs on MNIST and CIFAR10 are quite far from empirical practice. In my opinion, experiments that actually significantly benefit from equivariance / approximate equivariance / equivariance detection would substantially improve the paper. There are many tasks / application areas that the community claims do or may benefit from equivariance, so it would be good to see experiments in such areas. I worry that factors such as the scalability of your method could harm its applicability in these tasks / applications, so experiments would be needed to reject this hypothesis.
I appreciate the author's further explanations and comments. I remain my score of a 7 and recommend acceptance, as I deem this a technically solid paper that will have high impact in the community. This is conditional on author's including some more discussion on the use of regularization and the computational concerns, as well as the issues raised by other reviewers.
Dear reviewer 1myY, we believe to have addressed any weaknesses raised in your original review, an acknowledgment or response to our rebuttal would be much appreciated. We are very much open to address any concerns in the remaining discussion period.
Dear reviewer 3Y5V, we believe to have addressed any weaknesses raised in your original review, an acknowledgment or response to our rebuttal would be much appreciated. We are very much open to address any concerns in the remaining discussion period.
Dear reviewer uLBU, we believe to have addressed any weaknesses raised in your original review, an acknowledgment or response to our rebuttal would be much appreciated. We are very much open to address any concerns in the remaining discussion period.
Thank you for your response, and we are glad some of your concerns have been addressed. Regarding your remaining concern about state-of-the-art performance, it is important to highlight that the primary aim of our approach is in line with regular group convolutional methods, focusing on learning useful constrained functions rather than competing directly with SOTA image models. This distinction is crucial as it frames our method as a tool for understanding and utilizing inherent symmetries in data. Note that the method can easily be extended to any task where signals act on a discretized domain, such as volumetric grids. In these cases there is some evidence that having unconstrained models are more prone to overfitting [1] and our method's ability to inherently recognize and utilize symmetries can be particularly advantageous. For the sake of developing the method, we consider them out of scope for this work.

As such, in the current experiments we utilize MNIST and CIFAR-10 to demonstrate the main claims about symmetry discovery and feasibility. Additionally, while many CNNs rely on extensive data augmentation to achieve SOTA results, there is emerging evidence suggesting that such strategies might not always align well with real-world data distributions, potentially leading to suboptimal generalization [2, 3, 4, 5]. By contrast, our constrained approach offers a systematic way to learn and adapt to symmetries present in the data.

In direct comparisons, our model shows that it can match the performance of unconstrained CNNs, underlining its efficacy even without clear accuracy gains in the standard experimental setups used (see point 4 in the general response). In these cases the CNN models have a considerably larger number of trainable parameters, implying that our method may not need to operate at the same scale/channel capacity to achieve satisfactory performance. This underlines the method's potential utility in settings where understanding and incorporating data-driven symmetries are crucial. We believe that further development and application of this method in contexts where equivariance is highly valued will substantiate its utility and is left as future work.
We kindly encourage considering the broader context of our research’s objectives and its potential contributions to the field.

**References**

[1] Regular SE(3) Group Convolutions for Volumetric Medical Image Analysis. Thijs P. Kuipers and Erik J. Bekkers. MICCAI 2023

[2] Learning Equivariances and Partial Equivariances from Data. David W Romero, Suhas Lohit. NeurIPS 2022

[3] Learning Invariances in Neural Networks. Gregory Benton, Marc Finzi, Pavel Izmailov, Andrew Gordon Wilson. NeurIPS 2020

[4] Learning Layer-wise Equivariances Automatically using Gradients. Tycho F. A. van der Ouderaa and Alexander Immer and Mark van der Wilk. 2023

[5] Relaxing Equivariance Constraints with Non-stationary Continuous Filters. Tycho FA van der Ouderaa, David W Romero, Mark van der Wilk. NeurIPS 2022
Hello authors. Thank you for the response, and apologies for my late response.

Your elaboration on (Question 1) is helpful (although it is clear from your paper, I forgot since I was thinking in terms of equivariant networks). I think that this, as well as your new ablations on less constraining of the learned representations, would be useful additions to your paper.

My remaining worries are more high-level, on the utility of the method. In my view, the utility of approximate equivariance or discovering symmetries is not adequately demonstrated or achieved with the given method. The experiments are only on augmentations of MNIST, or on non-SOTA regimes in CIFAR10. This could be fine, if the model were developed with methods that one could see being useful in different regimes in the future. However, I think the poor efficiency and scalability of the model severely harms this. On the axes of efficiency and accuracy, the model is not efficient, and does not have clear accuracy gains in interesting experimental areas. Perhaps this work would benefit from expriements in areas (say, in the physical sciences) that more clearly desire equivariance, where equivariant models are actually SOTA or near SOTA.
Dear Reviewers,

The deadline for the reviewer-author discussion period is approaching. If you haven't done so already, please review the rebuttal and provide your response at your earliest convenience.

Best wishes, AC