Title: Learning Infinitesimal Generators of Continuous Symmetries from Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: Symmetry Discovery

Geometric Deep Learning

Lie Point Symmetry

Neural PDE Solver
Abstract: Exploiting symmetry inherent in data can significantly improve the sample efficiency of a learning procedure and the generalization of learned models. When data clearly reveals underlying symmetry, leveraging this symmetry can naturally inform the design of model architectures or learning strategies. Yet, in numerous real-world scenarios, identifying the specific symmetry within a given data distribution often proves ambiguous. To tackle this, some existing works learn symmetry in a data-driven manner, parameterizing and learning expected symmetry through data. However, these methods often rely on explicit knowledge, such as pre-defined Lie groups, which are typically restricted to linear or affine transformations. In this paper, we propose a novel symmetry learning algorithm based on transformations defined with one-parameter groups, continuously parameterized transformations flowing along the directions of vector fields called infinitesimal generators. Our method is built upon minimal inductive biases, encompassing not only commonly utilized symmetries rooted in Lie groups but also extending to symmetries derived from nonlinear generators. To learn these symmetries, we introduce a notion of a validity score that examine whether the transformed data is still valid for the given task. The validity score is designed to be fully differentiable and easily computable, enabling effective searches for transformations that achieve symmetries innate to the data. We apply our method mainly in two domains: image data and partial differential equations, and demonstrate its advantages. Our codes are available at \url{https://github.com/kogyeonghoon/learning-symmetry-from-scratch.git}.

Learning Infinitesimal Generators of Continuous Symmetries from Data

Abstract

Exploiting symmetry inherent in data can significantly improve the sample efficiency of a learning procedure and the generalization of learned models. When data clearly reveals underlying symmetry, leveraging this symmetry can naturally inform the design of model architectures or learning strategies. Yet, in numerous real-world scenarios, identifying the specific symmetry within a given data distribution often proves ambiguous. To tackle this, some existing works learn symmetry in a data-driven manner, parameterizing and learning expected symmetry through data. However, these methods often rely on explicit knowledge, such as pre-defined Lie groups, which are typically restricted to linear or affine transformations. In this paper, we propose a novel symmetry learning algorithm based on transformations defined with one-parameter groups, continuously parameterized transformations flowing along the directions of vector fields called infinitesimal generators. Our method is built upon minimal inductive biases, encompassing not only commonly utilized symmetries rooted in Lie groups but also extending to symmetries derived from nonlinear generators. To learn these symmetries, we introduce a notion of a validity score that examine whether the transformed data is still valid for the given task. The validity score is designed to be fully differentiable and easily computable, enabling effective searches for transformations that achieve symmetries innate to the data. We apply our method mainly in two domains: image data and partial differential equations, and demonstrate its advantages. Our codes are available at https: //github.com/kogyeonghoon/learning-symmetry-from-scratch.git .

1 Introduction

Symmetry is fundamental in many scientific disciplines, crucial for understanding the structure and dynamics of physical systems, datasets, and mathematical models. The ability to uncover and leverage symmetries has become increasingly important in machine learning and scientific research due to its potential to improve model efficiency, generalization, and interpretability. By capturing inherent symmetrical properties, models can learn more compact and informative representations, leading to improved performance in tasks like supervised learning [31, 29, 4, 7, 33], self-supervised learning [8, 15, 23], and generative models [19, 11, 18].

Previous methods for learning symmetry have often relied on the explicit parameterization of group representations based on predefined generators, which can be limited in capturing various symmetries, including transformations that do not align along the generators. For example, when searching for Lie group symmetries in images or physics data, existing methods [3, 34] parameterize a group action g as the matrix exponential of a linear combination of linear or affine Lie algebra generators L i with their learnable coefficients w i as g = exp( ∑ i w i L i ) . In the affine transformations of images in

( x 1 , x 2 ) -coordinates, there are six generators, each corresponding to translation, scaling, and shearing operations with respect to the x 1 -axis and x 2 -axis. Although there exist some methods that directly learn the generators, they are either bound to the general linear group GL ( n ) , which cannot account for non-affine or non-linear transformations [24], or are not guaranteed to find the correct symmetry in real-world image datasets [9, 13].

When searching for symmetries in high-dimensional real-world datasets, we can take advantage of the fact that the data can be interpreted as a function f : X → Y , such as images, which are functions from the 2D Euclidean space to the color space. Another notable example of such data is partial differential equations (PDEs), where the data take the form u : X → U and the Lie symmetries are defined as transformations on the space X × U . There have been significant advances in Lie symmetry analysis in recent years, for both academic and industrial purposes, mostly involving extensive symbolic calculations and relying on computer algebra systems [26]. Discovering Lie symmetries of PDEs from data without prior knowledge is an unexplored topic, except for the work of Gabel et al. [14], which learns the symmetry generators of various PDEs in a supervised learning setup.

In this work, we propose a novel method for learning continuous symmetries, including non-affine transformations, from data without prior knowledge. By modeling one-parameter groups using Neural Ordinary Differential Equation (Neural ODE) [6], we establish a learnable infinitesimal generator capable of producing a sequence of transformed data through ODE integration. We design an appropriate validity score function that measures how much the transformation violates the invariance to certain criteria defined depending on the target task, and learn the generators by optimizing towards the validity score of the data transformed through ODE integration. For example, in an image classification dataset, we use a pre-trained feature extractor and define the validity score to be the cosine similarity between the features extracted from the original image and the transformed image. For PDEs, the validity score is defined by the numerical errors of the original equations after the transform. The validity scores are chosen based on the characteristics of the target tasks, and designed to be fully differentiable, so that the symmetry can be learned via gradient descent in an end-to-end fashion. We also incorporate two regularizations, orthonormality and Lipschitz loss, which prevent the learned generators from converging to a trivial solution.

Subsequently, we demonstrate that our method indeed discovers the correct symmetries in both image and PDE datasets. To the best of our knowledge, our research is the first to retrieve affine symmetry in the entire space of continuous transformations using the CIFAR-10 classification dataset, as shown in Figure 1. Moreover, our method excels in identifying non-affine symmetries and approximate symmetries in PDE tasks. We further demonstrate that the learned generators can be leveraged to develop automatic augmentation generators, which can be used to produce augmented training data for both image classification tasks and neural operator learning tasks of PDEs [21]. We provide empirical evidence that the models trained with data augmented by our learned generators perform competitively with those trained with traditional closed-form transforms such as Lie point symmetry (LPS) [4]. Moreover, we show that the approximate symmetries discovered by our method, which cannot be found by classical methods, can also boost the performance of the models, especially when the size of the training data is small.

(a)

(b)

(c)

InfinitesimalGeneratorsofSymmetries

TransformationofImages

TransformationofPDEs

t-tsl

V3

t

V3

V

V7

gal

t

0.0

0.1

0.2

0.3

0.00

X

0.05

X

0.10

X

0.15

X

Transformation Scale

TransformationScale

2 Preliminaries: One-parameter Group

In this section, we present the basic definitions of a one-parameter group, which we use to parameterize the symmetric transformations learned from the data.

Consider an unknown Euclidean domain Z ⊆ R n and a smooth vector field V : Z → R n . A path γ : I = ( a, b ) ⊆ R →Z satisfying d ds γ ( s ) = V ( γ ( s )) for all s ∈ I is a curve that travels around the domain Z with a velocity given by the vector field V . Along the curve γ , a point x 0 = γ ( a 0 ) can be transported to γ ( a 0 + s ) by flowing along the vector field V by time s . We define the flow ϑ V s by ϑ V s ( x 0 ) = γ ( a 0 + s ) of V as in Figure 2. This flow is computed by solving an ODE

with initial condition ϑ V 0 ( x ) = x for all x ∈ Z [20].

The flow ϑ V is governed by an autonomous ODE, i.e., an ODE independent of the temporal variable s . Due to properties of autonomous ODEs, the flow ϑ V exists uniquely and it is smooth in both variables s and x . Assuming a mild condition on V , such as V extends to a compactly supported vector field ˜ V on R n , the ODE does not terminate in R n in finite time and hence ϑ V s is defined for all s ∈ R . In that case, the flow satisfies a group equation

for all s 1 , s 2 ∈ R . It means that the flow can be regarded as a group action of R on R n , transforming elements of Z ⊆ R n . For this reason, ϑ V s is also called a one-parameter group , and the vector field V is called an infinitesimal generator of the one-parameter group.

On Z = R n , a constant vector field V ( x ) = v ∈ R n gives rise to a translation x ↦→ x + s v . For a matrix A ∈ R n × n , a vector field V ( x ) = Ax gives rise to an affine transformation x ↦→ exp( s A ) x , where exp is the matrix exponentiation. Multiple infinitesimal generators may span a vector space g , and if g satisfies some algebraic condition (closure under the Lie bracket), then g forms a Lie algebra . Composing the elements of one-parameter groups of elements in g gives rise to a Lie group G . The correspondence between G and g is called Lie group-Lie algebra correspondence.

Continuous symmetries are commonly defined by a Lie group G , acting on some domain and keeping the transformed objects invariant with respect to some criterion. We model symmetries by specifying their infinitesimal generators whose composition of one-parameter groups comprises the symmetries of that domain.

Below, we describe two representative examples that will be discussed extensively in the remainder of the paper: images (interpreted as functions on 2D planes) and PDEs.

2.1 Images and Their Symmetries

Consider a rescaled image of the form f : X = [ -1 , 1] 2 ⊆ R 2 → Y = [0 , 1] 3 ∈ R 3 . The affine transformations on R 2 have the form x = ( x 1 , x 2 ) ↦→ Ax + b for a matrix A ∈ R 2 × 2 and a vector b ∈ R 2 . The Affine transformations form the 6-dimensional Affine group Aff(2) , and it has a corresponding 6-dimensional Lie algebra having a basis { L 1 , . . . , L 6 } given as in Table 1. The symmetries of images are often exploited as a data augmentation strategy for learning image classifiers, under an assumption that the transforms do not alter the identity or semantics of the images to be classified.

2.2 PDEs and Their Symmetries

Given an n -dimensional independent variable x = ( x 1 , · · · , x n ) ∈ X ⊆ R n and an m -dimensional dependent variable u = u ( x ) = ( u 1 ( x ) , · · · , u m ( x )) ∈ U ⊆ R m , we denote by u ( i ) the collection of all i -th partial derivatives of u with respect to x . A partial differential equation ∆ on u ( x ) of order k is defined by a set of algebraic equations ∆ ( x , u , u (1) , · · · , u ( k ) ) = 0 involving all the variables and their partial derivatives. For example, two scalar independent variables x, t ∈ R and one scalar dependent variable u ( x, t ) ∈ R governed by equation ∆ = u t + uu x + u xxx = 0 gives the 1-dimensional Korteweg-de Vries (KdV) equation, where we denote partials using subscripts, e.g. u t = ∂u ∂t and u xxx = ∂ 3 u ∂x 3 . The KdV equation is commonly used to model the dynamics of solitons, e.g. shallow water waves [36]. The KdV equation described above is an example of 1-dimensional scalar-valued evolution equation. Such an equation takes the form u = u ( x, t ) ∈ R with its governing equation of the form

for some function F . In this paper, we only deal with 1D scalar-valued evolution equation on a fixed periodic domain x ∈ [0 , L ] .

Continuous symmetries of PDEs are commonly parametrized by a one-parameter group on X × U . Denote ( ξ , µ ) = ( ξ [ x , u ] , µ [ x , u ]) an infinitesimal generator defined on X × U . Then the PDE ∆ possesses the infinitesimal generator of symmetry ( ξ , µ ) if the equation is still satisfied after transforming both the independent variable x and the dependent variable u [27, 5, 26]. Symmetries of PDEs are categorized by how the generators ( ξ , µ ) depend on ( x , u ) . The symmetry is a Lie point symmetry (LPS) if the value of ( ξ , µ ) at each point ( x , u ( x )) depends only on the point value ( x , u ( x )) itself. If ( ξ , µ ) also depends on the derivatives u (1) , · · · , u ( k ) at that point, it is called a Lie-Bäcklund symmetry or generalized symmetry. If ( ξ , µ ) depends on integrals of u , then it is called a nonlocal symmetry. Finding an LPS of a PDE ∆ can be done algorithmically under some mild assumptions on ∆ . However, there is no general recipe of finding Lie-Bäcklund symmetries or nonlocal symmetries, and discovering such symmetries remains an active area of research.

3 Related Work

Symmetry discovery. Approaches to learning symmetries can be categorized by addressing two questions: (a) where do they search for symmetries , and (b) what are they aiming to learn . One line of research aims to learn ranges, focusing on determining the ranges of transformation scales that enhance learning when employed as augmentation techniques. For example, Benton et al. [3] learns transformation ranges of predefined transformations by treating them as learnable parameters and backpropagating through differentiable transformations.

Another line of research aim to learn subgroups of bigger candidate groups, typically a linear group GL ( n ) or an affine group Aff( n ) . For example, Desai et al. [10] use the Generative Adversarial Network (GAN) to search for symmetries, with the generator transforming data by group elements sampled from the candidate group and the discriminator verifying whether the transformed data sill lies in the data distribution. Similarly, Yang et al. [34] employ the GAN approach, but generator of GANmodels infinitesimal generators instead of the subgroup itself, and learns affine symmetries such as rotation of images and Lorentz symmetry of high-energy particles. As an alternative, Moskalev et al. [24] proposed an idea of extracting symmetries from learned neural network by differentiating through it, and retrieved 2D rotation in the linear group using the rotation MNIST dataset.

Finally, learning symmetries with minimal assumption , i.e. without assuming the infinitesimal generators are linear or affine, is an area of large interest. An early attempt of Rao & Ruderman [28] models infinitesimal generator by a learnable matrix from the pixel space to the pixel space, and learn 2D rotation by solving a task that compares original images and rotated ones, where the images are 5 × 5 random pixels. Sohl-Dickstein et al. [32] takes the similar approach with eigen-decomposing the learnable matrix. Dehmamy et al. [9] builds a convolution operation whose kernel encodes learnable infinitesimal generators, and retrieved 2D rotation from random 7 × 7 images by comparing original and transformed ones, and Yang et al. [35] uses an autoencoder to simplify nonlinear symmetries into linear ones. Our work closely aligns with Liu & Tegmark [22] and Forestano et al. [13], which model one-parameter groups by an MLP and learn the symmetries from an invariant scalar quantity. To the

best of our knowledge, learning correct symmetries with minimal assumption was only achieved with toy datasets, far from real-world datasets such as CIFAR-10.

Utilizing symmetries in deep learning. An effective method for leveraging symmetries in deep learning is data augmentation [31]. In the image domain, there are numerous augmentation techniques available [29], most of which are based on geometric properties of images. Although data augmentation techniques have been primarily explored in the context of images, recent studies by [4, 23] have demonstrated that symmetries can also be used for augmenting data in the training of neural PDE solvers. In addition to data augmentation, some approaches involve designing new neural network architectures that inherently reflect the group symmetries of the input data [7]. Wang et al. [33] applied a similar strategy within the PDE domain.

4 Learning Continuous Symmetries with One-Parameter Groups

4.1 Training Process

Given a learning task with a dataset D ⊂ A in an underlying space A = { f | f : X → Y} , we aim to model symmetry by a one-parameter group ϑ s acting on A , as explained in § 4.3. We define a continuous symmetry by stating that ϑ is a symmetry of this task if there exists some σ > 0 such that for any data point f ∈ D and transformation scale s ∈ [ -σ, σ ] , the transformed data point ϑ s ( f ) remains valid for this task. We assume the existence of a differentiable validity score S ( ϑ s , f ) ∈ R , such that ϑ s ( f ) ∈ A is valid if S ( ϑ s , f ) < C for a certain threshold C ∈ R . Then, a one-parameter group ϑ is a symmetry of the task if S ( ϑ s , f ) < C for all f ∈ D .

The validity score depends on the nature of the target task, though no strict criterion exists. As long as it is differentiable and the valid data aids learning, it is considered acceptable. For instance, we can define the validity based on a negative log-likelihood of a probabilistic model. In § 4.2, we discuss the validity scores to be used for image and PDE data.

Once a validity score is defined, we learn a symmetry ϑ ∗ by minimizing the validity scores of transformed data,

where the arg min is taken over the entire class of smooth one-parameter groups. Since the learning is performed in function space, we appropriately constrain the function space using a regularizer, as described in § 4.4. Once symmetries are learned, they reveal the symmetrical properties of the target task, which can then be exploited to augment the training data.

4.2 Task-specific Definition of Validity Score S

Images. In image-related tasks, we define a validity score using a pre-trained neural network. Let D be an image classification dataset consisting of data of the form ( f, y ) ∈ A × R , where f is an image and y is a label. Also let H cls ◦ H fext : A → R be a learned neural network, where we denote by H fext : A → R k the feature extractor and H cls : R k → R the classifier. We define the validity score S ( ϑ s , f ) as the cosine similarity between the features before and after the transformation:

where sim is the cosine similarity defined as sim( v 1 , v 2 ) = | v 1 · v 2 | ∥ v 1 ∥∥ v 2 ∥ for all v 1 , v 2 ∈ R k \ { 0 } .

PDEs. Let u ( x ) be a solution of a given PDE ∆ , discretized on a rectangular grid X grid = { x i } N grid i =1 . For a transformed data ϑ s ( u ) , we measure the violation of the equality ∆ = 0 to assess whether the transformed data is still a valid solution. Using an appropriate numerical differentiation method, we directly compute the value of the PDE, denoted as ∆ ( ϑ s ( u )) , which represents the error of ϑ s ( u )

as a solution of ∆ , taking a value ∆ ( ϑ s ( u )) i at grid point x i . The validity score is defined by the summation of all PDE errors across the grid points:

For example, for a solution u ( x ) of the 1D KdV equation, we examine whether the transformed solution ˜ u = ϑ s ( u ) satisfies ˜ u t +˜ u ˜ u x +˜ u xxx = 0 where the partials are computed using a numerical differentiation method.

4.3 Parametrization of One-Parameter Groups using Neural ODE

On a Euclidean domain Z ⊆ R n , we model an infinitesimal generator with an MLP h θ : Z ⊆ R n → R n . The infinitesimal generator h θ gives rise to a one-parameter group ϑ h θ s . We sample a transformation scale α ∼ Unif([ -σ, σ ]) for a predefined hyperparameter σ ∈ R > 0 . To transform a point x ∈ Z along this one-parameter group by an amount α ≥ 0 , we use a numerical ODE solver to solve the ODE for γ : [0 , α ] →Z satisfying

and obtain a transformed data point ˜ x = ϑ h θ α ( x ) = γ ( α ) . If α < 0 , we compute ϑ h θ α ( x ) = ϑ -h θ -α ( x ) by integrating -h θ instead of h θ using the ODE solver. We can backpropagate through the numerical ODE solver using the adjoint method [6] to learn θ .

Let f : X → Y be a data point on a domain A . As A is a space of functions, naïvely modeling symmetry on A may ignore the geometry implied in the input space X . Instead, we define two transformations: ϑ X on X and ϑ Y on Y , and induce a transformation of f by

where we abuse notation and write the transformed function as ϑ X ( f ) and ϑ Y ( f ) . For an image represented as a discretized function f : X → Y from X = [ -1 , 1] 2 and Y = [0 , 1] 3 , ϑ X corresponds to spatial transformations such as translation or rotation, and ϑ Y corresponds to color space transformations. For a PDE, a 1D scalar-valued evolution equation on a fixed periodic domain takes the form u ( x, t ) ∈ U = R with ( x, t ) ∈ [0 , L ] × [0 , T ] = X ⊆ R 2 , and we parameterize an infinitesimal generator on a product space X × U ⊆ R 3 by an MLP. Then, a transformation on ( x, t, u ) ∈ X × U induces a transformation on the solution of the PDE u ( x, t ) .

4.4 Objective Functions

Symmetry loss. Let N sym be the number of symmetries to be learned. Let ( h ( a ) θ ) N sym a =1 be the infinitesimal generators computed from a single MLP. For each a ∈ { 1 , . . . , N sym } , we sample a transformation scale s a ∼ Unif([ -σ, σ ]) to transform f via numerical integration. The parameter θ is optimized by minimizing the average validity score over the training data,

Orthonormality loss. Learning only with the symmetry loss may result in trivial solutions such as the zero vector field or the same vector field repeated in multiple slots. To prevent this, we introduce the orthonormality loss to regularize the model towards learning orthonomral vector fields. Specifically, given two vector fields V 1 , V 2 : Z → R n , we define an inner product as,

with a suitable weight function ω ( x ) : Z → R and a discretized grid Z grid of Z of size |Z grid | . Given this definition, we first normalize each generator by its norm to ensure ∥ h ( a ) θ ∥ 2 = 1 . Then we compute the orthonormality loss as,

where sg( · ) denotes the stop-gradient operation to ensure that the constraint ⟨ h ( a ) θ , h ( b ) θ ⟩ = 0 only affects the latter slot ( b ). By doing this, if the true number of symmetries N ∗ sym is less than or equal to the assumed number of symmetries N sym , the learned symmetries will be aligned in the first N ∗ sym slots.

Lipschitz loss. We further introduce inductive biases to the infinitesimal generators we aim to learn. For instance, an infinitesimal generator moving only a single pixel near the boundary by a large scale would be undesirable. This idea can be implemented using Lipschitz continuity. For a grid point x i ∈ Z grid and its neighboring point x j ∈ nbhd( x i ) ⊂ Z grid, we expect the vector field V to satisfy the Lipschitz condition,

To regularize the model toward the Lipschitz condition, we introduce the Lipshictz loss,

Total loss and loss-scale-independent learning. We jointly minimize the three loss functions with suitable weights w sym , w ortho , w Lips > 0 and learn the weights θ of MLP using a stochastic gradient descent:

To minimize the computational burden of hyperparameter tuning, we ensure that all the loss terms have a natural scale , i.e. a dimensionless scale independent of the context. For example, when penalizing the inner product in Equation 11, we apply arccos to the normalized inner product to ensure the loss term lies in [0 , π/ 2) . Similarly, the scale of the PDE validity score S ( ϑ s , u ) in Equation 9 depends on the scale of the data u . When penalizing it, we apply the log function so that the gradients are scaled automatically as ∇ θ log( S ( ϑ s , u )) = ∇ θ S ( ϑ s , u ) /S ( ϑ s , u ) .

Here we describe the generic training process, but the actual implementation requires non-trivial task-specific designs, such as the choice of the weighting function w ( x ) or the method for locating the transformed data on the target grid. We defer these details for image and PDE tasks to Appendix A.

4.5 Comparison With Other Methods

Here, we compare our method with other recent symmetry discovery methods. The differences mainly arise from (a) what they aim to learn (e.g., transformation scales or subgroups from a larger group) and (b) their assumptions about prior knowledge (e.g., complete, partial, or no knowledge of symmetry generators). Another important distinction is the viewpoint on symmetry: some methods learn symmetries that raw datasets inherently possess (implicit), while others learn symmetries from datasets explicitly designed to carry such symmetries (explicit).

Some recent symmetry discovery works are listed in Table 2. We emphasize that our method excels in two key aspects: (a) our learning method reduces infinitely many degrees of freedom, (b) our method works with high-dimensional real-world datasets. For example, while LieGAN [33] and LieGG [24] reduce a 6-dim space (affine) to a 3-dim space (translation and rotation) in an image dataset, ours reduces an ∞ -dim space to a finite one. L-conv [9] also does not assume any prior knowledge, but it is limited in finding rotation in a toy task, where it learns rotation angles of rotated images by comparing them with the original ones, which are 7x7 random pixel images.

5 Experiments

5.1 Images

We use images of size 32 × 32 from the CIFAR-10 classification task. Since our method does not model discrete symmetry, we use horizontal flip with 50% probability by default. We train a ResNet-18 model, which will be used as the feature extractor H fext in Equation 5. The weight function

(a)

(b)

(c)

InnerProducts

LearnedGenerators

V2

V1

0.00

00

-0.00 0.010.00 0.01 -0.00 0.010.010.000.00

-0.00 0.00 0.010.000.000.000.000.00

(1,0)

-0.00

0.97

-0.010.040.03-0.01-0.060.10-0.07-0.13

1.2

V5

V3

V4

0.01-0.00

0.00 -0.00 0.00

0.01 -0.01 0.00 0.00

1.00

0.000.000.000.000.000.000.00

1.00

0.00

1.00

0.000.000.000.00

0.00-0.000.000.00

0.00

0.00

(×1,0)

(0,1)

86'0-

0.000.02

-0.000.07-0.020.020.04-0.06-0.030.10-0.07

0.16

-0.79

-0.45-0.36

0.01

0.10

0.030.05

Measure

1.0

0.8

V

V6

0.000.000.000.00

0.01-0.00 -0.000.000.00 0.00

0.00

0.00 0.00 0.00 0.00

0.00

-0.000.00

(0,x2)

0.12

-0.00-0.10

-0.55

0.79

0.18

-0.12-0.020.080.01

0.6

0.4

V10

Vg

V8

VV2VV4VsV6VVsVgV10

0.000.000.000.000.000.000.00 0.00

00°000000000000000000°0000000

0.01

0.000.000.000.00-0.000.00

1.00

-0.000.00

1.00

-0.00

(×2,0)

(0,x1)

-0.050.00

V1V2V3V4V5V6V7VsV9V10

0.01

-0.13

-0.81

0.49

0.03

0.22

0.04

0.39

-0.54-0.07-0.19-0.070.01

-0.72

0.01

0.09-0.04-0.04

0.2

0.0

LearnedGenerators

LearnedGenerators

LearnedGenerators

on the pixels is computed as explained in Appendix A.1. We expect to find 6 affine generators and we use an MLP modeling 10 vector fields in the pixel space [ -1 , 1] 2 ⊆ R 2 , expecting the first six learned vector fields to be the affine generators. We learn the Equation 14 using stochastic gradient descent with w sym = 1 and w ortho , w Lips = 10 . The parameter σ , which controls the scale of transformation, is set to σ = 0 . 4 , and the Lipschitz threshold τ is set to τ = 0 . 5 . Other details are described in Appendix B.1. We conducted three trials with random initializations and report the full results in Appendix C.1. Furthermore, we also learn symmetries in the color space , and their results are shown in Appendix G.

Learned symmetries. Since we expect to learn affine symmetries, we compare the results with the affine basis { L 1 , · · · , L 6 } defined in Table 1. We compute the inner products ⟨ V, L i ⟩ of the learned vector field V with L i for i = 1 , · · · , 6 to measure how much the learned vector fields contain the affine basis and measure the affine-ness of vector field by Affine-ness ( V ) 2 = ∑ 6 i =1 ⟨ V, L i ⟩ 2 .

In all experiments, we successfully retrieve six linearly independent affine generators in the first six slots. Figure 4a shows that the learned generators are orthogonal to each other, as desired. Figure 4b shows the inner product between the learned generators and the affine generators. Since the affineness measure of the first 6 learned generators in Figure 4c is almost close to 1, we can read out the affine components in Figure 4b and say that e.g., V 1 ≈ (0 , -0 . 98 + 0 . 12 x 2 ) . Notably, two translation generators are found in the first two slots, indicating that the two translations are the most invariant one-parameter group among the entire class of one-parameter groups on the pixel space. After the two translation generators, four affine generators are learned, indicating that affine transformations are the next most invariant transformations. In particular, the third and fourth generators are close to the rotation generator and the scaling generator, respectively. The remaining four generators fix pixels close to the center and transform boundary pixels by a large magnitude.

nKdV

cKdV

Components

Components

1.0

x-tsl

-1.00-0.060.02-0.00

1.0

x-tsl

-0.920.28-0.270.02

0.5

g2

0.06

-1.000.030.02

0.5

g2

0.21-0.22-0.950.03

g3

-0.02 -0.03-1.00 0.01

t-tsl

0.33

0.93-0.14-0.08

0.0

0.0

V1

1 V2V3V4

V1 V2 V3 V4

V1

1V2V3 V4

V1 V2 V3 V4

Analysis. Unlike other symmetry discovery researches [24, 9] that use datasets which are explicitly designed to be symmetric such as rotation MNIST, we discovered affine transformations from CIFAR10 with no augmentation except horizontal flip. Moreover, we extract the symmetries from the ResNet trained with CIFAR-10 without augmentation. This implies that although CIFAR-10 is not explicitly composed to be symmetric under affine transformations, the dataset possesses intrinsic affine symmetry. This also implies that even the ResNet trained without augmentation possesses invariance under affine transformation. It is widely believed that the strong generalization power of neural networks is linked to the augmentation insensitivity of the neural networks [25]. Our results show that ResNet is insensitive to affine transformation even when not explicitly designed to be so, supporting this hypothesis. This result implies that augmentation explicitly amplifies the insensitivity by using transformed data for training. Our analysis is tangential to that of Gruver et al. [16], which measures the extent of invariance using derivatives along the infinitesimal generators instead of ODE integration.

Augmentation results. We train a ResNet-18 model using CIFAR10 classification data, applying the learned symmetries as data augmentation. We compare the results of no-augmentation, default augmentation (horizontal flip and random crop), and affine transformation, with transformation scale searched in { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } . We conduct five experiments with random initialization for all the settings and report the results in Table 3.

5.2 PDEs

We follow the experimental setting of Brandstetter et al. [4], which use the Korteweg-de Vries (KdV) equation, the Kuramoto-Shivashinsky (KS) equation, and the Burgers' equation on a 1D periodic domain as experiments. They all have time translation, space translation, and the Galilean boost as LPSs. To consider PDEs with non-trivial and non-affine symmetries, we add two variants of the KdV, namely the nKdV equation and the cKdV equation. The nKdV is yielded by a nonlinear time translation of the original KdV, and the cKdV is the cylindrical KdV equation, having an extra time-dependent term. The equations are listed in Table 4, and their symmetries are listed in Table 5. Note that some symmetries are ruled out since we work within a fixed periodic domain, e.g., the scaling symmetry of the KdV. Since there are at most three symmetries, we open four slots in an MLP and learn symmetries using weights w sym , w Lips = 1 and w ortho = 3 .

Learned symmetries. We compare the learned symmetries with the ground-truth symmetries using inner products. We found the ground truth symmetries in all the experiments as in Figure 5 and Figure 10.

KdV

KS

nKdV

cKdV

Ground truth

Ground truth

Learned

10-3

Learned

NMSE

Learned

NMSE

E

10%

一

Learned

NMSE

E

Test NMSE

10%

10-1

10-4

10-1

Test

Test

Test

10-2

10-2

10-5

10-1

25

27

29

25

27

29

25

27

29

25

27

29

#Data

#Data

#Data

#Data

Interestingly, for the Burgers' equation and the cKdV equation, we additionally found the u -axis rescaling operation (0 , 0 , u ) and the time translation (0 , 1 , 0) respectively. Applying the u -axis rescaling u ↦→ cu for c ≈ 1 to the Burgers' equation u t + uu x -νu x x = 0 gives cu t + c 2 uu x -νcu x x = c ( c -1) uu x , leaving only the uu x term. The uu x term in Burgers' is called the convection term , and it is approximately zero in most region and spikes in some small region. Similarly, the time translation t ↦→ t + c for c ≈ 0 fixes the first three terms in the cKdV equation u t + uu x + u xxx + u/ (2( t +1)) = 0 and only changes the last term u/ (2( t +1)) by a negligible amount. These are not LPSs of the given equations, but the error of the PDE after transformation is smaller than the error of the numerical differentiation method. These are approximate symmetries (AS), and the theory of AS is also of great interest in the symmetry analysis of PDEs [2, 1].

Augmentation results. We use the learned symmetries as data augmentation and train Fourier Neural Operators (FNOs). The detailed experiment setting is described in Appendix B.2. Since FNOs are extremely sensitive to numerical error, we employ Whittaker-Shannon interpolation, explained in Appendix D.2, to resample the transformed results. The results are depicted in Figure 6. In all cases, data augmentation using the learned symmetries improve the performance, almost close to the results using the ground truth symmetries. The detailed results are in Appendix C.2. Additionally, we verify that the approximate symmetry of cKdV is also beneficial to training, as shown in Table 6, especially when the numbers of data points is low, proving the effectiveness of symmetries extracted from data.

Ablations. Additional ablation studies on numerical methods, such as numerical differentiation and interpolation, and hyperparameter sensitivity are conducted, and their results are presented in Appendices E and F.

6 Conclusion

We have introduced a novel method for learning continuous symmetries, including non-affine transformations, from data without prior knowledge. By leveraging Neural ODE, our approach models one-parameter groups to generate a sequence of transformations, guided by a task-specific validity score function. This approach captures both affine and non-affine symmetries in image and PDE datasets, enhancing automatic data augmentation in image classification and neural operator learning for PDEs. The learned generators produce augmented training data that improve model performance, particularly with limited training data.

Limitation. However, despite its flexibility, our method requires careful selection of numerical methods, such as numerical differentiation and interpolation, to ensure stable training and the ODE integration can be computationally large for augmentation generation compared to other augmentation methods. While we focus on image symmetries and LPSs of PDEs, the method could potentially model other symmetries and domains with proper validity scores, suggesting future applications in learning complex symmetries, including conditional and non-local symmetries, in various data types.

Acknowledgments and Disclosure of Funding

This work was partly supported by Institute of Information & communications Technology Planning &Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS-2019-II190075, Artificial Intelligence Graduate School Program(KAIST), No.2022-0-00713, Meta-learning Applicable to Realworld Problems, No.2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics, No.RS-2024-00509279, Global AI Frontier Lab)

References

Baikov, V., Gazizov, R., and Ibragimov, N. K. Approximate symmetry and formal linearization. Journal of Applied Mechanics and Technical Physics , 30(2):204-212, 1989. 10

Baumann, G. Approximate symmetries of partial differential equations. Symmetry Analysis of Differential Equations with Mathematica® , pp. 404-423, 2000. 10

Benton, G. W., Finzi, M., Izmailov, P., and Wilson, A. G. Learning invariances in neural networks from training data. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022) , 2020. 1, 4

Brandstetter, J., Welling, M., and Worrall, D. E. Lie point symmetry data augmentation for neural pde solvers. In International Conference on Machine Learning , pp. 2241-2256. PMLR, 2022. 1, 2, 5, 9, 15, 16

Cantwell, B. J. Introduction to symmetry analysis . Cambridge University Press, 2002. 4

Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018. 2, 6

Cohen, T. and Welling, M. Group equivariant convolutional networks. In International conference on machine learning , pp. 2990-2999. PMLR, 2016. 1, 5

Dangovski, R., Jing, L., Loh, C., Han, S., Srivastava, A., Cheung, B., Agrawal, P., and Soljacic, M. Equivariant self-supervised learning: Encouraging equivariance in representations. In International Conference on Learning Representations (ICLR) , 2022. 1

Dehmamy, N., Walters, R., Liu, Y., Wang, D., and Yu, R. Automatic symmetry discovery with lie algebra convolutional network. Advances in Neural Information Processing Systems , 34: 2503-2515, 2021. 2, 4, 7, 9

Desai, K., Nachman, B., and Thaler, J. Symmetry discovery with deep learning. Physical Review D , 105(9):096031, 2022. 4

Dey, N., Chen, A., and Ghafurian, S. Group equivariant generative adversarial networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021. 1

Dumbser, M. and Käser, M. Arbitrary high order non-oscillatory finite volume schemes on unstructured meshes for linear hyperbolic systems. Journal of Computational Physics , 221(2): 693-723, 2007. 15, 17, 19

Forestano, R. T., Matchev, K. T., Matcheva, K., Roman, A., Unlu, E. B., and Verner, S. Deep learning symmetries and their lie groups, algebras, and subalgebras from first principles. Machine Learning: Science and Technology , 4(2):025027, 2023. 2, 4

Gabel, A., Quax, R., and Gavves, E. Data-driven lie point symmetry detection for continuous dynamical systems. Machine Learning: Science and Technology , 2024. 2

Garrido, Q., Najman, L., and LeCun, Y. Self-supervised learning of split invariant equivariant representations. In Proceedings of The 40th International Conference on Machine Learning (ICML 2023) , 2023. 1

Gruver, N., Finzi, M. A., Goldblum, M., and Wilson, A. G. The lie derivative for measuring learned equivariance. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 , 2023. 9

Hoarau, E. and David, C. Lie group computation of finite difference schemes. arXiv preprint math/0611895 , 2006. 9

Hoogeboom, E., Satorras, V. G., Vignac, C., and Welling, M. Equivariant diffusion for molecule generation in 3d. In Proceedings of The 39th International Conference on Machine Learning (ICML 2022) , 2022. 1

Huang, Y., Peng, X., Ma, J., and Zhang, M. 3dlinker: An E(3) equivariant variational autoencoder for molecular linker design. In Proceedings of The 39th International Conference on Machine Learning (ICML 2022) , 2022. 1

Lee, J. M. Smooth manifolds . Springer, 2012. 3

Li, Z., Kovachki, N. B., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A. M., and Anandkumar, A. Fourier neural operator for parametric partial differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021. 2

Liu, Z. and Tegmark, M. Machine learning hidden symmetries. Physical Review Letters , 128 (18):180201, 2022. 4

Mialon, G., Garrido, Q., Lawrence, H., Rehman, D., LeCun, Y., and Kiani, B. Self-supervised learning with lie symmetries for partial differential equations. Advances in Neural Information Processing Systems , 36:28973-29004, 2023. 1, 5

Moskalev, A., Sepliarskaia, A., Sosnovik, I., and Smeulders, A. W. M. Liegg: Studying learned lie group generators. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022) , 2022. 2, 4, 7, 9

Novak, R., Bahri, Y., Abolafia, D. A., Pennington, J., and Sohl-Dickstein, J. Sensitivity and generalization in neural networks: an empirical study. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings , 2018. 9

Oliveri, F. Lie symmetries of differential equations: Classical results and recent contributions. Symmetry , 2(2):658-706, 2010. ISSN 2073-8994. doi: 10.3390/sym2020658. URL https: //www.mdpi.com/2073-8994/2/2/658 . 2, 4

Olver, P. J. Applications of Lie groups to differential equations , volume 107. Springer Science &Business Media, 1993. 4

Rao, R. and Ruderman, D. Learning lie groups for invariant visual perception. Advances in neural information processing systems , 11, 1998. 4

Shorten, C. and Khoshgoftaar, T. M. A survey on image data augmentation for deep learning. Journal of big data , 6(1):1-48, 2019. 1, 5

Sil, S. Nonclassical symmetries, nonlinear self-adjointness, conservation laws and some new exact solutions of cylindrical kdv equation. International Journal of Applied and Computational Mathematics , 9(5):69, 2023. 9

Simard, P. Y., Steinkraus, D., Platt, J. C., et al. Best practices for convolutional neural networks applied to visual document analysis. In Icdar , volume 3. Edinburgh, 2003. 1, 5

Sohl-Dickstein, J., Wang, C. M., and Olshausen, B. A. An unsupervised algorithm for learning lie group transformations. arXiv preprint arXiv:1001.1027 , 2010. 4

Wang, R., Walters, R., and Yu, R. Incorporating symmetry into deep dynamics models for improved generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021. 1, 5, 7

Yang, J., Walters, R., Dehmamy, N., and Yu, R. Generative adversarial symmetry discovery. In Proceedings of The 40th International Conference on Machine Learning (ICML 2023) , 2023. 1, 4

Yang, J., Dehmamy, N., Walters, R., and Yu, R. Latent space symmetry discovery. In International Conference on Machine Learning , 2023. 4

Zabusky, N. J. and Kruskal, M. D. Interaction of" solitons" in a collisionless plasma and the recurrence of initial states. Physical review letters , 15(6):240, 1965. 4

Zhang, Y.-T. and Shu, C.-W. Eno and weno schemes. In Handbook of numerical analysis , volume 17, pp. 103-122. Elsevier, 2016. 15, 17

A Implementation Details

A.1 Image Dataset

Notation. Images are functions of the form f : X → Y , where X = [ -1 , 1] 2 is the spatial domain and Y = [0 , 1] 3 is the normalized RGB domain. Pixels X grid = { x i } N grid i =1 are discretized through a rectangular grid of X , and images f are discretized on the pixel space by f i = f ( x i ) for all i .

ω ( x ) in orthonormality loss. We first learn a symmetry on the spatial domain X using a feature extractor H hidden taken from a pretrained neural network. One obstacle is that in most image datasets, the main subjects of images are mostly located around the centers, and the regions close to the boundary are filled with backgrounds. In other words, each pixel has a different level of importance, and we may end up learning infinitesimal generators that only move boundary pixels and fix the center.

We take into account the importance of pixels using the weight function ω ( x ) : X → R ≥ 0 . For a discretized image f = { f i } and for each pixel x i , we measure a pixel sensitivity of the image f at the i -th pixel up to the feature extractor H fext as

which can be computed by querying a Jacobian-vector product of H fext with respect to the image gradient at each pixel ∂f i ∂ x i . We define the weight ω ( x ) as the average of pixel sensitivity across the dataset:

However, computing this weight function ω needs N grid · N data times of computation of Jacobianvector products. Instead, we use a Gaussian kernel κ ( x ; ˆ x , σ ) = 1 √ 2 πσ exp( -|| x -ˆ x || 2 2 σ 2 ) , with a fixed σ = 0 . 1 and the center ˆ x sampled uniformly on X , and approximate the weight function by

so that we compute the weight in a stochastic manner. Intuitively, it computes Jacobian-vector product for each larger pseudo-pixel represented by the Gaussian kernels instead of each individual pixel. We iterate over the dataset 20 times, and the computed weight function is shown in Figure 7.

Training details. The infinitesimal generator h ( a ) θ transforms each pixel x i into ˜ x i via ODE integration. In this process, the transformed pixels { ˜ x i } may no longer be located on the rectangular grid X grid, so we use the bilinear interpolation method to resample the transformed image ϑ s ( f ) on X grid. Note that the bilinear interpolation operation is differentiable, thereby we can train the MLP by minimizing the validity score S ( ϑ s , f ) , which is the cosine similarity between the features of f and ϑ s ( f ) :

A.2 PDE Dataset

Notation. Solutions of a 1D scalar-valued evolution equation on a periodic domain take the form u ( x, t ) : X → U for X = [0 , L ] × [0 , T ] and U = R , where [0 , L ] is the spatial domain and [0 , T ] is the temporal domain. Similar to the image task, the domain X is discretized by the rectangular grid X grid = { x i } N grid i =1 = { ( x i , t i ) } N grid i =1 . A discretized solution is a set of tuples ( x i , t i , u i ) where u i = u ( x i , t i ) .

ω ( x ) in orthonormality loss. Unlike in images, we assume that all discretized grid points hold equal importance, so we set ω ( x ) = 1 for all x ∈ X .

Training details. The infinitesimal generators on the product space X × U transform a point u = { ( x i , t i , u i ) } into ˜ u = { (˜ x i , ˜ t i , ˜ u i ) } . Therefore, the transformed solutions are no longer on the rectangular grid. To compute the PDE value ∆(˜ u ) for ˜ u = ϑ s ( u ) in Equation 6, we need to compute partial derivatives numerically such as u x (˜ x i , ˜ t i ) , u t (˜ x i , ˜ t i ) or u xx (˜ x i , ˜ t i ) . We use the finite difference method , in which the numerical derivative is approximated by finite differences, e.g.,

for some small ∆ x .

In particular, we use the weighted essentially non-oscillating (WENO) scheme as a numerical differentiation method [37], with a careful choice of parameters as in Dumbser & Käser [12]. In the WENO method, multiple estimates for derivatives are made using multiple sets of neighboring gridpoints (called stencils ). The multiple estimates are then averaged with weights (called smoothness indicator ) that approximate how stable the derivative estimates are . We implement the WENO method working on a nonuniform grid, and the model learns symmetries by backpropagating through it. A detailed description on the WENO scheme is in Appendix D.1.

B Experiment Details

In this section, we present the detailed experimental settings of the experiments in § 5.

B.1 Images

Training ResNet-18. When training the ResNet-18 with CIFAR-10, both the feature extractor H fext and models after augmentation, we train the model in 200 epochs with a batch size 128. The learning rate is set to 10 -1 and decreases by a factor of 0 . 2 at the 60th, 120th, and 160th epoch. The model is trained by SGD optimizer with Nesterov momentum 0.9 and weight decay 0.4.

Learning symmetries. To learn the symmetry generators, we train the MLP using two shared hidden layers, each with a width of 256, followed by a hidden layer of width 32 for each output vector field. We use the swish activation function to ensure the learned vector fields are smooth. The MLP is trained for 50 epochs with a batch size 128 and fixed learning rate of 10 -4 using the Adam optimizer. The learning process takes less than 10 hours on a GeForce RTX 2080 Ti GPU.

B.2 PDEs

Data generation. We follow the data generation method of Brandstetter et al. [4]. Given an 1D evolution equation u t = F ( x, t, u, u x , u x xx, · · · ) for u = u ( x, t ) on periodic domain [0 , L ] , we start with an initial condition u ( x, 0) by random Fourier series as

where P is the number of Fourier modes and ( A p , l p , ϕ p ) are random coefficients. For time-stepping, we compute x -derivatives u x , u xx , · · · using pseudospectral method, which computes derivatives in Fourier domain and converts them back to the original domain. We use an ODE solver to compute the time evolution of u ( x, t ) from t = 0 to t = T . The solution is discretized on regular grid of size N x × N t on [0 , L ] × [0 , T ] , where N x = 256 and N t = 140 by default.

When simulating Burgers' equation, we instead solve the PDE for the heat equation ϕ t = ϕ xx for ϕ = νϕ ( x, t ) . The Burgers' equation u ( x, t ) and the heat equation ϕ ( x, t ) is related via the Cole-Hopf transformation:

After data generation, we transform the heat equation back to the Burgers' equation.

Learning symmetries. We use an MLP with the same architecture as described in Appendix B.1. Since all the tuples ( x, t, u ) must pass through the MLP, totaling N x × N t for each data instance, we set a small batch size 4. Symmetries are learned using 1024 data instances over 50 epochs. We use the Adam optimizer and train the network with a learning rate of 10 -4 in the first 25 epochs and 10 -5 for the remaining 25 epochs. The learning process also takes less than 10 hours on a GeForce RTX 2080 Ti GPU.

Sobolev regularization. To ensure smoothness in the vector fields, we apply additional regularization using the Sobolev norm of order 2 in the x -domain. For a vector field V ( x, t ) , the Sobolev norm can be efficiently computed in Fourier domain:

where n freq = min( n, N x -n ) and ˆ V ( · , t ) is the discrete Fourier transformation of V ( · , t ) . Since we already enforce || V || = 1 , we penalize towards the Sobolev norm excluding the zeroth order term:

We apply the Sobolev regularization during the final 10 epochs.

Training FNOs. For the KdV and KS equations, we train an autoregressive FNO solver, which takes 20 timesteps as input and predicts the subsequent 20 timesteps, following the experimental setup of Brandstetter et al. [4]. For the nKdV and cKdV equations, which are time-dependent and contain explicit t terms, we train FNOs as single-time neural operators. These models utilizes the initial conditions of the equations to predict the states after 70 timesteps. We train the FNO over 40 epochs, with each epoch comprising 280 iterations across the dataset for the KdV and KS equation and 100 iterations for the nKdV and cKdV equations. The learning rate begins at 10 -4 and decreases by a factor of 0 . 4 every 10 epochs.

C Experiment Results

In this section, we provide detailed results of the experiments outlined in § 5. For symmetry learning tasks, we conducted each experiment three times and randomly selected one for reporting in § 5. The complete set of results is provided here. Also, we report the detailed evaluation metrics in augmentation tasks.

C.1 Images

Figure 8 is a visualizations of the learned symmetries discussed in § 5.1. Figure 9 displays results from experiments conducted under the same settings but with different model initializations compared to Figure 4 in § 5.1. It is notable that affine symmetries consistently occupy the first six slots across all experiments.

C.2 PDEs

We report experiments results for learning symmetries, conducting three trials for each equation in figure Figure 10. The ground truth symmetries and the approximate symmetries are consistently found in the former slots in all the experiments. The remaining fourth slots occasinally converge to learned symmetries despite the orthonormality loss or converge to some unknown vector fields with high validity scores. The results from the first trials are used in the augmentation experiments.

In Table 7 and Table 8, we provide the augmentation results of FNOs using the learned symmetries, which are illustrated in Figure 6.

(a)

(b)

LearnedGenerators

V2

V8

V10

Transformation of Images

V1

V2

V3

V4

V5

V6

V

V8

Vg

V10

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

TransformationScale

D Technical Details

D.1 Weighted Essentially Non-Oscillating (WENO) Scheme

This section is largely based on Zhang & Shu [37], Dumbser & Käser [12].

WENOscheme in 1D. Consider a smooth function f : X ⊆ R → R , where we only have access to the function on the grid X grid = { x 1 , · · · , x N grid } , x 1 < · · · < x N grid . To estimate the derivatives f ( k ) ( x ) = d k f dx k ( x ) for some x ∈ X , we use k +1 neighboring points in I = { x i , · · · , x i + k } and compute the unique k -th order polynomial p I ( x ) that interpolates the function f on the points in I . In other words, we ensure p I ( x i ) = f ( x i ) , · · · , p I ( x i + k ) = f ( x i + k ) . Then we can pick up the estimates of the derivatives using the polynomial as f ( k ) ( x ) ≈ p ( k ) I ( x ) . We call the set of neighboring points I a stencil , and the polynomial p I the reconstruction polynomial . Usage of reconstruction polynomials is a fundamental concept in the numerical differentiation method known as the finite difference method .

In the weighted essentially non-oscillating (WENO) scheme, we compute multiple estimates p ( k ) I 1 ( x ) , · · · , p ( k ) I Ns ( x ) for the derivatives f ( k ) ( x ) using multiple stencils I 1 , · · · , I N s . The estimates

Affine Components

(b) Third experiment.

(a) Second experiment. Affine Components

InnerProducts

0.00 0.01 0.01 0.00 0.00 0.00 0.00 0.00

(1, 0)-0.00

0.97

-0.01 0.00 0.00 0.04 -0.12 0.02 0.150.02

ators

enerat

2.00

(X1,0)

(0,1)

-0.00

0.97

0.01

000200S10-100-00-00-000-010-000-

-0.20

0.79

-0.00

0.36

-0.15

-0.04

-0.40

0.01

Measure

0.8

1.0

1.2

(0, x2)

0.14

0.00

0.150.00

-0.95

0.07-0.01

-0.180.020.02

0.6

(X2,

0)

0.01

-0.16-0.40-0.38

-0.01

0.76

-0.18

0.04

0.21

0.08

0.4

(0,X1)-0.07-0.00

0.81

0.45

-0.03 -0.080.02

-0.29

0.2

V1V2V3V4VsV6V7V8VgV10

0.0

V1 V2 V3 V4 Vs V6 V7. V8 Vg V10

VV2V3VaVsV6VVsVgV10

Learned Generators

LearnedGenerators

Learned Generators

InnerProducts

0.01

0.00 -0.00 0.00 0.00 -0.00 0.00

(1,0)-0.00

0.97

-0.030.01-0.040.03-0.010.070.03-0.14

Generators

V2

(0,1)

-0.98

-0.00 -0.11 -0.02 -0.01 -0.03 0.050.13-0.03 0.06

1.0

1.2

(x1, 0)0

0.00

0.01-0.090.07

0.98

-0.160.01 0.01 0.040.01

Measure

0.8

V6

(0,x2)

0.13

0.00-0.01

-0.93

0.100.250.15 0.10-0.01 0.06

0.6

ed

(x2,0)

0.00

-0.12

-0.39

0.220.10

0.84

-0.070.07-0.14-0.18

0.4

(0,X1)-0.07-0.00

0.87

0.100.140.370.12-0.080.10-0.17

0.2

Viol

0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00

VV2V3V4VsV6VV8V9V10

0.0

V1 V2 V3 V4 Vs V6 V7 V8 Vg V10

VV2V3VaVsV6VVVgV10

Learned Generators

Learned Generators

Learned Generators

are then averaged with weights ω 1 , · · · , ω N s which take account of the quality of each estimation. In the WENO scheme, we assume the grid points are sufficiently dense, hence the estimates are more accurate when the reconstruction polynomials p I m are smooth. The smoothness of p I m is measured by the smoothness indicator , defined as:

where ∆ = ( x j , x j +1 ) ⊂ R is an interval between two grid points containing x , and | ∆ | is the length of ∆ . The weights ω 1 , · · · , ω N s are defined as:

where γ m is called the linear weight , usually chosen heuristically, b is a positive integer usually set to 2 or 4, and ϵ > 0 is a small positive number preventing the denominator from being zero. The final estimate for the derivative f ( k ) ( x ) is computed by averaging the estimates p ( k ) I 1 ( x ) , · · · , p ( k ) I Ns ( x ) with the weights ω 1 , · · · , ω N s :

WENOscheme for multivariate function. The WENO scheme extends smoothly to multidimensional function f : R n → R . For k = ( k 1 , · · · , k n ) ∈ Z n ≥ 0 , we denote f ( k ) ( x ) = ∂ k f ∂x k 1 1 ··· ∂x kn n ( x ) . On a stencil I with an appropriate number of grid points, we compute the reconstruction polynomial p I ( x ) with a nonzero k -degree term, and the smoothness indicator is defined similarly as:

(a) First Experiment

(b) Second Experiment

(c) Third Experiment

KdV

KS

Burgers

Components

1.0

X-tsl1-0.06

-1.00

-0.07

0.99

Components

1.0

X-tSl-0.33

0.91

0.270.00

1.0

X-tSl-0.59 0.810.04 0.00

0.5-

t-tsl

1.00

0.06 0.04 0.05

0.5

t-tsl-

0.94

0.32 0.09 0.01

t-tsl

gal-0.00-0.00.01

-0.80-0.590.11 0.00

-1.00

0.0

VV2V3 V4

gal-0.03-0.07

VV2V3V4

1.00

-0.15

0.0

V4

gal-0.01 0.28

VV2V3V4

-0.96

0.01

U-SCI-0.110.04

VV2V3V4

0.99

-0.00

nKdV

cKdV

1.0

X-tsl

-1.00

-0.06 0.02 -0.00

1.0

x-tsl

0.92

0.28-0.270.02

g2-0.06

-1.00

0.03 0.02

92

0.21-0.22

-0.95

00

g30.02 -0.03

-1.00

0.01

t-tsl-0.33

60

-0.14 -0.08

V1

V2

V3

V4

VV2V3 V4

V1

V2

V3

V4

VV2V3V4

KdV

KS

Burgers

Components

1.0

x-tsl

-0.62-0.78

-0.010.62

1.0

X-tsl-0.20

-0.98

-0.09-0.09

1.0

X-tSI-0.53-0.700.480.00

.5-

t-tsl

0.78

3-0.62-0.04

-0.78

t-tsl

-0.98-0.20-0.02

1.00

t-tsl

-0.78-0.190.60

0.00

-1.00

0.0

gal-0.03 0.04

-1.00

-0.07

1.00

0.01

U-SCl-0.33-0.69-0.64

-0.00

V1

1V2V3V4

VV2V3 V4

VV2

V3V4

VV2V3V4

VV2V3V4

VV2V3V4

nKdV

cKdv

1.0

x-tsl

0.99

-0.13 0.01 -0.01

1.0

x-tsl

0.93

-0.12 0.35

0.00

g2

0.13

0.99

-0.01 0.03

92

-0.12

0.99

-0.03 0.00

930.01-0.01

-1.00

0.03

0.0

t-tsl

-0.35

0.01

0.94

0.04

VV2V3V4

VV2V3V4

VV2V3V4

VV2V3V4

KdV

KS

Burgers

1.0

X-tSI-0.38-0.92-

0.02 0.00

1.0

X-tS1-0.28

0.95

-0.090.00

1.0

X-tSI-0.460.28-0.84

0.00

0.5-

t-tsl

-0.92

20.380.07-0.01

0.5-

t-tsl

0.96

-0.28 0.02 -0.01

t-tsl

-0.88

-0.060.47

-0.00

gal-0.05-0.05

1.00

0.02

gal-0.01 0.10

1.00

0.01

U-SCI-0.080.960.280.00

VV2V3V4

VV2V3V4

VV2V3V4

nKdV

cKdV

1.0

x-tsl

0.97

-0.23 0.01 -0.00

Components

1.0

x-tsl

0.97

-0.20-0.05-0.00

92

0.23

0.97

-0.02 0.01

0.5

921

0.05 0.02

1.00

0.00

g30.00-0.02-0

-0.73

-0.02

t-tsl

-0.23

0.98

0.000.02

VV2V3V4

V3V4

VV2V3V4

where α = ( α 1 , · · · , α n ) ∈ Z n ≥ 0 , | α | = ∑ n i =1 α i and the interval ∆ is now a n -dimensional cell containing the point x with volume | ∆ | . Like the 1D case, the weights are defined using the linear weights and smoothness indicator.

Choice of parameters. We follow the choice of parameters of Dumbser & Käser [12]. The linear weight γ m is chosen to be 100 if x is inside the center cell of I m and 1 otherwise, reflecting the fact that the estimates are accurate as much as the point x is close to the center of I m . The parameters b and ϵ are chosen as b = 4 and ϵ = 10 -6 .

D.2 Whittaker-Shannon Interpolation on a Periodic Domain

Let · · · , f [ -1] , f [0] , f [1] , · · · be a discretization of a continuous signal on a real line. The WhittakerShannon interpolation recovers the original signal f as:

where sinc is the normalized sinc function sinc( t ) = sin( πx ) πx . The Nyquist-Shannon sampling theorem states that states that f ( t ) is the perfect reconstruction of f , in a sense that if the original function does not contain any frequencies higher than a certain threshold, called the Nyquist frequency , then f is perfectly reconstructs the original signal.

Similarly, if f [0] , f [1] , · · · , f [ N -1] is a discretization of a continuous signal on a periodic domain [0 , N ] for some even positive integer N , the Whittaker-Shannon interpolation is given as:

where D N ( t ) = ∑ ∞ n = -∞ sinc( t -mN ) is the Dirichlet kernel with period N . The Dirichlet kernel D N ( t ) has a closed-form expression

D.3 Nonlinear Time Transformation of the KdV Equation

We apply a nonlinear time transformation t defined as t = t 0 ( e ˆ t t 0 -1) , where t 0 is a constant scaling factor, to the KdV equation u t + uu x + u xxx . The derivative with respect to t is transformed to

hence the KdV equation with nonlinear time transformation (nKdV) becomes

The three symmetries of the KdV changes accordingly. The space translation is left untouched, and the other two infinitesimal generators are tranformed as:

where we used the derivative notation (1 , 0 , 0) = ∂ ∂x , (0 , 1 , 0) = ∂ ∂t and (0 , 0 , 1) = ∂ ∂u .

E Ablation Studies

E.1 Numerical differentiation.

When searching for symmetries of PDEs, we use the WENO scheme as a numerical differentiation method. We have found that the choice of numerical differentiator is critical, as gradients must flow through numerical differntiation during the backpropagation step. To compare different methods, we experimented with using the WENO scheme and another method: bilinear interpolation followed by discrete numerical differentiation on a regular rectangular grid.. When using bilinear interpolation followed by discrete differentiation, the loss failed to converge well, even with very small learning rate 10 -6 . The learned vector fields were not orthogonal, and they only spanned the time translation and the space translation at best, as in Figure 11. These experiments were conducted using the KS equation.

We hypothesize that the bilinear interpolation distributes each transformed point into two adjacent grid points, but discrete differentiation is done by repeatedly subtracting values of two adjacent grid points, so the gradient may not flow well during the backpropagation.

E.2 Whittaker-Shannon interpolation.

In the PDE augmentation task, when the transformed PDEs are resampled into the regular rectangular grid, we use Whittaker-Shannon instead of more commonly used bilinear interpolation. This step is also crucial for training FNOs with transformed data, since FNOs are extremely sensitive to numerical error and hence any aliasing effects must be avoided. We compared the augmentation results using Whittaker-Shannon interpolation with those using bilinear interpolation, using the KS equation with 512 training data and report the result in Table 9. The results clearly demonstrate that bilinear interpolation adversely affects the training FNO models.

(a)

(b)

Learning Curve

Bilinear Int + Discrete

eDiff

2.8

5

S

S

2.6

2.4

2.2

n

Components

1.0

x-tsl

-0.70-0.660.65

-0.60

V1

V2

0.831.00-0.990.05

1.000.83-0.80-0.14

2.0

0.5

t-tsl

0.70-0.440.390.80

V3

0.80-0.991.00-0.08

1.8

gal

0.08-0.540.55-0.01

V4

0

1020304050

Epochs

0.0

V1 V2 V3 V4

V1V2V3V4

-0.140.05-0.08 1.00

V1V2V3V4

F Hyperparameter analysis

In this section, we analyze the roles of various hyperparameters in our learning scheme. We use experiment setting of the KS equation in § 5.2. We selected the KS equation for analysis as it proved to be the most challenging in learning symmetries, possibly due to its fourth-order derivative term u xxxx . In particular, among the three symmetries (space translation, time translation, Galilean boost) the algorithm easily learns the space and time translation in a few epochs, but learning the Galilean boost takes more epochs as in Figure 12. We evaluate the results by examining whether the three symmetries are correctly found in the first three slots.

LearningCurveofEachGenerator

-1.25

-1.50-

V1

-1.75

V2

-2.00

V3

S

S

-2.25

V4

5

-2.50

-2.75

-3.00

-3.25

0

10

20

30

40

50

Epochs

F.1 Lipschitz Threshold

The Lipschitz threshold τ in Equation 13 is the only parameter that constrains the function space in which we search for the symmetries. The threshold τ is set to τ = 3 in the main experiments. We conduct additional experiments with τ = 1 , 2 , 4 , 8 , 16 , 32 . Surprisingly, in all experiments, regardless of the value of τ , we found the three ground truth symmetries.

F.2 Transformation Scale

The transformation scale σ controls how much we transform the data when searching for symmetries. The tuple ( x, t, u ) in X ×U is scaled so that x and t form a uniform grid on [0 , 1] 2 , and u is scaled so that the standard deviation of u closely matches that of x and t , which is approximately 0 . 29 . Hence, the default transformation scale σ = 0 . 4 transforms data with slightly more than its standard deviation at most. We conduct experiments with various transformation scales σ = 0 . 1 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 . The ground truth symmetries were found when σ = 0 . 1 , 0 . 2 , 0 . 4 . When σ = 0 . 6 , the Galilean boost was found but allocated in the fourth slot, meaning that the learning was unstable. When σ = 0 . 8 , the model only found space and time translation correctly.

t=1.0

t=2.0

Components

1.0

VV2V3V4

X-tSI-0.180.980.040.10

t-tsl

gal-0.000.81.0-0.0

VV2V3V4

0.98-0.170.010.09

V1

V4

0.7-0.10.01.00

0.010.041.00.00

-0.00

VV2V3V4

0.000.010.07

1.000.04-0.11

Components

0.5-

1.0

0.0

VV2V3V4

X-tSI-0.120.90.020.05

t-tsl

ga-0.0.00

VV2V3V4

0.990.120.01-0.01

V

V1

V4

0.010.050.001.00

+0.000.001

VV2V3V4

.00

0.000.000.01

1.00

00.00

t=4.0

t=8.0

Components

1.0

x-tsl

-0.240.950.220.01

V1

1.00

0-0.000.010.09

Components

1.0

x-tsl

-0.55-0.80-0.260.00

V1

1.000.00-0.01-0.02

0.5-

t-tsl

0.970.250.02-0.09

0.010.51.0-.05

00.05 0.00

0.5

t-tsl

0.830.550.08-0.02

-0.00

-0.010.011.00-0.03

1.00

00.01-0.01

gal-.020.1-0.970.08

V4

0.090.00.051.00

0.0

V41

0.02-0.01-0.031.00

VV2V3V4

V1V2V3V4

VV2V3V4

VV2V3V4

V1V2V3V4

VV2V3V4

t= 16.0

t=32.0

Components

1.0

0.5

X-tSI1-0.150.90.01-0.04

t-tsl

0.99-0.150.000.03

V1

0.000.021.-000

-0.00

1.000.00-0.000.03

1.00

-0.02-0.04

Components

0.5

1.0

x-tsl

t-tS1-0.36-.930.0.06

-0.93-0.360.03-0.05

V1

0.001.0

0.01-0.001.000.00

1.00-0.000.010.07

0.000.04

0.0

VV2V3V4

V4

-0.03-0.040.001.00

0.0

gal-0.030.01,1.00.0

VV2V3V4

V41

-0.07-0.040.00

1.00

VV2V3V4

VV2V3V4

VV2V3V4

VV2V3V4

0=0.1

α=0.2

Components

0.5

1.0

0.0

VV2V3V4

x-tsl

t-tsl

gal -0.040.2-.970.0

0.420.870.23-0.00

VV2V3V4

-0.900.420.070.06

V2

V3

V4

V1

-0.050.020.011

-0.01-0.021

0.001.00-0.020.02

VV2V3V4

00

-0.000.01 0.05

.00

11.00

0.01

Components

0.5

1.0

0.0

VV2V3V4

X-tS1-0.42-0.880.230.03

t-tsl

gal-0.01-0.,0.970.0

V1V2V3V4

0.910.41-0.08-0.75

V41

V2

V3

V1

VV2V3V4

-0.001

-0.01-0.031

0.68-0.350.01

.00

0.000.01-0.68

1.00

0-0.030.35

1.00

0.01

1.00

0=0.4

0=0.6

Components

1.0

0.0

x-tsl

t-tsl

-0.330.900.300.00

VV2V3V4

0.940.320.08 0.08

50.01

V2

V1

V3

V4

0.080.0-0.001.00

0.00

0.010.031

.00

00.00-0.010.08

1.00

0000000

1.00-0.00

Components

0.5

1.0

0.0

X-tS1-0.080.90.0-0.05

t-tsl

gal-0.00.070.09.99

VV2V3V4

1.00-0.080.030.00

V2

V3

V4

V1

0.000.030.031.00

+0.001.00-

0.03-0.021

1.00

0-0.000.030.00

0-0.02-0.03

0-0.03

VV2V3V4

VV2V3V4

VV2V3V4

VV2V3V4

α=0.8

Components

X-tS1-0.120.90.00.0

1.00-0.000.010.83

0.5

t-tsl

0.990.120.00-0

0.82

40.001

0.01-0.041.00-0.01

1.00-0.040.00

gal-.010.07-0.460.30

0.0

V1V2V3V4

V1V2V3V4

O.83-0.000.01

VV2V3V4

.00

F.3 Weights of Three Losses

Although the three loss terms Equation 14 are designed to have dimensionless scales, tuning the three loss terms in appropriate ranges is inevitable.We perform a grid search over the three weights w sym , w ortho , w Lipschitz of the total loss in a fixed grid [1 , 3 , 10] 3 . We report the number of correctly learned symmetries across different values of w sym, w Lips, and w ortho in Table 10. The number 3 is the maximum number of symmetries to be discovered. We found that the ratio between w sym and w ortho significantly affects the results. In this case, the weight w ortho should be larger than w sym to prevent the learning of redundant infinitesimal generators.

G Color-space Results

Recall that in our formulation, images take the form X → Y where X = [ -1 , 1] 2 is the pixel space and Y ∈ R 3 is the RGB color space. Our formulation is not limited to searching for the symmetries on the 2D plane X , but is capable of learning symmetries in the color space Y . In this section, we elaborate how we modeled learning scheme for color-space symmetries and present the results.

Transformation of Images: V3

G.1 Learning Scheme

Similar to other cases, we model one-parameter groups on Y by a neural ODE, integrating over 3D vector fields Y → R 3 parametrized by a neural network. Colors are non-uniform - for example, black, brown and gray appear much more frequently than pink or yellow in CIFAR-10 dataset. To address this, we first form a grid Y grid on the color space Y by randomly sampling 1024 colors from the CIFAR-10 dataset. Then we define the weight function w : Y → R , like the weight function in Equation 17, by the color sensitivity of the neural network H fext. For a grid point y i ∈ Y grid , the weight ω ( y i ) is defined as:

To estimate the magnitude of the gradient with respect to the change of y i , we use neighboring points y j ∈ nbhd( y i ) ⊂ Y grid. For a discretized image { x i , f ( x i ) } x i ∈X grid , we use the nearest neighbor

algorithm to collect f ( x i ) ∈ Nearest( y i ) close to y i , and measure how much H fext changes when the color f ( x i ) shifts towards the direction y j -y i :

✶ Once the weight function is computed, we learn symmetries by optimizing through the validity score S , defined by cosine similarity of features of the learned neural network. Since there are ( batch size ) · ( number of pixels ) number of colors in a single batch of images, it's impractical to feed all of them directly into the MLP. Instead, we use the grid Y grid, and compute the vector field values on the grid and interpolate them using the nearest neighbor algorithm.

where ✶ Nearest( y i ) is a vector whose i -th entry is 1 if f ( x i ) ∈ Nearest( y i ) and 0 otherwise, and ∇ Nearest( y i ) H fext ( f ) is computed by Jacobian-vector product.

G.2 Results

We open three slots for color-space symmetries and train the model with w sym , w Lips = 1 and w ortho = 3 . We found the brightness control, color contrast and blue-yellow shift respectively as in Figure 15.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clarified our contribution in the last two sentences in the abstract and the last paragraph in the introduction. Both sections mentioned that our scope is learning continuous symmetry with minimal prior knowledge.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We mentioned our limitations in the last two sentences in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Although we did not provide any theorems or propositions, which requires proofs, all formulas are numbered and cross-referenced.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We checked the reproducibility of the reported experiments and we provided the hyperparameter analysis in the paper and we shared our source code. Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We attached our source code and we used public datasets, which are freely accessed by anyone.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All hyperparameters used in the experiments are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We measured the performance with 3-5 repeats. In Table 1 and Table 4, we reported the standard deviation, and in Figure 6(B), we described it using the error bar.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We also mentioned such information in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We checked the code of ethics and our paper does not cause any hazard because it only explores theoretical symmetries.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Since we explore only theoretical symmetries present in mathematical space, our work is far from the societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not have such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly credited when adopting the existing codes and properly cited the original paper that produced the code and dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We shared our code with anoymization and documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve any potential risks incurred by study participants

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using neural ODEs to parameterize symmetries by viewing the ODEs flow as an element of a one-parameter group. They show that by learning the parameters of the neural ODEs, they are able to recover ground truth symmetries in image classification and PDE tasks.\n\nSTRENGTHS:\nThe paper is easy to read. The proposed ideas are clear, and appear to be mostly novel.\n\nWEAKNESSES:\nSee questions below.\n\nQUESTIONS:\n1. Line 174: Do you also need \"and for all $s\\in[-\\sigma,\\sigma]$\" in addition to the \"for all $f\\in\\mathcal{D}$\"?\n2. Section 4.1: According to your definition of a symmetry $\\vartheta$ (there exists $\\sigma>0$ such that $\\vartheta_s(f)$ is \"valid\" for all $f\\in\\mathcal{D}$ and all $s\\in[-\\sigma,\\sigma]$), it seems like solving for $\\vartheta^*$ from (4) does not guarantee that $\\vartheta^*$ is actually a symmetry, even if the optimal value of (4) is less than $C$. That is, if the optimal value of (4) is less than $C$, then you only know that the validity score is less than the threshold on average, not for all data $f$ and all transformation scales $s$. Can you please clarify this aspect?\n3. Line 217: Something is strange with the definition of $\\vartheta_{\\mathcal{X}}(f)$. In particular, what is $T_{\\mathcal{X}}$, and why does $f$ not appear in the right-hand expression? It seems to me like perhaps you meant to define $\\vartheta_{\\mathcal{X}}(f)(x) = f(\\vartheta_{\\mathcal{X}}^{-1}(x))$, and hence that you need to assume the transformation $\\vartheta_{\\mathcal{X}}$ on $\\mathcal{X}$ to be invertible. Can you please clarify whether this is a typo, or if I am missing something? Also, it is unclear what $\\vartheta_{\\mathcal{Y}}(f)$ is doing or why it is needed. Indeed, as written, you are defining two different transformations of $f$ in (8) (resulting in two different functions of $x$).\n4. Line 226: How reasonable is it to assume that you know the actual number of possible symmetries for a given task? Is it possible for you to select $N_{\\textup{sym}}$ to be too small, and then to miss out on learning some of the most important symmetries of a task?\n5. Line 232: Typo (\"orthonomral\").\n6. Line 230: In practice, are symmetries actually mutually orthogonal (as vector fields) to one another? It seems like this is not always the case per your Table 1, which shows that uniform scaling is not orthogonal to $x_1$-axis scaling. A comment on the validity of the orthogonality assumption underlying the use of this regularizer would be appreciated.\n7. Line 243: Typo (\"undersirable\").\n8. Line 246: Typo (\"Lipshictz\").\n9. Line 280: \"...the two translations are the most invariant one-parameter group...\" Can you please explain in further detail how you come to this conclusion?\n10. Table 2 and Figure 5: I suggest moving these to be centered with the rest of the text; it looks poorly placed in the right-hand margin as is, and causes Figure 5 values to be too small and hard to read.\n11. Line 322: \"We found the ground truth symmetries in all the experiments as in Figure 5...\" It looks like you did not recover the true symmetries in the cKdV setting, as there are non-negligible components coming from each of the ground truth symmetries that appear in the learned symmetries.\n12. Did you verify in your experiments that the learned \"symmetries\" are actually symmetries according to your definition using the validity score threshold? Overall, your experimental results in Figures 4, 5, and 10 seem to me like you are indeed learning orthogonal vector fields, and that sometimes the vector fields end up being equal to one of the ground truth vector fields, but other times not (turns out to be linear combination of multiple ground truths). This does not seem super convincing that you are reliably learning/recovering the ground truth symmetries.\n13. Why did you introduce (8)? I don't see this transformation being used anywhere else in the paper.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper pertains to the topic of data-driven symmetry discovery. The authors propose a method allowing symmetry discovery beyond pre-defined Lie groups, by learning to transform datapoints, potentially in a non-affine manner, via a learned ODE (referred to as the *one-parameter group*, where the single parameter is the time variable in the ODE), the velocity field of which is typically parametrised by an MLP. \n\nCrucially, to optimise this, the authors choose an objective - *validity score*, which is a predefined measure of the extent to which a transformed datapoint is symmetric to the input one (in their examples: for images, they use the cosine similarity between features extracted by a pretrained NN, while for PDEs, they measure the value – error – of the PDE for the transformed datapoint). Additional regularisers are used to ensure the diversity of the symmetries learned (orthogonality between different learned velocity fields) and smoothness (minimisation of an estimate of their local Lipschitz constants). Experimentally, the method is tested on image classification (CIFAR10) and PDE solving (KdV, KS, Burger’s equations) showing that known symmetries are retrieved along with additional approximate symmetries, while the learned symmetries are subsequently used for data augmentation showing competitive results to methods using pre-defined augmentations.\n\nSTRENGTHS:\n**Significance** . The paper studies an important problem (*data-driven symmetry discovery*) in machine learning, but also physical sciences where symmetries are abundant but potentially unknown. Identifying unknown symmetries and incorporating them in downstream ML models (e.g. via augmentation) can improve generalisation, especially in low-data regimes, while additionally, it can potentially provide novel insights about the task at hand.\n\n**Novelty/Generality**\n- The presented methodology has the capacity to recover symmetries arising from *non-affine* data transformations. This is contrary to prior work, where mostly linear/affine transformations are dealt with.\n- Additionally, this method does not require making assumptions about the structure of the target group. This is common in prior art, where typically a subgroup of a predefined group is learnt.\n- The authors take advantage of well-established concepts that are underexplored by the ML community (e.g. modelling transformations via the one-parameter group) - this helps to broaden the available toolbox in the field of ML & symmetries/ equivariant ML.\n\n**Execution/Implementation**\n- Although the proposed method has multiple complicated components (NeuralODEs, difficult objective to optimise for), it is nonetheless well-executed yielding competitive results and recovering known symmetries in popular testbeds.\n\nWEAKNESSES:\n**Applicability and scope**. Perhaps the biggest limitation of the proposed method is the *reliance on the validity score*. Although the authors claim to be able to learn symmetries by making as few assumptions as possible (see strengths), this seems to be contradicted by the need to manually design a validity score. Moreover, I have the impression that the validity score is not merely a hyperparameter, but it is decisive for the symmetries that will be learnt (basically it is the objective function of the optimisation problem). \n-  For example, in the case of images, the choice seems ad hoc (closeness in the representation space of a pre-trained encoder NN). What leads the authors to believe that the features of equivalent (symmetric) images extracted from the pre-trained NN should be close? Have the authors tried to verify this assumption? I think the empirical validation is insufficient here (section 5.1.), so I am not entirely convinced.\n- In general, I do not see a generic way to define validity scores and perhaps the authors have slightly overclaimed in that respect. I would like to read the authors' viewpoints on that. For PDEs, the validity scores are indeed reasonable and generic, so perhaps, they would like to put more emphasis on this perspective.\n\nFurthermore, the authors introduce the concept of learning symmetries via the one-parameter group, claiming that it is more general than prior parametrisations that can only learn linear/affine groups. However, it is unclear what the present parameterisation can express, e.g. does it allow learning any continuous group or implicit assumptions are made here as well? \n- Additionally, could the authors discuss if it would be possible to learn finite groups with this method as well and if not, how could those be incorporated?\n\n\n**Related Work/Comparisons**. The work of Forestano et al., MLST’2023 is quite relevant to the present manuscript, with the main difference being that in that work, the transformations are parameterised by an MLP instead of a NeuralODE (the oracle used in this work seems similar to the validity score used here). Since the two works have many similarities, I think that the authors should discuss in more detail their differences and the advantages of their work (e.g. as far as I understand the MLP cannot guarantee that the transformations form a group). Note that modelling transformation via an MLP (or any NN in general) instead of a NeuralODE seems more straightforward and probably easier to train and more computationally friendly.\n\n**Experiments**. I believe some additional empirical evidence would strengthen the authors' claims.\n- Most importantly, an experimental comparison against the type of parameterisation used in Forestano et al. (MLP) should be provided, to verify if NeuralODEs are indeed a more appropriate parameterisation.\n- Moreover, baselines are mostly missing, e.g. comparing against other methods for data-driven symmetry discovery (I am not super familiar with these works, but if I am not mistaken LieGAN by Yang et al., ICML’23 is a recent example). \n- The reported results after augmenting with the learned symmetries do not seem to improve significantly compared to known/default augmentations. Can the authors discuss why this might be the case? This is important since it might undermine the necessity of the proposed approach.  To be more convincing, perhaps the authors should perform experiments on problems where the symmetries are not known a priori.\n- Additionally, ablation studies seem to provide important insights but are only discussed in the appendix. I would recommend being more upfront in the main paper and discussing in the rebuttal the following: sensitivity to hyperparameters (multiple are needed: loss coefficients, $\\sigma$ and $\\tau$), the method for choosing them, the difficulty of optimisation  (3 terms are used in the objective) and if all losses are optimised in a balanced manner. Similarly, for the parameter $N_sym$, which is now chosen based on prior knowledge of the number of existing symmetries.\n\n**Presentation/Exposition**. (disclaimer - this is a minor weakness) Given that the notions discussed here are probably not widely known across the ML community, I believe that the authors should aim to provide more in-depth explanations to make their work more accessible. For example,\n- Multiple group theory/symmetry concepts are discussed without definitions (group generators, Lie group, Lie algebra, Lie bracket etc.). Additional examples that are not well-contextualised include in section 2 the one-parameter group discussion, the Lie algebra of the affine group and the discussion on the PDE symmetries (Lie point symmetries etc.). Adding some references here and providing some examples for the mentioned PDE symmetries would help.\n- In section 5.2., some concepts regarding the experimental details are mentioned without appropriate explanations, while others are only mentioned in the appendix, although it appears that they are crucial for the method. Perhaps the authors should be more upfront and explanatory regarding the aforementioned.\n\nQUESTIONS:\n- How is the weight function in L234 defined, and how important is this for optimisation? Are different weight functions ablated?\n- It’s unclear why the stop-gradient is needed in L237. I did not find the justification fully convincing. Could the authors elaborate? What happens experimentally if one does not use stop-gradient?\n- Although intuitive, it’s unclear why the inline Eq. in L212 holds for negative $\\alpha$.\n\n**Suggestion**.\nIn case the authors do want to present their method as generic, I think a deeper experimental evaluation in data beyond PDEs would help a lot (e.g. testing on other image datasets, ablating different validity scores etc).\n\n**Minor**:\n- What if the chosen validity score is not differentiable?\n- *Notation*.\n  - The notation $\\theta^V_s(x)$ is a bit dense (why use V as a superscript?). Eq (1) is a bit confusing. The inline Eq in line 83 sees clearer to me. \n  - Notation in sec. 4 could be also simplified a bit or made more accessible with examples (e.g. 4.1: give an example for $\\mathcal{A}$, i.e. the set of all natural images).\n\nLIMITATIONS:\nSome of the limitations are adequately discussed. An important missing point is, in my opinion, the need to manually design the validity score in domains beyond PDEs.\n\nNo foreseeable negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a symmetry learning algorithm based on transformations defined via infinitesimal generators. Using Neural ODE, an infinitesimal generator is learned that is capable of producing a sequence of transformed data through ODE integration. Validity score has been defined to check if the transformed data is valid wrt to a given task. For images, the validity score is picked to be cosine similarity while for PDEs the validity score is defined as the numerical errors in the original equations after the transformation. In addition to symmetry loss, two regularizations, orthonormality loss, and Lipschitz loss have been added, to remove trivial solutions. The authors present experiments on CIFAR10 and KdV equation and Burgers' equation in 1D for PDE.\n\nSTRENGTHS:\n1. The paper motivates the need for learning continuous symmetries well. \n2. The idea presented in the paper is very neat and shows potential beyond the presented experiments. \n3. This approach can learn both affine and non-affine symmetries in image classification and PDE tasks as shown in the experiments section.\n\nWEAKNESSES:\nThe discussion on compute and model parameters comparisons with baseline missing. No other method was shown as a baseline in either of the experiments. There is some ambiguity in how exactly the validity score is used and in some cases, can be defined if the given task is equivariant.\n\nQUESTIONS:\n1. How does the validity score take into consideration, invariant tasks vs equivariant tasks? \n2. Using cosine similarity between the extracted features for validity score, can be a problem for extreme transformations (as cosine values do not linearly change with parameter). How does this affect the learning of symmetry build on validity scores?\n3. In Table 2, it is unclear how the validity score players out in learning the symmetries. Especially for default augmentation case. Could you please elaborate on this?\n4. Total loss and loss-scale section: How is the true scale learned, if the inner product is normalized? Additionally if all the terms are affected by a data-dependent scale, shouldn't $w_{Lips}$ affect the overall weights?\n\n### Clarifications\n1. The figures can be made a little bigger. \n2.  In line 16, 'whether the transformed data are still valid for the given task'. This phrasing is confusing.\n3. In line 172, what does valid mean? simply put; does it comprise invariant transformations and approximately equivariant up to the threshold C?\n4. Intuition on how to locate transformed data on grid is missing.\n\nLIMITATIONS:\n1. The authors do not compare with other methods of learning symmetry, as those methods use datasets that have explicit symmetry (like Rot-MNIST). Similarly for that of the PDE experiments. It would have been useful to have them in the paper.\n2. The number of parameters and compute comparison across the proposed method and baseline approach like [9,14] is missing.  Also discussion the scale of numerical computation with an increase in dimension missing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}