Learning Infinitesimal Generators of Continuous Symmetries from Data
Authors: Gyeonghoon Ko
Hyunsu Kim
Juho Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Symmetry Discovery
Geometric Deep Learning
Lie Point Symmetry
Neural PDE Solver
Primary area: machine_learning_for_physical_sciences

Abstract: Exploiting symmetry inherent in data can significantly improve the sample efficiency of a learning procedure and the generalization of learned models. When data clearly reveals underlying symmetry, leveraging this symmetry can naturally inform the design of model architectures or learning strategies. Yet, in numerous real-world scenarios, identifying the specific symmetry within a given data distribution often proves ambiguous. To tackle this, some existing works learn symmetry in a data-driven manner, parameterizing and learning expected symmetry through data. However, these methods often rely on explicit knowledge, such as pre-defined Lie groups, which are typically restricted to linear or affine transformations. In this paper, we propose a novel symmetry learning algorithm based on transformations defined with one-parameter groups, continuously parameterized transformations flowing along the directions of vector fields called infinitesimal generators. Our method is built upon minimal inductive biases, encompassing not only commonly utilized symmetries rooted in Lie groups but also extending to symmetries derived from nonlinear generators. To learn these symmetries, we introduce a notion of a validity score that examine whether the transformed data is still valid for the given task. The validity score is designed to be fully differentiable and easily computable, enabling effective searches for transformations that achieve symmetries innate to the data. We apply our method mainly in two domains: image data and partial differential equations, and demonstrate its advantages. Our codes are available at \url{https://github.com/kogyeonghoon/learning-symmetry-from-scratch.git}.

Learning Infinitesimal Generators of Continuous Symmetries from Data
Gyeonghoon Ko, Hyunsu Kim, Juho Lee Kim Jaechul Graduate School of AI
KAIST
Seoul, South Korea
(kog, kim. him.hyunsu, juholee) ac. kr
Abstract
Exploiting symmetry inherent in data can significantly improve the sample effi- ciency of learning procedure and the generalization of learned models. When data clearly reveals underlying symmetry, leveraging this symmetry can naturally inform the design of model architectures or learning strategies. Yet, in numerous real-world scenarios, identifying the specific symmetry within given data distribu- tion often proves ambiguous. To tackle this, some existing works learn symmetry in a data-driven manner, parameterizing and learning expected symmetry through data. However, these methods often rely explicit knowledge, such pre-defined Lie groups, which are typically restricted to linear or affine transformations. In this paper, we propose novel symmetry learning algorithm based on transformations defined with one-parameter groups, continuously parameterized transformations flowing along the directions of vector fields called infinitesimal generators. Our method is built upon minimal inductive biases, encompassing not only commonly utilized symmetries rooted in Lie groups but also extending to symmetries derived from nonlinear generators. To learn these symmetries, we introduce a notion of a validity score that examine whether the transformed data is still valid for the given task. The validity score is designed to be fully differentiable and easily computable, enabling effective searches for transformations that achieve symmetries innate the data. We apply our method mainly in two domains: image data and partial differ- ential equations, and demonstrate its advantages. Our codes are available https: //g trub..u.... com/ /tt////gggggoo////////// learnningeeannnnee sorrgssymmmyyyrr from scratch. git.
Introduction
Symmetry is fundamental in many scientific disciplines, crucial for understanding the structure and dynamics of physical systems, datasets, and mathematical models. The ability uncover and leverage symmetries has become increasingly important in machine learning and scientific research due its potential to improve model efficiency, generalization, and interpretability. By capturing inherent symmetrical properties, models can learn more compact and informative representations, leading to improved performance in tasks like supervised learning [31, 29, 4, 7, 33), self-supervised learning [8, 15, 23], and generative models [19, 11, 18].
Previous methods for learning symmetry have often relied the explicit parameterization of group representations based on predefined generators, which can be limited in capturing various symmetries, including transformations that do not align along the generators. For example, when searching for Lie group symmetries in images or physics data, existing methods [3. 34] parameterize group action as the matrix exponential linear combination of linear or affine Lie algebra generators L; with their learnable coefficients w; as = exp (E; w.L.). In the affine transformations of images in
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(21, m2) (21, coorddinates, there are six generators, each corresponding to translation, scaling, and shearing operations with respect the 21 -axis and øz-axis. Although there exist some methods that directly learn the generators, they are either bound to the general linear group GT.,n), which cannot account for non-affine or non-linear transformations [24], or are not guaranteed find the correct symmetry in real-world image datasets [9, 13].
When searching for symmetries in aighdiimensionl real-world datasets, we can take advantage of the fact that the data can be interpreted as a function f X' + V' such as images, which are functions from the 2D Euclidean space to the color space. Another notable example of such data is partial differential equations (PDEs), where the data take the form u X' + ll and the Lie symmetries are defined as transformations on the space X' U. There have been significant advances in Lie symmetry analysis in recent years, for both academic and industrial purposes, mostly involving extensive symbolic calculations and relying on computer algebra systems [26]. Discovering Lie symmetries of PDEs from data without prior knowledge is an unexplored topic, except for the work of Gabel et al. [14], which learns the symmetry generators of various PDEs in a supervised learning setup.
In this work, we propose a novel method for learning continuous symmetries, including non-affine transformations, from data without prior knowledge. By modeling one-parameter groups using Neural Ordinary Differential Equation (Neural ODE) [6], we establish learnable infinitesimal generator capable of producing a sequence of transformed data through ODE integration. We design an appropriate validity score function that measures how much the transformation violates the invariance to certain criteria defined depending on the target task, and learn the generators by optimizing towards the validity score of the data transformed through ODE integration. For example, in an image classification dataset, we use a pre-trained feature extractor and define the validity score to be the cosine similarity between the features extracted from the original image and the transformed image. For PDEs, the validity score is defined by the numerical errors of the original equations after the transform. The validity scores are chosen based on the characteristics of the target tasks, and designed to be fully differentiable, so that the symmetry can be learned via gradient descent in an end-to-end fashion. We also incorporate two regularizations, orthonormality and Lipschitz loss, which prevent the learned generators from converging to trivial solution.
Subsequently, we demonstrate that our method indeed discovers the correct symmetries in both image and PDE datasets. To the best of our knowledge, our research is the first to retrieve affine symmetry in the entire space of continuous transformations using the CIFAR- 10 classification dataset, as shown in Figure 1. Moreover, our method excels in identifying non-affine symmetries and approximate symmetries in PDE tasks. We further demonstrate that the learned generators can be leveraged to develop automatic augmentation generators, which can be used to produce augmented training data for both image classification tasks and neural operator learning tasks PDEs [21]. We provide empirical evidence that the models trained with data augmented by our learned generators perform competitively with those trained with traditional closed-form transforms such as Lie point symmetry (LPS) [4]. Moreover, we show that the approximate symmetries discovered by our method, which cannot be found by classical methods, can also boost the performance of the models, especially when the size of the training data is small.
Infiniteimmal Generators of Symmetries
Transformation Images
Transformation PDEs
W
qaI
0.0
6.1
6.
0.00
0.55
0.0
ois
Tanefomation (b)
(a)
(c)
Figure 1: (a) Examples of the vector fields. V3 is a learned symmetry which is approximately rotation, while V/ is not a symmetry, thus having high validity score. (b) Transformed C1 images using the learned generators. All the vector fields and transformations learned from CIFAR-10
are presented in Figure of Appendix C. (c) Transformation of PDEs (KS equation) with learned symmetries: time translation (t-tsl) and Galilean boost (gal).

---

Table 1: Infinitesimal generators of the Affine group (2). The set of six generators {L,,-.. Le} forms a basis of the corresponding Lic algebra.
Flow ay
Generator
Expression
One-P One-Paarmmeeer Group
Description
(1.0)
1 a(1,
translation at aatis trinsaation stg-axis
La
[0: 1)
3 ++ x + a(0, 1)
Xg
(#1.0)
scaling aaxis
Vector Beld
La
(0.42)
scaling of any aaxis shear purallel at shear parallel axis
Lo
(F2.0)
(0,a))
Figure 2: An example of a flow.
La [a1. #2)
uniform sculing
Lu Ls
(-F2,45)
rotation
2 Preliminaries: One-parameter Group
In this section, we present the basic definitions of one peee-aamttter group, which we use to parameter- ize the symmetric transformations learned from the data.
Consider an unknown Euclidean domain Z Re and a smooth vector field V Z- R", A path d
y: I (a, b) S RR + Z satisfying di (s) (s)( for all E I is curve that travels around the domain Z with a velocity given by the vector field V. Along the curve y, a point zy = 7(ag) can be transported to r(as + s) by flowing along the vector field V by time We define the flow Oy by 0% (wo) = r(ao + 8) of as in Figure 2. This flow is computed by solving an ODE
d
(x) V(x)
(I)
ds
with initial condition 05 for all Z [20].
The flow Sv governed by an autonomous ODE, i... an ODE independent of the temporal variable s. Due to properties of autonomous ODEs, the flow 0V exists uniquely and it is smooth in both variables 8 and æ. Assuming a mild condition on V, such V extends to compactly supported vector field V on R", the ODE does not terminate R in finite time and hence vy is defined for all E R. In that case, the flow satisfies a group equation
0V tg (a) = OY, (x)
for all 81, 82 E R. It means that the flow can be regarded as group action of IR on R.,"" transforming elements of 2 C R". For this reason, 0y is also called a one-parameter group, and the vector field V is called an infinitesimal generator of the one-parameter group.
On 2 = R" constant vector field V(x)= = v E R" gives rise to a translation + sv. For matrix A E Rax" a vector field V (x) Aæ gives rise to an affine transformation æ + exp(s AA) ax where exp the matrix exponentiation. Multiple infinitesimal generators may span a vector space g. and if satisfies some algebraic condition (closure under the Lie bracket), then forms a Lie algebra. Composing the elements of one-parameter groups of elements in g gives rise Lie group G. The correspondence between G and is called Lie group-Lie algebra correspondence.
Continuous symmetries commonly defined by Lie group G, acting on some domain and keeping the transformed objects invariant with respect to some criterion. We model symmetries by specifying their infinitesimal generators whose composition of one-parameter groups comprises the symmetries of that domain.
Below, we describe two representative examples that will be discussed extensively in the remainder of the paper: images (interpreted as functions on 2D planes) and PDEs.
2.1 Images and Their Symmetries
Consider a rescaled image of the form f: X f transformations on IR² have the form x (α), 222) H Ar + b for matrix vector b E R The Affine transformations form the 6-dimensional Affine group Aff and it has corresponding 6-dimensional Lie algebra having a basis (LI,..., La) given as in Table 1. The symmetries of images are often exploited as a data augmentation strategy for learning image classifiers, under an assumption that the transforms do not alter the identity or semanties of the images to be classified.
E R33 The affine
R2x2 and
3

---

2.2 PDEs and Their Symmetries
Given an n-dimensional independent variable x = (I1, ... ,n) E C IR" and an m-dimensional dependent variable u = =u(x)= = (my (2), , um (2)) E E CR, we denote by ( the collection of all i-th partial derivatives of u with respect to æ. A partial differential equation A on u(x) of order & is defined by set of algebraic equations (ar, u, tz(t), : (*())) 0 involving all the variables and their partial derivatives. For example, two scalar independent variables z, t E IR and one scalar dependent variable u(a, t) E R governed by equation A ue + uug + Uezz = 0 gives the I-dimensional Korteweg-de de Vries (KdV) equation, where we denote partials using subscripts, e.g. 21 du and Uzzz .. e.g. shallow water waves [36]. The KdV equation described above is an example of I-dimensional scalar-valued evolution equation. Such equation takes the form u w(x; 1) E R with its governing equation of the form
8yy The KdV equation is commonly used to model the dynamics of solitons,
= Fxx, l, u, Uzy Many Mazans
for some function F. In this paper, we only deal with ID scalar-valued evolution equation fixed periodic domain x E [0, L].
Continuous symmetries of PDEs are commonly parametrized by one- ane-paraeeter group on X' x U. Denote (E. vb) (E xr, 11 uI) an infinitesimal generator defined on X' x U. Then the PDE A possesses the infinitesimal generator of symmetry (E, (1) if the equation is still satisfied after transforming both the independent variable x and the dependent variable u [27, 5, 26]. Symmetries of PDEs are categorized by how the generators (E. depend on (æz, u). The symmetry is a Lie point symmetry (LPS) if the value of (E, 11) at each point (zz, u u(x)) depends only on the point value (ax u(x)) itself. If (E. (1) also depends on the derivatives u²)), .."),... (² at that point, called a Lie- Bäckläc symmetry or generalized symmetry. If (E, 14) depends on integrals of u, then it is called a nonlocal symmetry. Finding an LPS of a PDE A can be done algorithmically under some mild assumptions on A. However, there is no general recipe of finding Lie- fäeBBäcklnnd symmetries or nonlocal symmetries, and discovering such symmetries remains an active area of research.
3 Related Work
Symmetry discovery. Approaches to learning symmetries can be categorized by addressing two questions: (a) where do they search for symmetries, and (b) what are they aiming to learn. One line of research aims to learn ranges, focusing on determining the ranges of transformation scales that enhance learning when employed as augmentation techniques. For example, Benton et al. [3] learns transformation ranges of predefined transformations by treating them as learnable parameters and backpropagating through differentiable transformations.
Another line of research aim to learn subgroups of bigger candidate groups, typically a linear group (L)(n) or an affine group Aff For example, Desai et al. [10] use the Generative Adversarial Network (GAN) to search for symmetries, with the generator transforming data by group elements sampled from the candidate group and the discriminator verifying whether the transformed data sill lies in the data distribution. Similarly, Yang et al. [34] employ the GAN approach, but generator of GAN models infinitesimal generators instead of the subgroup itself, and learns affine symmetries such as rotation of images and Lorentz symmetry of high-energy particles. As an alternative, Moskalev et al. [24] proposed an idea of extracting symmetries from learned neural network by differentiating through it, and retrieved 2D rotation in the linear group using the rotation MNIST dataset.
Finally, learning symmetries with minimal assumption, i.c. without assuming the infinitesimal generators are linear or affine, is an area of large interest. An early attempt of Rao & Ruderman [28] models infinitesimal generator by a learnable matrix from the pixel space the pixel space, and learn 2D rotation by solving a task that compares original images and rotated ones, where the images are 5 x 5 random pixels. Sohl-Dickstein et al. [32] takes the similar approach with
the learnable matrix. Dehmamy et al. [9] builds a convolution operation whose kernel encodes learnable infinitesimal generators, and retrieved 2D rotation from random 7 x 7 images by comparing original and transformed ones, and Yang et al. [35] uses an autoencoder to simplify nonlinear symmetries into linear ones. Our work closely aligns with Liu & Tegmark [22] and Forestano et al. [13], which model aeepparmetter groups by an MLP and learn the symmetries from an invariant scalar quantity. To the

---

best of our knowledge, learning correct symmetries with minimal assumption was only achieved with toy datasets, far from real-world datasets such as CIFAR-10.
Utilizing symmetries in deep learning. An effective method for leveraging symmetries in deep learning is data augmentation [31]. In the image domain, there are numerous augmentation tech- niques available [29], most of which are based geometric properties of images. Although data augmentation techniques have been primarily explored in the context of images, recent studies by [4, 23] have demonstrated that symmetries can also be used for augmenting data in the training of neural PDE solvers. In addition to data augmentation. some approaches involve designing new neural network architectures that inherently reflect the group symmetries of the input data [7]. Wang et al. applied similar strategy within the PDE domain.
4 Learning Continuous Symmetries with One-Parameles Groups
4.1
Training Process
Given a learning task with a dataset C A in an underlying space (flf to model symmetry by a one-parameter group 0, acting A, as explained in $ 4.3. We define continuous symmetry by stating that is a symmetry of this task if there exists some a such that for any data point f E D and transformation scale E [-o, o., the transformed data point 1, (f) remains valid for this task. We assume the existence of a differentiable validity score S(V,. f) R, such that o, (f) A is valid if S(D,, f) < C for certain threshold C' R. Then, mne-parmettr group / a symmetry of the task if (0,,, < C for all f E D.
y}. we aim
The validity score depends on the nature of the target task, though no strict criterion exists. As long as is differentiable and the valid data aids learning, it is considered acceptable. For instance, we can define the validity based on negative log-likelihood of a probabilistic model. 4.2, we discuss the validity scores be used for image and PDE data.
Data
Infinitesimal Generator
Modilled by MIP
Update S(U., f)
Transformed via ODE integration d.
Once validity score is defined, we learn a symmetry 8 by minimizing the validity scores of transformed data,
Figure 3: Process of learning symmetry.
0- = arg min R/ (S(Os, (4)
where the arg min is taken over the entire class of smooth one-parameter groups. Since the learning performed in function space, we appropriately constrain the function space using regularizer, as described in $ 4.4. Once symmetries are learned, they reveal the symmetrical properties of the target task, which can then be exploited to augment the training data.
4.2 Task-specific Definition of Validity Score S
Images. In image-related tasks, we define validity score using pre-trained neural network. Let D be an image classification dataset consisting of data of the form (f. y) E A R, where f is an image and is label. Also let H&s o Hisu A R be learned neural network, where we denote by Hies -> & the feature extractor and Has R -> R the classifier. We define the validity score S(0,, f) the cosine similarity between the features before and after the transformation:
S(0.. /) = sim (Hrea (O,())))))) H(e)())),
where sim is the cosine similarity defined as sim(v1,00 v2) = || [|lee
for all v1, E TR (0).
Let (z:) be a solution a given PDE A, discretized on rectangular grid Xgria = far; 1 For transformed data 0, (u), we measure the violation of the equality A = 0 to assess whether the transformed data is still valid solution. Using an appropriate numerical differentiation method, we directly compute the value of the PDE, denoted as A (D (u)). which represents the error of (u)
PDEs.
5

---

as solution of A, taking a value (u))s at grid point as. The validity score is defined by the summation of all PDE errors across the grid points:
S(0., u)
(6)
For example, for solution u(x) of the ID KdV equation, we examine whether the transformed solution = 0,(u (u) satisfies û, + uü, + Ugzz = where the partials are computed using a numerical differentiation method.
4.3 Parametrization of One-Parameter Groups using Neural ODE
On a Euclidean domain C IK"", we model an infinitesimal generator with an MLP hg 2 S In + R". The infinitesimal generator he gives rise one-parameter group & We sample transformation scale a ~ Unif([-o, aI) for predefined hyperparameter a E R2 To transform point E E along this one-parameter group by an amount 2 0, we use numerical ODE solver to solve the ODE for y : [0, al + Z satisfying
(ss) = he (r((),, Vs E a),
7(0) =
and obtain a transformed data point ă - ohe (xx) = r(a). If a < O, we compute 0 (x) 0 0=2 (a) by integrating -hg instead of he using the ODE solver. We can backpropagate through the numerical ODE solver using the adjoint method [6] to learn 0.
Let f X y be data point on domain A. As A is a space of functions, naively modeling symmetry on A may ignore the geometry implied in the input space X. Instead, we define two transformations: Ox on X' and By on V, and induce a transformation of f by
(Ux(f)(=) = f(0x' fz)x(())) (z)), (by(f)(z) 0y(f(z)),
(8)
where abuse notation and write the transformed function 0x and Oy( For an image represented as a discretized function ʃ : X' + y from X' = [-1, 1 and y y [0. 1]s. dx corre- sponds spatial transformations such as translation or rotation, and ey corresponds to color space transformations. For a PDE, a ID scalar-valued evolution equation on a fixed periodic domain takes the form u(x,t) E U I with (x, t) [0, L] x [0. 7] X' RR, and we parameterize an infinitesimal generator on product space X' x u C R by an MLP. Then, a transformation on (xr, t, u) E X x LI induces transformation on the solution of the PDE t(.
4.4
Objective Functions
learned. Let (hg (ar) be the
Let Neym be the number of symmetries infinitesimal generators computed from single MLP. For each a E {1,.... Naym we sample transformation scale 8, ~ Unif([-o,,0) a)) to transform via numerical integration. The parameter is optimized by minimizing the average validity score over the training data,
Symmetry loss.
Naym
S (00t)) (a) .r)]
Laym (0)
Σ UaDDaa-
Orthonormality loss. as the zero vector field or the same vector field repeated in multiple slots. To prevent this, we introduce the orthonormality loss to regularize the model towards learning orthonomral vector fields. Specifically, given two vector fields Vi. V22 Z R", we define an inner product as,
Learning only with the symmetry loss may result in trivial solutions such
(V1, V22) = vol(z) / (V) V2(x)) R
Σ
w(xx)(Vi(2;)) V2(xz)), (10)
Zgridl
m
with suitable weight function w(x) 2 - IR and a discretized grid Zgri of Z of size Zgnia Given this definition, we first normalize each generator by its norm to ensure | (a) = 1. Then we compute the orthonormality loss as,
Lonto (0) =
Σ
(sg(h(o)) h )
(11)
ISacos Naym

---

(0) where sg(-) denotes the stop-gradient operation to ensure that the constraint (hg . - 0 only affects the latter slot (6). By doing this, if the true number of symmetries N sym is less than or equal to the assumed number of symmetries Neym. the learned symmetries will be aligned in the first Naym slots.
Lipschitz loss. We further introduce inductive biases to the infinitesimal generators aim to learn. For instance, an infinitesimal generator moving only a single pixel near the boundary by a large scale would be undesirable. This idea can be implemented using Lipschitz continuity. For grid point 2; E 2grid and its neighhoring point 2; E nbbd((;;) C Zyri.. we expect the vector field V to satisfy the Lipschitz condition,
Lips wi, mn)) < where LV; m.) [IV V(x;)] || x3
(12)
To regularize the model toward the Lipschitz condition, introduce the Lipshictz loss,
Neym
Lips (
Σ
max(Lips(hgg (a), :2;; - T, 0).
(13)
a=1 E Eait.ary (
Total loss and los--sall--indeeendnnt learning. We We jointly minimize the three loss functions with suitable weights Wsyms Wortho: Wpps > 0 and learn the weights 0 of MLP using stochastic gradient descent:
0 = arg min Wsym Syym (0) (0) + Conha (0) + Wr Lipp
(14)
To minimize the computational burden of hyperparameter tuning, ensure that all the loss terms have a natural scale, i.e. a dimensionless scale independent of the context. For example, when penalizing the inner product in Equation 11, apply arccos to the normalized inner product to ensure the loss term lies in [0, 77 /22.. Similarly, the scale of the PDE validity score S(U,, u) in Equation depends on the scale of the data u. When penalizing it, we apply the log function so that the gradients are scaled automatically as V log(((0), u)) VeS( u)/S(0,, u).
Here we describe the generic training process, but the actual implementation requires non-trivial task-specific designs, such as the choice the weighting function w w(x) or the method for locating the transformed data the target grid. We defer these details for image and PDE tasks to Appendix A.
4.5 Comparison With Other Methods
Here, we compare our method with other recent symmetry discovery methods. The differences mainly arise from (a) what they aim to learn (e.g., transformation scales subgroups from larger group) and (b) their assumptions about prior knowledge (e.g., complete, partial, or no knowledge of symmetry generators). Another important distinction is the viewpoint symmetry: some methods learn symmetries that raw datasets inherently possess (implicit), while others learn symmetries from datasets explicitly designed to earry such symmetries (explicit).
We emphasize that our method excels in two key aspects: (a) our learning method reduces infinitely many degrees of freedom, (b) our method works with high-dimensional real-world datasets. For example, while LieGAN [33] and LieGG [24] reduce 6-dim space (affine) to 3-dim space (translation and rotation) in image dataset, reduces an oo-dim space to a finite one. L-conv [9] also does not assume any prior knowledge, but it is limited in finding rotation in a toy task, where it learns rotation angles of rotated images by comparing them with the original ones, which are 7x7 random pixel images.
Some recent symmetry discovery works are listed in Table 2.
5 Experiments
5.1 Images
We use images of size 32 x 32 from the CIFAR-10 classification task. Since our method does not model discrete symmetry, we use horizontal flip with 50% probability by default. We train a ResNet-18 18 model, which will be used as the feature extractor His in Equation 5, The weight function
7

---

Table 2: Comparison with other symmetry discovery methods.
Augerino
LieGAN
LieGG
L-conv
Forestano et al.
Ours
Symmetry generators
completely known
partially
partially known (affine)
completely unknown
completely unknown
completely unknown
known (affine)
Learn what?
transformation scales
symmetry
generater
symmetlry generator
symmettry generator (rotation)
generator (in low-dim task)
symmetay
synmeetry
generator
(rotation
(rotation)
(affine)
Lorentz)
Verified with what?
raw CIFAR-10
MNIST
rotation
rotation MNIST
7 x pixel image
random
toy data (dim S 10)
raw CIFAR-10 & PDEs
Top tageing
Implicit or explicit?
implicit
explicit
explicit
explicit
explicit
implicit
optimize while training
compare fake/true daatn
extracts from
compare rotated and original images
extracts from invariant oracle using Lie derivative
extracts from validity score using ODE integration
learned NN
How?
downstream task
GAN framework
using Lie derivative
Inner Products
Affine Components
Affine-ness
Leamndd Generators
Affine Generators
(1, 0) 40
CM
ar
4881
ADE
033 -u (C3
1.2
(0, 1)
Measure
1.0
(xg, 0)
0.38
0.0
(0, 030 4:30
AM
03
5.0
0.0
0.4
(Xa, 0)
0.2
V. V.V
(0, x2) 415 800
V1 V.
V. V. V. Vo V, Vn V4 Vin
035
004
5.5
000
0.30 301 M6
0.0
V1
V3 V1 Vo V, V V10
Learned Generators
Learned Generators
(a)
(b)
(c)
Figure 4: (a) Self inner-products of the learned generators. (b) Inner product comparison of the learned generators with the affine generators. (c) Affine-ness of learned generators.
on the pixels is computed as explained in Appendix A.1. We expect to find 6 affine generators and we use an MLP modeling 10 vector fields in the pixel space [-1, 1j² C IR², expecting the first six learned vector fields to be the affine generators. We learn the Equation 14 using stochastic gradient descent and Wonto: W''ips = 10. The parameter o, which controls the scale of transformation, is set to a = 0.4, and the Lipschitz threshold r is set to 7 0.5. Other details are described in Appendix B.1. We conducted three trials with random initializations and report the full results in Appendix C.I. Furthermore, we also learn symmetries in the color space, and their results are shown in Appendix G.
with uyym
Learned symmetries. Since we expect to learn affine symmetries, we compare the results with the affine basis [In,.... ,La} defined in Table 1. We compute the inner products (V. L,) of the learned vector field V with L, for 1, 1.... 6 to measure how much the learned vector fields contain the affine basis and measure the ness of vector field by Affine-ness
-V, L.)².
In all experiments, we successfully retrieve six linearly independent affine generators in the first six slots. Figure 4a shows that the learned generators are orthogonal to each other, as desired. Figure 4b shows the inner product between the learned generators and the affine generators. Since the affine- ness measure of the first learned generators in Figure 4c is almost close to 1, we can read out the affine components in Figure 4b and say that e.g., V1 R (0, -0.98 0.12rg). Notably, two translation generators are found in the first two slots, indicating that the two translations are the most invariant one pee-aaarmeeer group among the entire class of one parameter groups on the pixel space. After the two translation generators, four affine generators are learned, indicating that affine transformations are the nexl most invariant transformations. In particular, the third and fourth generators are close to the rotation generator and the scaling generator, respectively. The remaining four generators fix pixels close to the center and transform boundary pixels by a large magnitude.
8

---

nkdV
cKdV
Components
Components
1.0
X- tsl
1.0
X-
0.5
g2
0.5
g2
g3
t-tsi
0.0
0.0
1/2
1/2
1/3
V4
V/2
V33
112
1/2
V
V4
V,
Figure 5: Inner products between the learned non-affine symmetry generators and the ground truth.
The results including the affine symmetry generators are shown in Figure 10a.
Table 4: Definition of PDEs.
Table 5: LPS of PDEs.
Name
Equation
Name
Point Symmetrics
KdV
ande
Kdv (4), KS [4). Burgers [17]
(1,0,0
(0, 1,000
(4,0.1)
KS
Mr thet Mares Wike
sav (Appendix D.3)
(1.0.0)
(0.e 5.0)
(fu(efã 1),0,13
Burgers
ekdV [30]
(1,0,0) (VT 1, syep 0)
nKdV
1o 0g + uMg + Oaea
cKdV
Mazza 2(1411)
Analysis. Unlike other symmetry discovery researches [24, 9] that use datasets which are explicitly designed be symmetric such as rotation MNIST, we discovered affine transformations from CIFAR- 10 with no augmentation except horizontal flip. Moreover, we extract the symmetries from the ResNet trained with CIFAR- 10 without augmentation. This implies that although CIFAR-10 is not explicitly composed to be symmetric under affine transformations, the dataset possesses intrinsic affine symmetry. This also implies that even the ResNet trained without augmentation possesses invariance under affine transformation. It is widely believed that the strong generalization power of neural networks is linked to the augmentation insensitivity of the neural networks [25]. Our results show that ResNet is insensitive to affine transformation even when not explicitly designed to be so, supporting this hypothesis. This result implies that augmentation explicitly amplifies the insensitivity by using transformed data for training. Our analysis is tangential to that of Gruver et al. [16], which measures the extent of invariance using derivatives along the infinitesimal generators instead ODE integration.
Table 3: Test accuracy in CIFAR-10.
Augmentation results. We train ResNet-18 18 model using CIFAR-
10 classification data, applying the learned symmetries as data augmen- tation. We compare the results of no-augmentation, default augmenta- tion (horizontal flip and random crop), and affine transformation, with transformation scale searched in {0.1, 0.2, 0.3, 0.4, 0.5). We conduct five experiments with random initialization for all the settings and report the results in Table 3.
Method
Acc. (%)
No-aug Default
92.4 + 0.3
95.1 + 0.1
Affine
95.1 + 0.2
Learned
94.9 + 0.1
5.2 PDEs
We follow the experimental setting of Brandstetter al. [4], which use the Korteweg-de de Vries (KdV) equation, the domain as experiments. They all have time translation, space translation, and the Galilean boost as LPSs. To consider PDEs with non-trivial and non-affine symmetries, we add two variants of the KdV, namely the nkdV equation and the cKdV equation. The nKdV is yielded by nonlinear time translation of the original KdV, and the cKdV is the cylindrical KdV equation, having extra time-dependent term. The equations are listed in Table 4, and their symmetries are listed in Table 5, Note that some symmetries are ruled out since we work within fixed periodic domain, e.g., the scaling symmetry of the KdV. Since there are at most three symmetries, we open four slots in an MLP and learn symmetries using weights Wsym: Wips 1 and Worto 3.
(KS) equation, and the Burgers' equation on a 1D periodic
Learned symmetries. We compare the learned symmetries with the ground-truth symmetries using inner products. We found the ground truth symmetries all the experiments as in Figure 5 and Figure 10.

---

KdV
KS
nkdV
cKdV
Ground truth Learned
Ground truth Learned
Learnd
1
Learned
Test NMSE
Test NMSE
10
Test NMSE
Test NMSE
10
10 -
10
10-1
10-2
10-2
10
10-1
2
2
2
2
2
2
2
2
2
2
2
2
# Data
# Data
Data
#
Data
Figure 6: Comparison of augmentation performances using the ground truth symmetries and the learned symmetries with various numbers of data. The symbol @ stands for no-augmentation.
Table 6: Test NMSE comparison of augmentation using LPS and AS in FNO learning for eKdV.
# Data
None
LPS
AS
LPS+AS
27
(3:70 L 0.09) 10-6
(3.49 1 1.06)
10
(3:20 1 0.94) x 10
(2.66 1 0.36) x 10
2
(7.90 + 1.21) 10-1 (5.90 0.17) x 10-4
(4.45 + 0.15) x 10-1 (3.70 + 0.15) x 10-4
Interestingly. for the Burgers' equation and the cKdV equation, we additionally found the w-axis rescaling operation (0, 0, u) and the time translation (0, 1,0) respectively. Applying the u-axis rescaling u + cu for c R 1 to the Burgers' equation v, + Whl - uu = 0 gives Cu4 + cauu, vcu,z c(c 1) wu,. leaving only the utg term. The utl, term in Burgers' is called the convection term, and it is approximately zero in most region and spikes in some small region. Similarly, the time translation + t + c for e fixes the first three terms in the cKdV equation ut + uly + Usew u/(2(t+ + 1)) 0 and only changes the last term u/ u/(2(t + 1)) by negligible amount. These are not LPSs of the given equations, but the error of the PDE after transformation is smaller than the error of the numerical differentiation method. These are approximate symmetries (AS), and the theory of AS also of great interest the symmetry analysis of PDEs [2, i].
Augmentation results. We use the learned symmetries as data augmentation and train Fourier Neural Operators (FNOs). The detailed experiment setting is described in Appendix B.2. Since FNOs are extremely sensitive to numerical error, we employ Whittaker Shannnon interpolation, explained in Appendix D.2, to resample the transformed results. The results depicted in Figure 6. In all cases, data augmentation using the learned symmetries improve the performance, almost close to the results using the ground truth symmetries. The detailed results are in Appendix C.2. Additionally, that the approximate symmetry of cKdV is also beneficial to training, as shown in Table 6, especially when the numbers of data points is low, proving the effectiveness of symmetries extracted from data.
verify
Ablations. Additional ablation studies on numerical methods, such as numerical differentiation and interpolation, and hyperparameter sensitivity are conducted, and their results are presented in Appendices and F.
6 Conclusion
We have introduced novel method for learning continuous symmetries, including non-affine trans- formations, from data without prior knowledge. By leveraging Neural ODE, our approach models one-parameter groups to generate sequence of transformations, guided by a task-specific validity score function. This approach captures both affine and non-affine symmetries in image and PDE datasets, enhancing automatic data augmentation in image classification and neural operator learning for PDEs. The learned generators produce augmented training data that improve model performance, particularly with limited training data.
Limitation. However, despite its flexibility, our method requires careful selection of numerical methods, such as numerical differentiation and interpolation, to ensure stable training and the ODE integration can be computationally large for augmentation generation compared to other augmentation methods. While we focus on image symmetries and LPSs of PDEs, the method could potentially model other symmetries and domains with proper validity scores, suggesting future applications in learning complex symmetries, including conditional and non-local symmetries, in various data types.
10

---

Acknowledgments and Disclosure of Funding
This work was partly supported by Institute of Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea goV))) (No.RS-2019-1190073 Artificial Intelligence Graduate School Program(KAIST), No. 2022-0-00713, Meta-learning Applicable to Real- world Problems, No.2022-0-00188. Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics, No.RS 2024-0009999, Global Al Frontier Lab)
References
[1] Baikov, V., Gazizov, R., and Ibragimov, N. K. Approximate symmetry and formal lincarization. Journal of Applied Mechanics and Technical Physics, 30(2):204-21) 212, 1989. 10
[2] Baumann, G. Approximate symmetries of partial differential equations. Symmetry Analysis of Difserential Equations with Mathematicaa, pp. 404 423, 2000. 10
[3] Benton, G. W., Finzi, M., Izmailov, P., and Wilson, A. G. Learning invariances in neural networks from training data. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 2020. 1,
[4] Brandstetter, J.. Welling, M., and Worrall, D. Lie point symmetry data augmentation for neural pde solvers. In International Conference Machine Learning, pp. 2241-2256. PMLR, 2022. 1, 2, 5, 9, 15, 16
[5] Cantwell, B. J. Introduction symmetry analysis. Cambridge University Press, 2002. 4
[6] Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018. 6,
[7] Cohen, T. and Welling, M. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, PMLR, 2016. 1,5
[8] Dangovski, R., Jing, L., Loh, C., Han, S., Srivastava, A., Cheung, B., Agrawal, P., and Soljacic, M. Equivariant sulf-supe learning: Encouraging equivariance representations. In International Conference on Learning Representations (ICLR), 2022.
[9] Dehmamy, N., Walters, R., Liu, Y., Wang, D., and Yu, R. Automatic symmetry discovery with lie algebra convolutional network. Advances in Neural Information Processing Systems, 34: 2503-2515, 2021. 2, 4, 7,9
[10] Desai, K., Nachman, B., and Thaler, J. Symmetry discovery with deep learning. Physical Review D, 105(9), 2022.
[11] Dey, N., Chen, A., and Ghafurian, S. Group equivariant generative adversarial networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[12] Dumbser, M. and Käser, M. Arbitrary high order non-oscillatory finite volume schemes on unstructured meshes for linear hyperbolic systems. Journal of Computational Physics, 221(2): 693-723, 2007. 15, 17, 19
[13] Forestano, R. T., Matchev, K. T., Matcheva, K., Roman, A., Unlu, E. B., and Verner, Deep learning symmetries and their lie groups, algebras, and subalgebras from first principles. Machine Learning: Science and Technology, 4(2):025027, 2023. 2,
[14] Gabel, A., Quax, R., and Gavves, E. Data-driven lie point symmetry detection for continuous dynamical systems. Machine Learning: Science and Technology, 2024.7
[15] Garrido, Q., Najman, L.,, and LeCun, Y. Self-supervised learning of split invariant equivariant representations. In Proceedings of The 40th International Conference on Machine Learning (ICML 2023), 2023. 1

---

[16] Gruver, N., Finzi, M. A., Goldblum, M., and Wilson, A. G. The lie derivative for measuring learned equivariance. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
[17] Hoarau, E. and David, C. Lie group computation of finite difference schemes. arXiv preprint matth/0611955, 2006. 9
[18] Hoogeboom, E., Satorras, G.G., Vignac, C., and Welling, M. Equivariant diffusion for molecule generation in 3d. In Proceedings of The 39th International Conference on Machine Learning (ICML 2022), 2022. 1
[19] Huang, Y., Peng, X., Ma, J., and Zhang, M. 3dlinker: An E(3) equivariant variational autoen- coder for molecular linker design. In Proceedings of The 39th International Conference on Machine Learning (ICML 2022), 2022. 1
[20] Lee, M. Smooth manifolds. Springer, 2012. 3
[21] Li, Z., Kovachki, N. B., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A. M., and Anandkumar, A. Fourier neural operator for parametric partial differential equations. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[22] Liu. Z. and Tegmark, M. Machine learning hidden symmetries. Physical Review Letters, 128 (18):180201, 2022. 4
[23] Mialon, G., Garrido, Q., Lawrence, H., Rehman, D., LeCun, and Kiani, B. Self-supervised learning with lie symmetries for partial differential equations. Advances in Neural Information Processing Systems, 36:28933-29004, 2023. 1,
[24] Moskalev, A., Sepliarskaia, A., Sosmovik, I., and Smeulders, A. W. M. Liegg: Studying learned lie group generators. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022),
2022. 2, 4, 7,9
[25] Novak, R., Bahri, Y., Abolafia, D. A., Pennington, J., and Sohl-Dickstein, J. Sensitivity and generalization in neural networks: an empirical study. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018,
Conference Track Proceedings, 2018.
[26] Oliveri, F. Lie symmetries of differential equations: Classical results and recent contributions. Symmetry, 2(2):658-706, 2010. ISSN 2073-8994. doi: 10.3390/sym22206588 URL https: http//207--994/////// / //www.ddpi
2.
[27] Olver, P. Applications of Lie groups to differential equations, volume 107. Springer Science & Business Media, 1993. 4
[28] Rao, R. and Ruderman, D. Learning lie groups for invariant visual perception. Advances in neural information processing systems, 11, 1998. 4
[29] Shorten, C. and Khoshgoftaar, T. M. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019. 1,5
[30] Sil, S. Nonclassical symmetries, nonlinear self-adjointness, conservation laws and some new exact solutions of cylindrical kdv equation. International Journal of Applied and Computational Mathematics, 955)69,, 2023. 9
[31] Simard, P. Y., Steinkraus, D., Platt, J. C., et al. Best practices for convolutional neural networks applied to visual document analysis. In Icdar, volume 3. Edinburgh, 2003. 1,
[32] Sohl-Dickstein, Wang, C. M., and Olshausen, B. A. An unsupervised algorithm for learning lie group transformations. arXiv preprint 1001 2010. 4
[33] Wang, R., Walters, R., and Yu, R. Incorporating symmetry into deep dynamics models for improved generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. 1.7,
12

---

[34] Yang. J., Walters, R., Dehmamy. N., and Yu, R. Generative adversarial symmetry discovery. In Proceedings of The 40th International Conference on Machine Learning (ICML 2023), 2023.
4
[35] Yang. J., Dehmamy, N., Walters, R., and Yu, R. Latent space symmetry discovery. In Interna- tional Conference on Machine Learning, 2023. 4
[36] Zabusky, N. J. and Kruskal, M. D. Interaction of"" solitons" in a collisionless plasma and the recurrence of initial states. Physical review letters, 15(6):240, 1965. 4
[37] Zhang, Y.-T. and Shu. C.-W. Eno and weno schemes. In Handbook of numerical analysis, volume 17, pp. 103-122. Elsevier, 2016. 15, 17
13

---

05. (e())()) A Implementation Details
A.1 Image Dataset
Notation. Images functions of the form f V/, where A' 1-1, 1| is the spatial domain and y = [0, 1 is the normalized RGB domain. Pixels Xgris = { Ji are discretized through rectangular grid of X, and images f are discretized on the pixel space by f. = f(x,) for all i.
w(x) in orthonormality loss. We first learn a symmetry on the spatial domain using a feature extractor Hhiten taken from pre- trained neural network. One obstacle is that in most image datasets, the main subjects of images are mostly located around the centers, and the regions close the boundary In other words, each pixel has a different level importance, and may end up learning infinitesimal generators that only move boundary pixels and fix the center.
Weight Function
filled with backgrounds.
600
400
We take into account the importance of pixels using the weight function w(x) X + Rzo- For discretized image f = } and for each pixel æ;, we measure pixel sensifivity of the image the i-th pixel up to the feature extractor fext as
Figure 7: The weight function w(x).
Sensitivity(J, xx.) = (Hox(()
(15)
Ox:
0x:
0f:
which can be computed by querying a dncobian-vector product of Hau with respect to the image gradient at each pixel 0f: dataset:
Sm."" We define the weight wxx) as the average of pixel sensitivity across the
|| @H (f)
w(x)) //
EsnD
(16)
Ox;
However, computing this weight function w needs Ngnd Nuau times of computation of Jacobian- vector products. Instead, use a Gaussian kernel n(æ; @ σ) V2no o = 0.1 and the center sampled uniformly on X, and approximate the weight function by
_ exp(-
|
with fixed
2n
Σ HHua(()) ox,
w(xx) zo E. (((2;
no Maz 2
(17)
so that we compute the weight in a stochastic manner. Intuitively, it computes Jacobian-vector product for each larger pseudo-pixel represented by the Gaussian kernels instead of each individual pixel. We iterate over the dataset 20 times, and the computed weight function is shown in Figure 7.
Training details. The infinitesimal generator h transforms each pixel æ; into 7; via ODE
integration. In this process, the transformed pixels { { may longer be located on the rectangular grid Xgrid. so we use the bilinear interpolation method to resample the transformed image v, (f) on Xgria- Note that the bilinear interpolation operation differentiable, thereby we can train the MLP by minimizing the validity score S(1., J)), which is the cosine similarity between the features of / and f.()):
S S .. f) sim (Hrst (+ (f)), Han
(18)
A.2 PDE Dataset
Notation. Solutions of ID scalar-valued evolution equation periodic domain take the form
u(x,t) f) X - li for = [0. L.] x [0,T] and /4 = IR, where [0, L) is the spatial domain and [0. T] is the temporal domain. Similar the image task, the domain X' is discretized by the rectangular Nai discretized solution set of tuples (I,, 4, u;) where
grid Xgna- = ( Nai {(x,, 4))
u, ut)
14

---

w()) in orthonormality loss. Unlike in images, we assume that all discretized grid points hold equal importance, so we set w(x) for all æ E X.
Training details. The infinitesimal generators on the product space X' x U transform a point (14,4 l; u.) into {(0,, f,, ū,)]. Therefore, the transformed solutions are no longer on the rectangular grid. To compute the PDE value A(ū) for ū (2) in Equation 6, we need to compute partial derivatives numerically such as u, (T;; (4). uy (ã., 1,) or Uaz (ū:, f.). We use the finite difference method, in which the numerical derivative is approximated by finite differences, e.g..
u(3; + Ar. (2) u(a; Ar,4)
(Ti, (i)
(19)
2Ax
for some small Ac.
In particular, we use the weighted essentially non-oseillating (WENO) scheme as a numerical differentiation method [37], with careful choice of parameters as in Dumbser & Käser [12]. In the WENO method, multiple estimates for derivatives are made using multiple sets of neighboring gridpoints (called stencils). The multiple estimates are then averaged with weights (called smoothness indicator) that approximate how stable the derivative estimates are We implement the WENO method working on nonuniform grid, and the model learns symmetries by backpropagating through it. A detailed description on the WENO scheme is in Appendix D.1.
B Experiment Details
In this section, we present the detailed experimental settings of the experiments in 5.
B.1 Images
Training ResNet-18. When training the ResNet-18 with CIFAR-IO, 10, both the feature extractor Hiett and models after augmentation, we train the model in 200 epochs with a batch size 128. The learning rate is set to 10 and decreases by a factor of 0.2 at the 60th, 120th, and 160th epoch. The model is trained by SGD optimizer with Nesterov momentum 0.9 and weight decay 0.4.
Learning symmetries. To learn the symmetry generators, we train the MLP using two shared hidden layers, each with width of 256, followed by hidden layer of width 32 for each output vector field. We use the swish activation function to ensure the learned vector fields are smooth. The MLP is trained for 50 epochs with batch size 128 and fixed learning rate of 10 using the Adam optimizer. The learning process takes less than 10 hours on a GeForce RTX 2080 Ti GPU.
B.2 PDEs
Data generation. We follow the data generation method of Brandstetter et al. [4]. Given an ID evolution equation u4 = F F(x, t, u, Ug ur for u = u(x, t) on periodic domain [0, L], we start with initial condition u(2; 0) by random Fourier series
u(x, 0)
Σ Ap sin sin op)
(20)
where P is the number of Fourier modes and (Aps Op) are random coefficients. For time-stepping, we compute z-derivatives Fourier domain and converts them back to the original domain. We use an ODE solver to compute the time evolution of u( [z, t) from t = 0 to = T. The solution is discretized on regular grid of size N₂ x N₂ on [0, T.] x [0, T). where N₂ = 256 and N, = 140 by default.
Mzz, ... using pseudospectral method, which computes derivatives in
When simulating Burgers' equation, we instead solve the PDE for the heat equation Φi = duz for vo(x,t) t). The Burgers' equation u(x,t) and the heat equation ((x, t) is related via the Cole-Hopf transformation:
2 log(z).
(21)
dr
After data generation, we transform the heat equation back to the Burgers' equation.
15

---

Learning symmetries. We use an MLP with the same architecture as described in Appendix B.1. Since all the tuples (ar, t, u) must pass through the MLP, totaling N2 x N, for each data instance, we set small batch size 4. Symmetries are learned using 1024 data instances over 50 epochs. We use the Adam optimizer and train the network with learning rate of 10-4 in the first 25 epochs and 10-5 for the remaining 25 epochs. The learning process also takes less than 10 hours on GeForce RTX 2080 Ti GPU.
Sobolev regularization. To ensure smoothness in the vector fields, apply additional regulariza- tion using the Sobolev norm of order the r-domain. For a vector field V(x, t), the Sobolev norm be efficiently computed in Fourier domain:
IV(,012222 = Σ
2
vV(111) Oxt
No-1 Σ
(1
Rereq 2
1)
V(n,t)
(22)
where Réfreq min(n, N₂ n) and (-, the discrete Fourier transformation V (-,t). Since we 1, we penalize towards the Sobolev norm excluding the zeroth order term:
already enforce || ||
IV ( 1)1
Σ
vV(.t) Ozi
Nz-1 Σ
(1++1
Mhreyy
2 )²-1) (n, L
(23)
We apply the Sobolev regularization during the final 10 epochs.
Training FNOs. For the KdV and KS equations, we train an autoregressive FNO solver, which takes 20 timesteps as input and predicts the subsequent 20 timesteps, following the experimental setup of Brandstetter et al. [4]. For the nkdV and cKdV equations, which are time-dependent and contain explicit terms,
train FNOs as single-time neural operators. These models utilizes the initial conditions of the equations to predict the states after 70 timesteps. We train the FNO over 40 epochs, with each epoch comprising 280 iterations across the dataset for the KdV and KS equation and 100 iterations for the nkdV and cKdV equations. The learning rate begins at 10-4 and decreases by a factor of 0.4 every 10 epochs.
C Experiment Results
In this section, we provide detailed results of the experiments outlined in 5. For symmetry learning tasks, we conducted each experiment three times and randomly selected one for reporting 5. The complete set of results is provided here. Also, we report the detailed evaluation metrics in augmentation tasks.
C.1 Images
Figure a visualizations of the learned symmetries discussed in $ 5.1. Figure displays results from experiments conducted under the same settings but with different model initializations compared to Figure 5.1. It is notable that affine symmetries consistently occupy the first six slots across all experiments.
C.2 PDEs
We report experiments results for learning symmetries, conducting three trials for each equation in figure Figure 10. The ground truth symmetries and the approximate symmetries are consistently found in the former slots in all the experiments. The remaining fourth slots occasinally converge to learned symmetries despite the orthonormality loss or converge to some unknown vector fields with high validity scores. The results from the first trials used in the augmentation experiments.
In Table and Table 8, we provide the augmentation results of FNOs using the learned symmetries, which are illustrated in Figure 6.
16

---

Learned Generators
Transformation of Images
V1
V,
V2
V3
V
Va
V;
V.
Ve
Ve
Vg
V1
Via
-0.3
-0.1
0.0
0.2
Transformation Scale
aa)
(b)
Figure 8: (a) The learned vector fields. (b) Transformed images using the learned generators. The images with transformation scale 0 are the original images.
Table 7: Comparison of augmentation methods with different numbers of data for KdV and KS
Kdv
KS
# Data
None
Ground-truth
Ours
Nome
Ground-truth
Ours
0,398 +: 0.013
0.0640 = 0.0050
0.0824 + 0.0052
0.00693 + 0.00039
0,00229 + 0.00014
0.00614 = 0.00051
0.246 = 0.012
0.283 + 0.018
0.324 + 0.031
0.0241 + 0.0001
0.0422 0.0002
25
4.47 0.26
0.980 0.039
1.12 4 0.10
5.78 0.13
1.14 4 0.10
1.37 0.04
D Technical Details
D.1 Weighted Essentially Non-Oscillating (WENO) Scheme
This section is largely based on Zhang & Shu [37], Dumbser Käser [12].
WENO scheme in ID. Consider a smooth function to the function on the grid Xgna = {x1,-- N. 21 < ... N To estimate the derivatives
C R - R. where we only have access
(() d' (a) for some E X, we use à + 1 neighboring points in / = {o;,--- .i+k) } and compute the unique k-th order polynomial pr (x) that interpolates the function f on the points in In other words, we ensure pr (2,,) f(x),, (P((4)))) f(x(4K). Then we can pick up the (k)
estimates of the derivatives using the polynomial as (*) P7 (x:). We call the set of neighboring points I stencil, and the polynomial pr the reconstruction polynomial. Usage of reconstruction polynomials is a fundamental concept in the numerical differentiation method known as the finite difference method.
In the weighted essentially non-oscillating (WENO) scheme, we compute multiple estimates (k) Pi. (z), Pix, (x) for the derivatives (())(x) using multiple stencils 4,,** ,I,, The estimates
(k)
17

---

Inner Products
Affine Components
Affine Generators
(1, O)
Affine-ness
Learned Generators
1.2
(0, 1)
0:38
1.0
(xg,
4:20
020
Measure
0.8
(0, X2)
0.6
(12,.0) 0)
021
0:40
5.71
0.4
(0.x))
0.2
V12 V.V. V; Vi
1/2
Learned Generators
V/3
V3 V/ V/ V/ V/ V10
112
V22 V, V1 V, V, V,, V V 1/00 Leamed Generators
Learned Generstors
(a) Second experiment.
Inner Products
Affine Components
Affine Generstors
(1, O)
Affine-ness
Learned Gentrators
1.2
(0, 1)
1:31
-302
000
485
5
031 -301
1.0
Measure
0.8
(0, X2
4.0
0.6
0)
620
0.00
314
0.4
(0.x.)
0.2
V. V, V. U. K4V Learned Generators
111
Learndd Generators
V.3 V4
V/ V/ V9 V10
0.0
V1
V2 V3 V1 V, V, V, Vo V 1/0 Leamedd Generators
(b) Third experiment.
Figure 9: The same results (learned symmetries of images) under different initializations compared to Figure 4.
Table 8: Comparison of augmentation methods with different numbers of data for nKdV and cKdV
akd
ckdV
Data
None
Ours
None
Ours
2"
3352 0.11) x 10-3
0.17) x 10-3
(3.72 x 10-"
0.37 x 10-t
27
3.16 1
0.06) x 10
(2.6 1.500 x 10
(.719100.99) x 10
(1.86 1.100) x 10-6
(1.79 0.09) 10-1
02.11 10
1.21) x10
(2.98 0.337 x10
are then averaged with weights W1,-. ... NN,, which take account of the quality of each estimation. In the WENO scheme, we assume the grid points are sufficiently dense, hence the estimates are more accurate when the reconstruction polynomials pr,.. are smooth. The smoothness of pr... is measured by the smoothness indicator, defined as:
IS Σ
/
(x))
(24)
where A = (1j.:1+11) R is an interval between grid points containing x, and | is the length of A. The weights w1,- -- WN, are defined as:
Qkm
mm
Wm E ml Orm"
Cly
IC+IS
(25)
where 7' is called the linear weight, usually chosen heuristically, 6 is positive integer usually set to 2 or 4, and 0 small positive number preventing the denominator from being zero. The final estimate for the derivative (()) (x) is computed by averaging the estimates Pl, (a:) ... .Pis. (a)
(k)
with the weights wi,-.. WNN:
(())
(k) Wm Pim (x).
(26)
WENO scheme for multivariate function. The WENO scheme extends smoothly to multidimen- a²
sional function f IR. - R. For k (ki, h:n) E Z we denote (xr) 0= "t. kn
On a stencil with an appropriate number of grid points, we compute the reconstruction polynomial Pr (z)) with nonzero k--dgree term, and the smoothness indicator is defined similarly as:
IS = Σ
A la
(p/aa
(a) (z)) 2d
(27)
15
SK
A
18

---

KdV
KS
Burgers
Camponent
1.0
Comporents
1.0
X-ts
0d cen co
Compare
0.5
0.5
0.5
gal
0.0
ga
0.0
gal
0.0 Vi
U-SC
V,
V. V. V
Va
nkdv
ckdv
Comganents
1.0
X-tsi
0U00 3000 2000
Components
1.0
x-tsi
40 000
0.5
100
540
100
0.5
92 022
600
400
400
3.00
0.0
t-tsl
023
4014 4006
1/2 V/2 V.
V4
V1 V2
V3
V43
V2
V.
Va
V1
V2
V1
V
(a) First Experiment
KdV
KS
Burgers
Componeet ett
1.0
x-ts
1.0
x-ts
1.0
x-tsi
0.5
t-ts
0.5
t-ts
0.5
t-ts
3:30 B
gal
gal
V1 1/2
V/3
1/
V1 V22 V 1/
0.0 V1
V22
V/3
V2 V V4
0.0 V1
V22 V. Va
1/1 V,
kddv
cKdV
Companents
x-tsl
Comporents
1.0
X-tsl
0.5
92
0.5
92
0.0
93
0.0
V/2 V4
V2 Vs V4
V/
Va
(b) Sccond Experiment
KdV
KS
Burgers
Component
1.0
432
Componants
1.0
x-ts
000
Components
1.0
x-tsi
0.5
0.5
0.5
gal
0.0
gal
0.0
gal
0.0
U-sCl
V/2
12
V/2 V3 V/
V33
V. V.
V3 Va
V.
nndv
cKdv
Comgorents
1.0
t-ts
Comporents
1.0
x-tsl
0.5
92 0:79
0000 3001
0.5
92
000
93
A00
A00
0000
0.0
t-tsl
ocs 000
V1
V.
V.
V4
V1 Vs
Vs V4
V1
V2
V3
Va
V1 V2 V3 Va
(c) Third Experiment
Figure 10: Inner product comparison of the learned symmetry generators with the ground truth symmetries. (a), (b), and (c) are conducted with different random initializations.
where = (aI,-.. an) E E a = Σ a and the interval A is now a n--imensional cell containing the point with volume | Like the 1D case, the weights are defined using the linear weights and smoothness indicator.
Choice of parameters. We follow the choice of parameters of Dumbser & Käser [12]. The linear weight 7m is chosen be 100 if is inside the center cell of Im and otherwise, reflecting the fact that the estimates are accurate as much as the point æ is close to the center J, The parameters and are chosen as b and 10-6.
D.2 Whittaker-Shamnn Interpolation on Periodic Domain
Let 1], f(],, be a discretization continuous signal on real line. Shannon interpolation recovers the original signal f as:
f[......
The Whittake-
20
f(t) Σ fin]sine(t--) n)
(28)
where sinc is the normalized sinc function sinc(() sin(Tx)
The Nyquist-Shamnn sampling theorem states that states that [(() is the perfect reconstruction of f. in sense that if the original function does not contain any frequencies higher than certain threshold, called the Nyguist frequency, then f is perfectly reconstructs the original signal.
az
19

---

Similarly, if f(0), f[ fIN - 1] is a discretization of continuous signal on a periodic domain [0 for some even positive integer N, the Whittaker-Shamin interpolation is given as:
N-1 sinc(t m.)) Σ f(n)DN(t) (29)
f(t) Σ f[n mod M]sine(t- - n) Σ
f(m]
Σ
n=-00
where ((t))===--) sinc(t m.N) is the Dirichlet kernel with period N. The Dirichlet kernel (t) has a closed-form expression
sin(Tt (t N taa(TttNN)
(30)
D.3 Nonlinear Time Transformation of the KdV Equation
We apply a nonlinear time transformation t defined as = tole - 1). where to is a constant scaling factor, to the KdV equation u, + utl Uazz.
The derivative with respect to t is transformed to
0 ai
0
+ θ e
(31)
āt
at at
ai
hence the KdV equation with nonlinear time transformation (nKdV) becomes
e to Upl + uig + Uzzx
(32)
The three symmetries of the KdV changes accordingly. The space translation is left untouched, and the other two infinitesimal generators are tranformed as:
(33)
01 σi
0
0
t
to t0 - 1) Oz
(34)
dx
du
du
where we used the derivative notation
(1,0,0) 0
(0,1,0) 1,0)
&
and (0,0, 1)
a
8u.
E
Ablation Studies
E.1
Numerical differentiation.
When searching for symmetries of PDEs, we use the WENO scheme numerical differentiation method. We have found that the choice of numerical differentiator is critical, as gradients must flow through numerical differntiation during the backpropagation step. To compare different methods, experimented with using the WENO scheme and another method: bilinear interpolation followed by discrete numerical differentiation followed by discrete differentiation, the loss failed to converge well, even with very small learning rate 10-6 The learned vector fields were not orthogonal, and they only spanned the time translation and the space translation at best, as in Figure 11. These experiments were conducted using the KS equation.
regular rectangular grid.. When using bilinear interpolation
We hypothesize that the bilinear interpolation distributes each transformed point into two adjacent grid points, but discrete differentiation is done by repeatedly subtracting values of two adjacent grid points, so the gradient may not flow well during the backpropagation.
E.2 Whittaker-Shannor interpolation.
In the PDE augmentation task, when the transformed PDEs are resampled into the regular rectangular grid, we use Shittaker-Shannor instead of more commonly used bilinear interpolation. This step is also crucial for training FNOs with transformed data, since FNOs are extremely sensitive to numerical error and hence any aliasing effects must be avoided. We compared the augmentation results using Whittaker-Shanoon interpolation with those using bilinear interpolation, using the KS equation with 512 training data and report the result in Table 9. The results clearly demonstrate that bilinear interpolation adversely affects the training FNO models.
20

---

Learning Curve
Bilinear Int Discrete Diff
2.8
2.4 2.2
Components
1.0
V1
10. 660.85 4:50
V2 0.
2.0
0.5
t-tsl
V3 ALECS 100
gal
V
0.240.25 300
10 20
30 Epochs
40
50
0.0
V1V2V
11
V2
V
V1VaV
(b)
Figure 11: Experiment using using bilinear interpolation and discrete differentiation. (a) Learning
curve. (b) Learned symmetries.
Table 9: Test NMSE comparison of augmentation using Whittaker-Shannn interpolation and bilinear interpolation.
No-aug
Whittaker-Shannon
Bilinear
Test NMSE
0.00693 0.00039 0.006143 + 0.00051
0.542 + 0.051
Hyperparameter analysis
In this section, we analyze the roles of various hyperparameters in our learning scheme. We use experiment setting of the KS equation in 5.2. We selected the KS equation for analysis as it proved to be the most challenging in learning symmetries, possibly due to its fourth-order derivative term Uzzra- In particular, among the three symmetries (space translation, time translation, Galilean boost) the algorithm easily learns the space and time translation in few epochs, but learning the Galilean boost takes more epochs as Figure 12, We evaluate the results by examining whether the three symmetries correctly found in the first three slots.
Learnnng Curve of Each Generator
-1.25
-1.0
V1
-1.75
-2.00
Va
-2.25
Va
-2.50
-2.75
-3.00
-3.25
10
20
30
40
50
Epochs
Figure 12: Learning curves of four symmetry generators.
F11 Lipschitz Threshold
The Lipschitz threshold 7 Equation 13 is the only parameter that constrains the function space in which we search for the symmetries. The threshold is set to = 3 in the main experiments. We conduct additional experiments with r = 1, 2, 4, 8, 16, 32. Surprisingly, in all experiments, regardless of the value of v, we found the three ground truth symmetries.
F.2 Transformation Scale
The transformation scale controls how much we transform the data when searching for symmetries. The tuple (x. f, u) in X' x 24 is scaled so that r and f form uniform grid on [0, 1]2, and is scaled so that the standard deviation of closely matches that of x and t, which approximately 0.29. Hence, the default transformation scale = 0.4 transforms data with slightly more than its standard deviation most. We conduct experiments with various transformation scales a = 0.1, 0.2, 0.4, 0.6, 0.8. The ground truth symmetries were found when 0.1, 0.2, 0.4. When 0.6, the Galilean boost was found but allocated in the fourth slot, meaning that the learning was unstable. When a 0.8, the model only found space and time translation correctly.
21

---

1.00
T = 2.0
Companents
1.0
tsi
Componennts
1.0
x-tsI
t-ts
0.5
t-ts
gal
ga
3000
0.0
ViVaVaVa
0.0
V.V.V3V4
V3V4
V.V.V.V.
V.
V.V.V.
T= 4.0
8.0.0
Components
1.0
x-tsi
Componentss
1.0
x-tsi
0.5
0.5
t-ts
gal
gal
0.0
V2
V.V
V.VVVVV
D.0
V.
V.V.VVVa
V.V
T= 16.0
32.0
Comgonents
Components
x-tsl
x-tsi
onee
0.5
t-tsl
0.5
t-tsl
gal
gal
0.0
V. V2
V.VaV.VV
V.V.V.V.
0.0
V.Va
V.V.V
V.V.V.V.
Figure 13: Experiments with various values of hyperparameter
0 = 0.1
0= 0
1.0
X-tsi t-tsl
V1 BUE
comaonents
1.0
x-tsi
0.5
0.5
t-ts
gal Bodken
0.0
ViV.VaVa
0.0
V.V.V5V4
V2
V.V.VaV4
V.
V.V.VVV
0 0.4
0.6
Components
1.0
x-tsl
Components
1.0
x-ts
0.5
t-tsl
D.5
gal
gal
man
0.0
V2V2
V1V.V.VV
V.V.V.VV
V1 V. Va
V.V.V
V.V.V
0=
Components
1.0
x-tsl t-tsl
0.5
gal
0.0
V.V.VVVV
V4
V.V.V.
Figure 14: Experiments with various values of hyperparameter d.
F.3
Weights of Three Losses
Although the three loss terms Equation 14 are designed have dimensionless scales, tuning the three loss terms in appropriate ranges is inevitable. We perform grid search over the three weights Wsym: Wortoa WLisechizz of the total loss in a fixed grid [1,3, 10]3. We report the number of correctly learned symmetries across different values of Wayms WLips and Wonbo in Table 10. The number 3 is the maximum number of symmetries to be discovered. We found that the ratio between Waym and Wortho significantly affects the results. In this case, the weight Wonbo should be larger than Wsym to prevent the learning of redundant infinitesimal generators.
G Color-space Results
Recall that our formulation, images take form X - y where [-1, the pixel space and ) E RR3 is the RGB color space. Our formulation is not limited to searching for the symmetries on 2D plane X, but capable of learning symmetries the color space ) In this section, we elaborate how modeled learning scheme for color-space symmetries and present the results.
22

---

Table 10: The number of correctly learned symmetries across values of Wsyms WLipss and Wortho-
(a) Waym
(b) sym
3
(c) Msym
10
Wortho
WLips
Wortho
Liis
W Wotho
WLips
1
3
10
3
10
3
10
2
3
2
2
2
2
3
2
3
3
3
3
2
2
2
10
3
3
3
10
3
3
3
10
3
3
3
Transformation of Images: V1
Transformation of Images: V2
Transformation of Images: Va
Figure 15: Learned color-space transformations.
G.1 Learning Scheme
Similar to other cases, we model one- pee-paraetter groups on y by neural ODE, integrating over 3D vector fields y - I parametrized by neural network. Colors are non-uniform for example, black, brown and gray appear much more frequently than pink yellow in CIFAR-10 dataset. To address this, we first form a grid Vgna on the color space y by randomly sampling 1024 colors from the CIFAR-10 dataset. Then we define the weight function y - R, like the weight function in Equation 17, by the color sensitivity of the neural network H For grid point y, E Ygnds the weight w(y,) is defined
Dllleal (f)
( ES-D
(35)
Oy:
To estimate the magnitude of the gradient with respect to the change y we use neighboring points y, E nbbd((y.) C. Vgria. For discretized image {æ,, fxx Exa the nearest neighbor
23

---

algorithm to collect f(z;) Nearest(y,) close to yı and measure how much Hex changes when the color f(x;) shifts towards the direction yj yz
|| 1e oy,
Σ
[w) v,) V (p, Htex()) (f)l
(36)
(y.)
My (g
where (y)) is vector whose i-th entry is 1 if f(z;) Nearest(y,) and 0 otherwise, and V. (Neareatty)) Hex (f) is computed by Jacobian-vector product.
Once the weight function is computed, we learn symmetries by optimizing through the validity score S, defined by cosine similarity of features of the learned neural network. Since there (batch size). (number of pixels) number of colors in a single batch images, it's impractical to feed all of them directly into the MLP. Instead, we use the grid grid and compute the vector field values on the grid and interpolate them using the nearest neighbor algorithm.
G.2 Results
We open three slots for color-space symmetries and train the model with Weym: Wtips and Wortho 3. We found the brightness control, color contrast and blue-yellow shift respectively as in
Figure 15.
24

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clarified our contribution in the last two sentences in the abstract and the last paragraph in the introduction. Both sections mentioned that our scope is learning continuous symmetry with minimal prior knowledge.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
25

---

Justification:
We mentioned our limitations in the last two sentences in the conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Although we did not provide any theorems propositions, which requires proofs, all formulas are numbered and cross-referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We checked the reproducibility of the reported experiments and we provided the hyperparameter analysis in the paper and we shared our source code.
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We attached our source code and we used public datasets, which are freely accessed by anyone.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guiae htte//gguddeeeeeeoe Seliides//ooeeuumissseee haabbmmsssoooooccyyy...... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ inips.cc/publln httpc//ggiiee/////ueummmnnnee (CodeSubiiss wwwpsummisssooPPooooo asionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
27

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All hyperparameters used the experiments are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We measured the performance with 3-5 repeats. In Table I and Table 4, we reported the standard deviation, and in Figure 6(B), we described using the error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We also mentioned such information in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the itciieeeennleeessss
NeurlPS Code of Ethics https: //tpurrp///u//....///
Answer: [Yes]
Justification: We checked the code of ethics and our paper does not cause any hazard because it only explores theoretical symmetries.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Since we explore only theoretical symmetries present in mathematical space, our work far from the societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
29

---

Answer:
[NA]
Justification:
We do not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credited when adopting the existing codes and properly cited the original paper that produced the code and dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We shared our code with anoymization and documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
30

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve any potential risks incurred by study participants
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "JinE3zCbPw", "reviewer": "Reviewer_UU8r", "summary": "This paper proposes a symmetry learning algorithm based on transformations defined via infinitesimal generators. Using Neural ODE, an infinitesimal generator is learned that is capable of producing a sequence of transformed data through ODE integration. Validity score has been defined to check if the transformed data is valid wrt to a given task. For images, the validity score is picked to be cosine similarity while for PDEs the validity score is defined as the numerical errors in the original equations after the transformation. In addition to symmetry loss, two regularizations, orthonormality loss, and Lipschitz loss have been added, to remove trivial solutions. The authors present experiments on CIFAR10 and KdV equation and Burgers' equation in 1D for PDE.", "questions": "1. How does the validity score take into consideration, invariant tasks vs equivariant tasks? \n2. Using cosine similarity between the extracted features for validity score, can be a problem for extreme transformations (as cosine values do not linearly change with parameter). How does this affect the learning of symmetry build on validity scores?\n3. In Table 2, it is unclear how the validity score players out in learning the symmetries. Especially for default augmentation case. Could you please elaborate on this?\n4. Total loss and loss-scale section: How is the true scale learned, if the inner product is normalized? Additionally if all the terms are affected by a data-dependent scale, shouldn't $w_{Lips}$ affect the overall weights?\n\n### Clarifications\n1. The figures can be made a little bigger. \n2.  In line 16, 'whether the transformed data are still valid for the given task'. This phrasing is confusing.\n3. In line 172, what does valid mean? simply put; does it comprise invariant transformations and approximately equivariant up to the threshold C?\n4. Intuition on how to locate transformed data on grid is missing.", "limitations": "1. The authors do not compare with other methods of learning symmetry, as those methods use datasets that have explicit symmetry (like Rot-MNIST). Similarly for that of the PDE experiments. It would have been useful to have them in the paper.\n2. The number of parameters and compute comparison across the proposed method and baseline approach like [9,14] is missing.  Also discussion the scale of numerical computation with an increase in dimension missing.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper motivates the need for learning continuous symmetries well. \n2. The idea presented in the paper is very neat and shows potential beyond the presented experiments. \n3. This approach can learn both affine and non-affine symmetries in image classification and PDE tasks as shown in the experiments section.", "weaknesses": "The discussion on compute and model parameters comparisons with baseline missing. No other method was shown as a baseline in either of the experiments. There is some ambiguity in how exactly the validity score is used and in some cases, can be defined if the given task is equivariant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback.\n\n**Question 1. Validity score in equivariant task.**\nWe take a two-step approach: first, we learn symmetries based on validity scores, and second, we use these learned symmetries as augmentation and solve machine learning tasks. The validity score is designed to measure invariance of a certain function – the neural network in the image task, and the PDE residual values in the PDE task. We do not take equivariant tasks into account in this paper. However, if one were to design a validity score using equivariant neural networks, it could be achieved by modeling the transformation in both the input and output spaces and establishing an invariance criterion that corresponds to that equivariance.\n\n**Question 2. Cosine similarity and instability.**\nCosine similarity values lie in $[0,1]$, and if arccos is applied, the angle values lie in $[0,\\pi/2]$. The arccos function has an infinite gradient at 1, so the cosine similarity values must be clipped to a maximum $1-\\epsilon$ for some small $\\epsilon$. Except for this, we did not experience any instabilities coming from cosine similarity.\n\n**Question 3. Validity score and augmentation.**\nThe validity score is only used when learning symmetries. Once the symmetries are found, we use the learned symmetries to augment the dataset for the target task – image classification task in this case. Table 2 shows comparison of augmentation performance with other commonly used augmentation methods.\n\n**Question 4. Scale of losses.**\nSorry, but we didn’t fully understand your question. We will try to answer based on our understanding, but if you find it insufficient, please let us know.. The phrase “normalized inner product” in the paper refers to the following process: two vectors $v_1,v_2$ are normalized as $\\hat{v_i} = \\frac{v_i}{||v_i||}$ for $i=1,2$, and then the inner product $\\langle \\hat{v_1},\\hat{v_2}\\rangle$ is computed. Then we put arccos to make it lie in $[0,\\pi/2]$. Similar process is applied for all terms, to make sure all losses are on a similar scale.\n\n**Clarifications 2, 3. Meaning of validity.**\nWe say transformed data are valid for the given task when they are beneficial to the learning task. For example, in an image classification task, images rotated by small angles may help the task, but transformed images that are distorted too much may harm training. To approximate this validity, we design a function known as the validity score within our framework.\n\n**Clarification 4. Interpolation method.**\nWe use bilinear interpolation for images, and Whittaker-Shannon interpolation for PDEs. Due to page limits, discussion on that is in Appendix A. and Appendix E.2., including discussion on why bilinear interpolation does not work for PDE data but Whittaker-Shannon works.\n\n**Limitations 1. Baselines.**\nPlease refer to paragraph “Comparison with baselines” in the general comment.\n\n**Limitations 2. Computational costs.**\nWe use a fairly small MLP, with around 150k parameters. Direct comparison of computational complexity with other methods is not feasible since all symmetry learning approaches have varying assumptions and goals. For example, L-conv [9] learns rotation by comparing rotated 7x7 random pixel images with the original ones.\n\nWe provide some analysis on computational costs of our algorithm. The major computational burden of our algorithm arises from the ODE integration. Since we use the adjoint method for backpropagating [6], although the time complexity is $O(N_{\\text{step}})$ where $N_{\\text{step}}$ is the number of ODE steps, the memory requirement is $O(1)$. The choice of validity score also affects the complexity. However, the effect of dimension $n$ of the space is negligible, since it would only require changing the input and output dimension of the MLP by $\\mathbb{R}^n \\mapsto \\mathbb{R}^n$.\n\n[4] Dehmamy et al., “Automatic symmetry discovery with lie algebra convolutional network.” NeurIPS 2021.\n\n[6] Chen et al., “Neural ordinary differential equations.” NeurIPS 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MpPukWRBnG", "reviewer": "Reviewer_7EZa", "summary": "The paper pertains to the topic of data-driven symmetry discovery. The authors propose a method allowing symmetry discovery beyond pre-defined Lie groups, by learning to transform datapoints, potentially in a non-affine manner, via a learned ODE (referred to as the *one-parameter group*, where the single parameter is the time variable in the ODE), the velocity field of which is typically parametrised by an MLP. \n\nCrucially, to optimise this, the authors choose an objective - *validity score*, which is a predefined measure of the extent to which a transformed datapoint is symmetric to the input one (in their examples: for images, they use the cosine similarity between features extracted by a pretrained NN, while for PDEs, they measure the value – error – of the PDE for the transformed datapoint). Additional regularisers are used to ensure the diversity of the symmetries learned (orthogonality between different learned velocity fields) and smoothness (minimisation of an estimate of their local Lipschitz constants). Experimentally, the method is tested on image classification (CIFAR10) and PDE solving (KdV, KS, Burger’s equations) showing that known symmetries are retrieved along with additional approximate symmetries, while the learned symmetries are subsequently used for data augmentation showing competitive results to methods using pre-defined augmentations.", "questions": "- How is the weight function in L234 defined, and how important is this for optimisation? Are different weight functions ablated?\n- It’s unclear why the stop-gradient is needed in L237. I did not find the justification fully convincing. Could the authors elaborate? What happens experimentally if one does not use stop-gradient?\n- Although intuitive, it’s unclear why the inline Eq. in L212 holds for negative $\\alpha$.\n\n**Suggestion**.\nIn case the authors do want to present their method as generic, I think a deeper experimental evaluation in data beyond PDEs would help a lot (e.g. testing on other image datasets, ablating different validity scores etc).\n\n**Minor**:\n- What if the chosen validity score is not differentiable?\n- *Notation*.\n  - The notation $\\theta^V_s(x)$ is a bit dense (why use V as a superscript?). Eq (1) is a bit confusing. The inline Eq in line 83 sees clearer to me. \n  - Notation in sec. 4 could be also simplified a bit or made more accessible with examples (e.g. 4.1: give an example for $\\mathcal{A}$, i.e. the set of all natural images).", "limitations": "Some of the limitations are adequately discussed. An important missing point is, in my opinion, the need to manually design the validity score in domains beyond PDEs.\n\nNo foreseeable negative societal impact.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Significance** . The paper studies an important problem (*data-driven symmetry discovery*) in machine learning, but also physical sciences where symmetries are abundant but potentially unknown. Identifying unknown symmetries and incorporating them in downstream ML models (e.g. via augmentation) can improve generalisation, especially in low-data regimes, while additionally, it can potentially provide novel insights about the task at hand.\n\n**Novelty/Generality**\n- The presented methodology has the capacity to recover symmetries arising from *non-affine* data transformations. This is contrary to prior work, where mostly linear/affine transformations are dealt with.\n- Additionally, this method does not require making assumptions about the structure of the target group. This is common in prior art, where typically a subgroup of a predefined group is learnt.\n- The authors take advantage of well-established concepts that are underexplored by the ML community (e.g. modelling transformations via the one-parameter group) - this helps to broaden the available toolbox in the field of ML & symmetries/ equivariant ML.\n\n**Execution/Implementation**\n- Although the proposed method has multiple complicated components (NeuralODEs, difficult objective to optimise for), it is nonetheless well-executed yielding competitive results and recovering known symmetries in popular testbeds.", "weaknesses": "**Applicability and scope**. Perhaps the biggest limitation of the proposed method is the *reliance on the validity score*. Although the authors claim to be able to learn symmetries by making as few assumptions as possible (see strengths), this seems to be contradicted by the need to manually design a validity score. Moreover, I have the impression that the validity score is not merely a hyperparameter, but it is decisive for the symmetries that will be learnt (basically it is the objective function of the optimisation problem). \n-  For example, in the case of images, the choice seems ad hoc (closeness in the representation space of a pre-trained encoder NN). What leads the authors to believe that the features of equivalent (symmetric) images extracted from the pre-trained NN should be close? Have the authors tried to verify this assumption? I think the empirical validation is insufficient here (section 5.1.), so I am not entirely convinced.\n- In general, I do not see a generic way to define validity scores and perhaps the authors have slightly overclaimed in that respect. I would like to read the authors' viewpoints on that. For PDEs, the validity scores are indeed reasonable and generic, so perhaps, they would like to put more emphasis on this perspective.\n\nFurthermore, the authors introduce the concept of learning symmetries via the one-parameter group, claiming that it is more general than prior parametrisations that can only learn linear/affine groups. However, it is unclear what the present parameterisation can express, e.g. does it allow learning any continuous group or implicit assumptions are made here as well? \n- Additionally, could the authors discuss if it would be possible to learn finite groups with this method as well and if not, how could those be incorporated?\n\n\n**Related Work/Comparisons**. The work of Forestano et al., MLST’2023 is quite relevant to the present manuscript, with the main difference being that in that work, the transformations are parameterised by an MLP instead of a NeuralODE (the oracle used in this work seems similar to the validity score used here). Since the two works have many similarities, I think that the authors should discuss in more detail their differences and the advantages of their work (e.g. as far as I understand the MLP cannot guarantee that the transformations form a group). Note that modelling transformation via an MLP (or any NN in general) instead of a NeuralODE seems more straightforward and probably easier to train and more computationally friendly.\n\n**Experiments**. I believe some additional empirical evidence would strengthen the authors' claims.\n- Most importantly, an experimental comparison against the type of parameterisation used in Forestano et al. (MLP) should be provided, to verify if NeuralODEs are indeed a more appropriate parameterisation.\n- Moreover, baselines are mostly missing, e.g. comparing against other methods for data-driven symmetry discovery (I am not super familiar with these works, but if I am not mistaken LieGAN by Yang et al., ICML’23 is a recent example). \n- The reported results after augmenting with the learned symmetries do not seem to improve significantly compared to known/default augmentations. Can the authors discuss why this might be the case? This is important since it might undermine the necessity of the proposed approach.  To be more convincing, perhaps the authors should perform experiments on problems where the symmetries are not known a priori.\n- Additionally, ablation studies seem to provide important insights but are only discussed in the appendix. I would recommend being more upfront in the main paper and discussing in the rebuttal the following: sensitivity to hyperparameters (multiple are needed: loss coefficients, $\\sigma$ and $\\tau$), the method for choosing them, the difficulty of optimisation  (3 terms are used in the objective) and if all losses are optimised in a balanced manner. Similarly, for the parameter $N_sym$, which is now chosen based on prior knowledge of the number of existing symmetries.\n\n**Presentation/Exposition**. (disclaimer - this is a minor weakness) Given that the notions discussed here are probably not widely known across the ML community, I believe that the authors should aim to provide more in-depth explanations to make their work more accessible. For example,\n- Multiple group theory/symmetry concepts are discussed without definitions (group generators, Lie group, Lie algebra, Lie bracket etc.). Additional examples that are not well-contextualised include in section 2 the one-parameter group discussion, the Lie algebra of the affine group and the discussion on the PDE symmetries (Lie point symmetries etc.). Adding some references here and providing some examples for the mentioned PDE symmetries would help.\n- In section 5.2., some concepts regarding the experimental details are mentioned without appropriate explanations, while others are only mentioned in the appendix, although it appears that they are crucial for the method. Perhaps the authors should be more upfront and explanatory regarding the aforementioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback.\n\n**Weakness 1, 2. Reliance on the validity score.**\nWe see the requirement of a validity score as a trade-off for not requiring the predefined set of symmetry generators, and searching for symmetries across the entire class of continuous transformations, which increases the degree of freedom of search space by an infinite amount. In general, a validity score is a quantification of some criterion that transformed data should meet. If one aims to find symmetries of a given dataset, they should first question “what do symmetries of the dataset truly mean”, and then should come up with a criterion that the symmetries should meet. Our validation score is a quantification of that criterion.\n\nFor PDEs, the proposed validity score comes out quite naturally. For image data, while it might seem ad-hoc at a first glance, we argue that the proposed validity score is still based on the principle shared with PDE case – the invariance measure of the targets we wish to learn (PDE solutions for PDE data and features extracted from the encoder for image data). That is, the generic principle behind the validity score is to measure invariance of a target function with respect to transformations. For PDEs, we use the L1 error to measure this invariance. For images, we adopt the cosine similarity, which is a popular metric for measuring feature distances. We also note that other methods learning symmetries from images also have similar counterparts, such as the discriminator in LieGAN [2] and the classifier in Augerino [1] and LieGG [3].\n\n**Weakness 3. Range of the parametrization.**\nOur parametrization can model any continuous (or more rigorously, differentiable) groups acting on Euclidean space, given that neural networks, according to the universal approximation theorem, can approximate any continuous vector fields. The only additional inductive bias imposed on our model is Lipschitz continuity of the vector field to be learned. Below is a proof sketch of this argument. Note that we consider connected Lie groups, so discrete groups such as reflections are not considered.\n\nLet’s say a connected Lie group $G$ acts on a Euclidean domain $\\mathbb{R}^n$ as $g:x \\in \\mathbb{R}^n \\mapsto g\\cdot x$, with corresponding Lie algebra $\\mathfrak{g}$ and the exponential map $\\exp:\\mathfrak{g} \\rightarrow G$. By the properties of connected Lie groups, any group element $g \\in G$ can be expressed as $g = \\exp(\\epsilon_1\\alpha_1) \\cdots \\exp(\\epsilon_k\\alpha_k)$ for $\\alpha_1,\\cdots,\\alpha_k \\in \\mathfrak{g}$ and $\\epsilon_1,\\cdots,\\epsilon_k \\in \\mathbb{R}$. Each component $\\exp(\\epsilon_i\\alpha_i)$ for $i=1,\\cdots,k$ can be seen as an action of a one-parameter group defined by $(t,x) \\mapsto \\exp(t\\alpha_i)\\cdot x$, with the infinitesimal generator given by $V_i(x) = \\frac{d}{dt}|_{t=0} (\\exp(t\\alpha_i)\\cdot x)$.\n\n**Weakness 4. Discrete groups.**\nUnfortunately, our method cannot learn finite groups. As in the image task, if there exists a known finite group symmetry, it can be incorporated as augmentation.\n\n**Related work/Comparisons and Experiments 1, 2. Baselines.**\nPlease refer to paragraph “Comparison with baselines” in the general comment.\n\n**Experiment 3. Augmentation performances.**\nSince we parametrize the symmetry generators by MLPs, they may exhibit numerical instability. So the closed-form symmetries are expected to perform slightly better. \n\n**Question 1. importance of the weight function.**\nAs described in Appendix A.1., the choice of weight function is crucial in the image task since each pixel has a different level of importance when put in a learned neural network. For example, if we use a constant weight $\\omega \\equiv 1$, then we may end up learning transformations that only move the pixels near the boundaries by a large amount. We compute a pixel sensitivity function using jacobian-vector products and use it as a weight function.\n\nWe conducted additional experiments ablating different weight functions. We tested two handcrafted weight functions: the “plain weight” defined by $\\omega(x) = 1$ for all $x\\in [-1,1]^2$, and the “step weight” defined by $\\omega(x) = 5$ if $||x||<0.25$, $\\omega(x) = 4$ if $0.25<||x||<0.5$, and so on, decreasing in steps until $\\omega(x) = 1$ if $||x||>1$. The weight functions and the results are depicted in Figure 3 and Figure 4 in the general comment. Indeed, the correct affine symmetries are not found with these weight functions.\n\n**Question 2. Role of stop-gradient operation.**\nIn our framework, we assume knowledge on a rough upper bound $N_{\\text{sym}}$ on the true number $N_{\\text{sym}}^*$. The stop-gradient operation in the orthonormality loss turns the optimization problem into a sequence of optimization problems with hierarchical constraints; the first slot is optimized without being affected by other slots, the second slot is only affected by the first slot, and so on. Consequently, the true symmetries are learned in the first $N_{\\text{sym}}^*$ slots in the MLP. Stop-gradient operation reduces dependence of our algorithm on the hyperparameter $N_{\\text{sym}}$, and without the stop-gradient operation, we would learn mixtures of true symmetries and non-symmetries when $N_{\\text{sym}}$ is larger than $N_{\\text{sym}}^*$.\n\n**Question 3. One-param group in the negative direction.**\nSolutions of ODEs $\\frac{d}{ds}\\vartheta_s^V(x) = V(x)$ and $\\frac{d}{ds}\\vartheta_s^{-V}(x) = -V(x)$ with initial conditions $\\vartheta_0^V(x) = \\vartheta_0^{-V}(x) = x$ are related by $\\vartheta_s^{V} \\equiv \\vartheta_{-s}^{-V}$ due to uniqueness of ODE solutions.\n\n**Presentation/Exposition.** Thank you for your feedback. We will incorporate your suggestions in the revised version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uISaER07fo", "reviewer": "Reviewer_4Qv3", "summary": "This paper proposes using neural ODEs to parameterize symmetries by viewing the ODEs flow as an element of a one-parameter group. They show that by learning the parameters of the neural ODEs, they are able to recover ground truth symmetries in image classification and PDE tasks.", "questions": "1. Line 174: Do you also need \"and for all $s\\in[-\\sigma,\\sigma]$\" in addition to the \"for all $f\\in\\mathcal{D}$\"?\n2. Section 4.1: According to your definition of a symmetry $\\vartheta$ (there exists $\\sigma>0$ such that $\\vartheta_s(f)$ is \"valid\" for all $f\\in\\mathcal{D}$ and all $s\\in[-\\sigma,\\sigma]$), it seems like solving for $\\vartheta^*$ from (4) does not guarantee that $\\vartheta^*$ is actually a symmetry, even if the optimal value of (4) is less than $C$. That is, if the optimal value of (4) is less than $C$, then you only know that the validity score is less than the threshold on average, not for all data $f$ and all transformation scales $s$. Can you please clarify this aspect?\n3. Line 217: Something is strange with the definition of $\\vartheta_{\\mathcal{X}}(f)$. In particular, what is $T_{\\mathcal{X}}$, and why does $f$ not appear in the right-hand expression? It seems to me like perhaps you meant to define $\\vartheta_{\\mathcal{X}}(f)(x) = f(\\vartheta_{\\mathcal{X}}^{-1}(x))$, and hence that you need to assume the transformation $\\vartheta_{\\mathcal{X}}$ on $\\mathcal{X}$ to be invertible. Can you please clarify whether this is a typo, or if I am missing something? Also, it is unclear what $\\vartheta_{\\mathcal{Y}}(f)$ is doing or why it is needed. Indeed, as written, you are defining two different transformations of $f$ in (8) (resulting in two different functions of $x$).\n4. Line 226: How reasonable is it to assume that you know the actual number of possible symmetries for a given task? Is it possible for you to select $N_{\\textup{sym}}$ to be too small, and then to miss out on learning some of the most important symmetries of a task?\n5. Line 232: Typo (\"orthonomral\").\n6. Line 230: In practice, are symmetries actually mutually orthogonal (as vector fields) to one another? It seems like this is not always the case per your Table 1, which shows that uniform scaling is not orthogonal to $x_1$-axis scaling. A comment on the validity of the orthogonality assumption underlying the use of this regularizer would be appreciated.\n7. Line 243: Typo (\"undersirable\").\n8. Line 246: Typo (\"Lipshictz\").\n9. Line 280: \"...the two translations are the most invariant one-parameter group...\" Can you please explain in further detail how you come to this conclusion?\n10. Table 2 and Figure 5: I suggest moving these to be centered with the rest of the text; it looks poorly placed in the right-hand margin as is, and causes Figure 5 values to be too small and hard to read.\n11. Line 322: \"We found the ground truth symmetries in all the experiments as in Figure 5...\" It looks like you did not recover the true symmetries in the cKdV setting, as there are non-negligible components coming from each of the ground truth symmetries that appear in the learned symmetries.\n12. Did you verify in your experiments that the learned \"symmetries\" are actually symmetries according to your definition using the validity score threshold? Overall, your experimental results in Figures 4, 5, and 10 seem to me like you are indeed learning orthogonal vector fields, and that sometimes the vector fields end up being equal to one of the ground truth vector fields, but other times not (turns out to be linear combination of multiple ground truths). This does not seem super convincing that you are reliably learning/recovering the ground truth symmetries.\n13. Why did you introduce (8)? I don't see this transformation being used anywhere else in the paper.", "limitations": "N/A.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to read. The proposed ideas are clear, and appear to be mostly novel.", "weaknesses": "See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback.\n\n**Q1. Necessity of phrase in line 174.**\nYes. For all $f \\in \\mathcal{D}$ and for all $s \\in [-\\sigma, \\sigma]$.\n\n**Q2. Clarification in Section 4.1.**\nTo be more rigorous, our formulation leaves us a set of constraints $S(\\vartheta^*_s,f) <C$ for all $f \\in \\mathcal{D}, s \\in [-\\sigma,\\sigma]$, which is intractable to solve. Instead, with a mild assumption that non-symmetries have comparably higher values of the validity score than the true symmetries, we turn it into the proposed optimization problem. We will make this clearer in the revised version of our paper.\n\n**Q3. Typo in line 217.**\nYes. This is a typo. it should be $(\\vartheta_\\mathcal{X}(f))(x) = f(\\vartheta_\\mathcal{X}^{-1}(x))$. We apologize for the critical typo and thank you for pointing it out.\n\n**Q4. The number of symmetries in line 226.**\nIn our framework, we assume knowledge on a rough upper bound $N_{\\text{sym}}$ on the true number $N_{\\text{sym}}^*$. The stop-gradient operation in the orthonormality loss turns the optimization problem into a sequence of optimization problems with hierarchical constraints; the first slot is optimized without being affected by other slots, the second slot is only affected by the first slot, and so on. Consequently, the true symmetries are learned in the first $N_{\\text{sym}}^*$ slots in the MLP. We believe having a rough estimate on an upper bound of the number of symmetries is reasonable.\n\n**Q6. Orthogonality assumption in line 230.**\nA set of symmetry generators form a vector space. For example, a sum of the unit $x_1$-axis translation generator and the unit $x_2$-axis translation generator gives a translation in the direction of the $45^\\cdot$ vector. **Symmetry generators listed in Table 1 and Table 4 are in fact a basis of the space of symmetry generators.** Hence, if we set an inner product in the space of vector fields, there is always an orthonormal basis of symmetry generators.\n\n**Q9. Further details about the most invariant symmetries in line 280.**\nAs discussed in the answer to Q4, the stop-gradient operation gives hierarchical constraints in the optimization problem, causing it to learn the earlier slots first and then the latter slots. Since the translations are found in the first and second slots, we conclude that they are “the most” invariant ones. This means that the outputs of the neural network are altered by the smallest magnitude along those symmetries.\n\n**Q11. True symmetries in cKdV in line 322.**\nAs we elaborated in the answer to Q6, symmetry generators form a vector space. In the cKdV case, the exact and approximate symmetries form a vector space of dimension 3. We identified three orthonormal basis components of that vector space. Although these components do not match the closed-form expressions exactly, they span the same vector space as the closed-form symmetries. Therefore, we conclude that the three symmetries have been correctly learned.\n\n**Q12. Evidence for learning actual symmetries.**\nAgain, symmetry generators form a vector space, so learning linear combinations can be as well regarded as learning the true symmetries. Moreover, we compare the distributions of validity scores and report them, both with images (top) and PDEs (bottom) in the tables below, where the non-symmetries are modeled by neural networks with random initialization. The values of quantile 0.95 shows that the true symmetries indeed have lower validity scores than non-symmetries.\n\n|  | mean | median | quantile 0.9 | quantile 0.95 |\n|---|---|---|---|---|\n| symmetry (x-translation) | 0.036 | 0.024 | 0.085 | 0.114 |\n| non-symmetry | 0.092 | 0.049 | 0.224 | 0.289 |\n\n|  | mean | median | quantile 0.9 | quantile 0.95 |\n|---|---|---|---|---|\n| symmetry (galilean boost) | 0.133 | 0.035 | 0.435 | 0.570 |\n| non-symmetry | 95.9 | 0.859 | 29.0 | 132.1 |\n\n**Q13. Necessity of equation (8).**\nThe equation (8) is not explicitly used in implementation, but it states how transformations on the spaces $\\mathcal{X}$ and $\\mathcal{Y}$ induce a transformation of the function $f:\\mathcal{X} \\rightarrow \\mathcal{Y}$. For example, an image can be viewed as a function $f:\\mathbb{R}^2 \\rightarrow \\mathbb{R}^3$, which is then discretized as $ \\\\{ f(x\\_i) \\\\} $ on pixels $\\mathcal{X}\\_{\\text{grid}} = \\\\{ x_i \\\\}$. Transformation of pixels $x\\_i \\mapsto \\vartheta_\\mathcal{X}(x\\_i)$ results in a transformed function $\\tilde{f}:\\vartheta\\_{\\mathcal{X}}(x\\_i) \\mapsto f(x\\_i)$, which corresponds to the function described in Equation (8).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful comments and suggestions. We are pleased that all the reviewers agree on the importance of symmetry discovery and find our research novel. Below, we address some commonly raised questions and present additional experimental results.

# Comparison with baselines
**(refer to performance comparison in the attached PDF file)**

There have been several attempts at symmetry discovery, each with different experimental settings and goals. The differences mainly arise from (i) what they aim to learn (e.g., transformation scales or subgroups from a larger group) and (ii) their assumptions about prior knowledge (e.g., complete, partial, or no knowledge of symmetry generators). Another important distinction is the viewpoint on symmetry: some methods learn symmetries that raw datasets inherently possess (implicit), while others learn symmetries from datasets explicitly designed to carry such symmetries (explicit).

Some recent symmetry discovery works are listed below. We emphasize that our method excels in two key aspects: **(i) our learning method reduces infinitely many degrees of freedom, (ii) our method works with high-dimensional real-world datasets.** For example, while LieGAN [2] and LieGG [3] reduce a 6-dim space (affine) to a 3-dim space (translation and rotation), ours reduces an $\infty$-dim space to a finite one. L-conv [4] also does not assume any prior knowledge, but it is limited in finding rotation in a toy task, where it learns rotation angles of rotated images by comparing them with the original ones, which are 7x7 random pixel images.

|  |Augerino [1]|lieGAN [2]|lieGG [3]|L-conv [4]|Forestano et al. [5]|Ours|
|-|-|-|-|-|-|-|
|symmetry generators|completely known|partially known (affine)|partially known (affine)|completely unknown|completely unknown|completely unknown|
|learn what?|transformation scale (e.g. rotation angle)|symmetry generator (rotation)|symmetry generator (rotation)|symmetry generator (rotation)|symmetry generator (in various low-dim settings)|symmetry generator (affine)|
|verified with what?|raw CIFAR-10|rotation MNIST|rotation MNIST|random 7x7 pixel images|toy data (dim <=10)|raw CIFAR-10|
|implicit or explicit|implicit|explicit|explicit|explicit|explicit|implicit|
|how?|optimize while training downstream task|compare fake/true data in GAN framework|extracts from learned NN using Lie derivative|compare rotated and original images|extracts from invariant oracle using Lie derivative|extracts from validity score using ODE integration|

Comparing those research with ours (i.e. without requiring any candidate symmetries, finding implicit symmetries, and using high-dimensional real-world datasets) requires highly nontrivial modification of those methods. For example, LieGAN learns six linear coefficients corresponding to six affine generators, which cannot be extended to ours. Thus, we compare our method with Forestano et al. [5] by learning symmetries in both a high-dimensional image dataset and a low-dimensional dataset using the Lie derivative and ODE integration. The experiment with the image dataset can also be seen as a comparison with LieGG.

**Ablation studies: learning symmetries of images using Lie derivative (comparison with LieGG and Forestano et al.)**

LieGG method extracts symmetries of images from a learned neural network using the Lie derivative and retrieves rotation from the rotation MNIST. The original LieGG assumes affine symmetry; however, for comparison purposes, we have adapted LieGG to our setting by using the Lie derivative instead of ODE integration in our method. As shown in Figure 1, this approach fails to learn the correct symmetries. Note that, to evaluate whether the learned symmetries $\\{V_1,\cdots,V_{10} \\}$ recover the space of true symmetries with an orthonormal basis $\\{L_1,\cdots,L_6\\}$, we follow this process: we first check whether $V_1,\cdots,V_6$ are mutually orthogonal. We then compute the values $\sqrt{\sum_{j=1}^6 \langle V_i, L_j \rangle^2}$ and check whether the first six such values are close to 1.

We describe our intuition on why the Lie derivative fails. The Lie derivatives are taken in the space of images, which are regarded as flattened RGB vectors. We claim that due to the high-dimensional nature of the image data, derivatives are less informative and much noisier than ODE integration. For example, rotating images moves the pixels, requiring the RGB values on the regular grid to be resampled by interpolating neighboring RGB vectors, such as with bilinear interpolation. Hence rotation of images in the space of flattened RGB vectors is a highly complicated and fluctuating non-smooth operation with noisy derivatives.

**Ablation studies: learning isometries of (3+1)-dimension Minkowski space using Lie derivative and ODE integration (comparison with Forestano et al.)**

We further demonstrate that usage of ODE integration is indeed capable of learning symmetries in the setting of Forestano et al.. We learn isometries of (3+1)-dimensional flat Minkowski space by setting the Lorentz metric $t^2 - x^2 - y^2 - z^2$ as a validity score and successfully find the six symmetries (three Lorentz boosts and three rotations), both using Lie derivative and ODE integration, as shown in Figure 2. This experiment shows that our approach works in both low-dimensional and high-dimensional settings, **while using Lie derivatives fails with high-dimensional image dataset.**

[1] Benton et al., “Learning invariances in neural networks from training data.” NeurIPS 2022.

[2] Yang et al., “Generative adversarial symmetry discovery.” ICML 2023.

[3] Moskalev et al., “Liegg: Studying learned lie group generators.” NeurIPS 2022.

[4] Dehmamy et al.,” Automatic symmetry discovery with lie algebra convolutional network.” NeurIPS 2021.

[5] Forestano et al., “Deep learning symmetries and their lie groups, algebras, and subalgebras from first principles.” MLST 2023.

Meta-review: This paper uses neural ODEs (NODEs) to parameterize the infinitesimal generator of continuous symmetry groups.  The NODE is fit by defining and then optimizing a “validity score” that determines if the generated data is valid for the target application.  Experiments are reported on learning affine transformations of images and time translation, space translation, and Galilean boost symmetries for PDEs.  The reviewers had a generally favorable opinion of the work (2 x Weak Accept, 1 x Borderline Accept), with the reviewers appreciating that this is a general technique that can discover non-affine symmetries, whereas most previous work has either been constrained to affine or makes otherwise comparatively stronger assumptions about the group structure.  The two primary criticisms of the reviewers were that (1) the technique relies on defining a validity score, which may be unknown or hard to devise for some applications, and (2) the lack of comparison to baselines in the experiments.  My own assessment of these critiques is that, for the former, using a validity score is the natural price to pay for the lack of assumptions placed on the group and the symmetry type.  In other words, the structure has to come from somewhere, and I think the validity score is a reasonable way to do that.  Regarding the latter, in the rebuttal, the authors have provided a table that very nicely summarizes the relationship to prior work (and justifies the lack of empirical comparisons).  This has convinced me that the lack of baselines is reasonable.  The authors should include this table in the camera-ready version of the paper.
Decision: Accept (poster)