Symmetry Discovery Beyond Affine Transformations
Authors: Ben Shaw
Abram Magner
Kevin R. Moon
Venue: NeurIPS 2024 poster (2024)
Keywords: symmetry detection
isometries
infinitesimal generators
Killing vectors
Riemannian metric
transformation groups
manifold learning
Primary area: interpretability_and_explainability

Abstract: Symmetry detection has been shown to improve various machine learning tasks. In the context of continuous symmetry detection, current state of the art experiments are limited to the detection of affine transformations. Under the manifold assumption, we outline a framework for discovering continuous symmetry in data beyond the affine transformation group. We also provide a similar framework for discovering discrete symmetry. We experimentally compare our method to an existing method known as LieGAN and show that our method is competitive at detecting affine symmetries for large sample sizes and superior than LieGAN for small sample sizes. We also show our method is able to detect continuous symmetries beyond the affine group and is  generally more computationally efficient than LieGAN.

Symmetry Discovery Beyond Affine Transformations
Ben Shaw Utah State University Logan, UT 84322 ben.shaw usu.edu
Abram Magner University at Albany Albany, NY 12222
Kevin R. Moon Utah State University Logan, UT 84322 devin.moon usu, edu
ernigeer@ubbnny.ede
Abstract
Symmetry detection improve various machine learning tasks. In the context continuous symmetry detection, current state the art experiments are limited detecting affine transformations. Under the manifold assumption, framework for discovering continuous symmetry in data beyond the affine trans- formation group. We also provide similar framework for discovering discrete symmetry. We experimentally compare our method to an existing method known as LieGAN and show that our method competitive at detecting affine symmetries for large sample sizes and superior than LieGAN for small sample sizes. We also show our method is able detect continuous symmetries beyond the affine group and is generally more computationally efficient than LieGAN.
outline
Introduction
Approaching various machine learning tasks with prior knowledge, commonly in the form of sym- metry present in datasets and tasks, has been shown to improve performance and/or computational efficiency [
2 3 4. While research in symmetry detection data has enjoyed recent success (5), the current state of the art leaves room for improvement, particularly with respect the detection of continuous symmetry [6]. Herein, we seek to address two issues at hand: primarily, the inability of current methods to detect continuous symmetry beyond the affine group: secondly, the apparent increase of difficulty in identifying continuous symmetries when compared with discrete symmetries.
Our methodology is summarized follows. First, given dataset, certain quantities of interest, hereafter referred to as machine learning functions, are identified, such as an underlying probability distribution which generates the dataset, classification regression function, reduction function. Second, we estimate tangent vector fields which annihilate the machine learning functions. The estimated vector fields themselves generators of continuous symmetries. Thus estimating the vector fields provides an indirect estimate the continuous symmetries. Third, we use the estimated vector fields to estimate functions which, in addition to the machine learning functions, are annihilated by the vector fields. The discovered functions define a feature space which is invariant with respect the continuous symmetries implied by the vector field generators. On this constructed feature space, the outcome of any machine learning task is, by definition, invariant with respect to the symmetries generated by the estimated vector fields, and we show in examples how symmetry-baat based feature engineering can improve performance machine learning tasks.
dimensionality
Vector fields have been used describe symmetry in the context of machine learning before (7]. However, our use of vector fields differs from previous approaches for two reasons. First, we fully exploit the vector fields objects which operate machine learning functions directly. Second, we estimate vector fields associated with continuous symmetries beyond affine transformations. Our main contributions are the following: We show that our method can detect symmetry in a number of cases previously examined with other methods, with our method offering a number of computational advantages. 2) We also show that method be used to detect continuous symmetries beyond the affine group, experimenting with both synthetic and real data.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

2 Related Work and Background
Some background is necessary for work, beginning with general description of symmetry. Throughout, we assume a dataset D = .......,T..)) with T; E RR. Various machine learning functions of interest may exist for given dataset, such an underlying probability distribution or class probabilities. Generally speaking. the object of symmetry detection in data is to discover functions S R + R that preserve machine learning function of interest. That is, for a machine learning function f. o S f. We consider a continuous symmetry of a particular machine learning function to be transformation S which is continuously parameterized, such rotation in a plane by an angle 0, with 0 being the continuous parameter. We consider a discrete symmetry of a machine learning function to be a transformation which is not continuously parameterized, such as a planar rotation by fixed angle. or reflection of point in the plane about a straight line. We primarily deal with continuous symmetry, though discrete symmetry detection is discussed in Appendix D33
Comparatively early work on symmetry detection in machine learning seems to have focused primarily on detecting symmetry in image and video data 8 91, where symmetries described by straight- line and rotational transformations were discovered. Other work has made strides in symmetry discovery by restricting the types of symmetries being sought. In one case, detection was limited to compact Abelian Lie groups [1]] Another case uses meta-learning to discover any finite symmetry group 6 Finite groups have also been used in symmetry discovery in representation learning (T]
Other work has focused on detecting affine transformation symmetries and encoding the discovered symmetries automatically into model architecture. Three such methods identify Lie algebra generators to describe the symmetries, as we do here. For example, augerino [2] attempts to learn a distribution over augmentations, subsequently training model with augmented data. The Lie algebra convolutional network (7). which generalizes Convolutional Neural Networks in the presence of affine symmetries, uses infinitesimal generators represented as vector fields to describe the symmetry.
SymmetryGAN 13 has also been used to detect rotational symmetry [5]
LieGAN appears to represent the current state of the art for continuous symmetry detection. LieGAN is a generative-adversaary network intended to return infinitesimal generators of the continuous sym- metry group of a given dataset [5). LicGAN has been shown to detect continuous affine symmetries, including even transformations from the Lorentz group. It has also been shown to identify discrete symmetries such as rotations by a fixed angle.
None of these related works have attempted detect continuous symmetries beyond affine trans- formations. Continuous symmetry detection is more difficult than discrete symmetry detection [6] since the condition f o S = ʃ must hold for all values of the continuous parameter of S. This is corroborated by the increasingly complex methods used to calculate even simple symmetries such
as planar rotations 12 7 5 Some methods introduce discretization, where multiple parameter values are chosen and evaluated. LieGAN does this by generating various transformations from the same infinitesimal generator [3]. Introducing discretization increases the complexity of continuous symmetry detection, though any reasonable symmetry detection method must establish whether continuous symmetry a finite subgroup of a continuous group has been discovered. We believe a vector field approach addresses the issue of discretization. Our vector based method reduces the required model complexity of continuous symmetry detection while offering means to detect symmetries beyond affine transformations.
We now provide some background on vector fields and their associated flows (1-parameter transfor- mations). We refer the reader to literature on the subject for additional information [1] Suppose that X is smooth] (tangent) vector field on RR::
X = O.
Σ a 0z.
where a² RD - TR. for i E [1, and where (r*)__ are coordinates on R X assigns a tangent vector each point and can also be viewed as function on the set of smooth functions which map Rn to R. E.g. if fr " + R is smooth,
0f
X()
Dzt
'That is. vector tield with C coefficient functions.

---

For example, for n = 2, if f(x,y) = ry and = y²z, then X(f) = =y² X is also derivation on
the set of smooth functions on R that is, for smooth functions f1. IR" + and a1, a1 E R.
( a2f22) = a₁ x + (zx
X(fff2) (f) + f1x(f2).
(3)
These properties are satisfied by derivatives. A flow on R" is smooth function V R x R"
K"
which satisfies
(00 p) p. (ss (t.pp) = ((s+t,p)
(4)
for all s.t E R and for all p E RR. flow is 1-parameter group of transformations. An example of
flow V X R + R is
(t, (z, v)) (x cos(t) - ysin(t), x sin(t) + y cos(t))
(5)
with being the continuous parameter known as the flow parameter. This flow rotates a point (x; y) about the origin by radians.
For given flow Vr one may define a (unique) vector field X given in Equation 2 where each function a² is defined as
NV
a²
(6)
Ot
Such vector field is called the infinitesimal generator of the flow v. For example, the infinitesimal generator of the flow given in Equation 5
-yd, xrgg-
Conversely, given a vector field X as in Equation Consider the following system of differential equations:
2 one may define a corresponding flow as follows.
dx
a'
x'(0) z
di
Suppose that solution x(t) to Equation 7 exists for all E R and for all initial conditions Xg E RR..
Then the function V/ E R + R" given by
(t.) = x(t)
(8)
a flow. The infinitesimal generator corresponding ve is X. For example, calculate the flow of -yO, + zdy, we solve
y, = x
xe (0) To. v() yo
and obtain the flow (t, (Zo, yo)) defined by Equation 5 It is generally easier obtain the infinitesi- mal generator of a flow than to obtain the flow of an infinitesimal generator.
Certain vector fields are particularly significant, having flows which correspond to distance-preserving transformations known as isometries. Such vector fields are known as Killing vectors. See Appendix and B.4 for details.
We can now connect vector fields and flows with symmetry. A smooth function f R" - R is said to be X-iivariant if (f) = 0 identically for smooth vector field X. The function f is iiivaariant if, for all E R. f = f(V(t, (tt.,)) -)) for a flow V. If X the infinitesimal generator V is V-invariant if and only if is X -i-ivvrriant. If the function f is a machine learning function for given data set, our strategy to identify vector fields X; for which ff is invariant. Each vector field X, for which f is invariant is associated, explicitly or implicitly, with flow V. each of which a 1 1-parameter subgroup, the collection of which generate the symmetry group. By identifying continuous symmetries by means of X(f) = 0, continuous symmetry detection is made to be of similar complexity to that of discrete symmetry detection, since no continuous parameters are present.
The price of this indirect characterization of continuous symmetry is often the loss of the explicit identification of the symmetry group, leaving one with only the infinitesimal generators and the symmetry group merely implied. However, infinitesimal generators can be used to construct an invariant machine learning model as follows. Suppose that f RD - R is a (smooth) function of interest for given data set, and suppose that X is a (smooth) vector field for which f is X- invariant. With X fixed, we can consider whether there exist additional functions { that are functionally independent of each other (meaning one function cannot be written as a function of the others), each of which are X-invariant. If such functions can be found, we can define a new feature space using the functions {h'}. That is, can transform each data point in D to
3

---

the point (h' (zz), h2 (x.)..... (z:)) for each point 2; expressed in the original coordinates {z²} }. By definition, the transformed data is invariant with respect to the symmetry group, whether that symmetry group is given explicitly or not.
The flow parameter can often be obtained explicitly well, written as a function of the original coordinates. It is found by solving X(0) 1. Where the flow parameters of vector fields can be estimated, our methodology thus inspires symmetry -baee feature engineering, with the new features consisting of flow parameters and invariant features.
For example, consider data set consisting of points D {2,)i- in R3, where the features of each data point are given in Cartesian coordinates (æ,, y. 2). Now suppose that each data point æ; satisfies the equation f = 0, where f = x² + y² learning function for this data set, since each point in the data set lies on the level set f = O. We note that X yy + x0,, is vector field for which f is X-invariant. With h = x² + y² and h² 2, the functions h' an h are X-invariant. We then construct the data set D Any machine learning task performed in this feature space will be, by definition, invariant to the flow of X, which flow defines the symmetry group of the original data set D.
The function f can be considered to be machine
(z;,, h (x:))
By estimating vector fields which satisfy X() = O, we find seemingly unwanted freedom. This is due to the fact that, assuming (() = 0. hx(()
0 for nonzero function h: the question, then, is which of the two symmetries-the the flow of X or the flow of hX-is "correct." By definition, both are symmetries, so that the function f is symmetric with respect to an apparently large class of symmetries. However, a function is X X-invariant if and only if it is also invariant with respect to hX, so that a set of X X-invariant functions is also set of invariant functions of h.X Thus, where a vector field X can be used to generate X -iinvariant functions, the result is a feature space which is also invariant with respect to h.X This point is further expounded upon in Appendix E
3 Methods
Our method assumes that a given dataset consists of points residing on a differentiable manifold M, and that a machine learning function ʃ is a smooth function on M. We also assume that the symmetry group we seek is Lie group generated by 1-parameter transformations known as flows, the infinitesimal generators of which are vector fields which reside in the tangent bundle of M.
There are three main stages that define our methodology. The first stage, described in Section 3.1 the estimation of machine learning functions specific to a dataset. The second and third steps are described in Section 3.2 In the second step, vector fields which annihilate the machine learning functions are estimated. Finally, the vector fields are used define a feature space which invariant with respect to the implied flows of the vector fields as described in Section 2 A discussion of known limitations of our methods is included in Appendix C
3.1 Estimating Machine Learning Functions
Probability density functions appear in both supervised and unsupervised problems. The output of a logistic regression model or, similarly, neural network model that uses the softmax function, can be interpreted as estimating the probability that given point belongs to particular class. In fact, the probabalistic outputs of many other machine learning models can be analyzed for symmetry, provided that the probability function is differentiable. Invariant features of the symmetries of these functions define level sets upon which the probability is constant. If the flow of an infinitesimal generator can be given, a point in the dataset can be transformed to another point-not generally in the given dataset-and the model should predict the same probability for the transformed point as it would for the original point. Applications include supervised feature engineering and model validation, where predictions on generated points can be used to assess a model's 's ability to generalize.
Particularly the unsupervised case, density estimation can be challenging, complicating methods that require their estimation. We propose to address this potential shortcoming of density-based symmetry detection by a method we call level set estimation. Level set estimation seeks to learn a O for each x; D. Any embedded submanifold
function F: RR" - & (k: < n) such that F(x:) can be, at least locally, expressed as a level set of smooth function [14, and so level set estimation operates on the assumption that the data points belonging to given dataset lie on embedded

---

submanifold. If such a function can be estimated for a given data set, one can estimate symmetries of the component functions f: of F.
Level set estimation can be done as follows, although we note that further development of level set estimation may lead to more effective and/or efficient implementations. First, we construct function/ f as a linear combination of pre-determined features, as with a generalized additive model or polynomial regression, with model coefficients {a=) An m-component column vector w is constructed using the coefficients ag Then, for each point in the dataset, we set the target to 0, so that a function f is approximated by estimating
Bw= = O,
(10)
where B is the feature matrix consisting of m columns and N rows, where N is the numher of points in the dataset, and where O is the zero matrix. OLS regression motivates the way in which we solve Equation (IO), though we note that a solution Wi is equivalent c. We therefore turn to constrained regression, choosing, by default, the constraint mm a = 1. Additionally, we note that any loss function can be used, with predictions and targets determined by the left- and hight-hand sides of Equation 110) respectively.
scaling cwo for some constant
We estimate more than single component function by appending additional columns to w, in which case we constrain the columns of w to he orthonormal. As appending columns to w is likely to increase any cost function used to estimate to, we use "elbow curve" to determine when the number of columns of w is sufficiently high. That is, in the presence of a sudden comparatively large increase in the cost function as the number of columns of is increased, the number of columns of w is chosen as the value immediately before the large increase is encountered. We handle the constrained optimization problem of (00) using existing constrained optimization algorithms 16).
The freedom of expression in the components of the level sets is, unfortunately, not fully contained by constrained optimization. When certain functions chosen part of the level set estimation, such as polynomials, this scaling issue can extend to multiplication by functions. For example, consider data set in R3 known priori to be approximately described by the function F 0, where the components of f1 and f2 of F are given as
= x² v² 1,
f2 = 2 - 1.
(11)
Such level set could also, at first glance, be described by the following equations:
+ 1=
=- 0, x== = 0.
(12)
From a mathematical point of view, the third equation degenerate and does not change the intrinsic dimension of the manifold characterized by this level set, owing to the fact that the rank of this function when x = 0 is 2. However, within the context of learning level sets from data, the extrinsic dimension may be large, and it may not be practical to examine the components of the learned level set. Furthermore, key points for determining the rank of the corresponding function may not be present in the data.
potential workaround for this problem is to incrementally increase the allowed degree of the learned polynomial functions in Equation 12 For the example considered in Equation I1 we can first optimize over the space of degree polynomials, ideally discovering scaling of the function f2- Next, when searching for quadratic components, we can extend the columns of w which correspond to any allowed terms of the form h(,, y, f2 With constrained optimization, the column space of w can then be expressed as orthonormal basis. The optimization routine can then continue, with the final level set model for w not including the artificially-createe columns. Alternatively, one is interested primarily in at most quadratic component functions, one can first optimize using affine functions, then project the data onto the lower- dimensional hyperplane defined by the affine components, subsequently searching for quadratic terms on the reduced space where no additional affine components should be present. We provide experiments which apply these concepts in practice.
One may well ask whether level set estimation can be accomplished using a neural network rather than constrained polynomial regression. Besides the problem of uniqueness just spoken of, a neural network approach would likely require plentiful off- manifold examples so as not to learn a function which is identically zero. A neural network approach to level set estimation is not explored in this paper, though may be explored in the future.
At degree 2, / could be, for example, x. but not x², since x² is not degree 2 polynomial.
5

---

The last type of machine learning function we consider is metric tensor. Given a (pseudo-) Riemannian manifold with a metric tensor, we can calculate infinitesimal isometries of the associated (pseudo-) Riemannian manifold. We provide means for approximating the metric tensor in certain cases in Appendix
3.2 Estimating the Infinitesimal Generators and the associated Invariant Feature Space
With scalar-valued machine learning functions fi, we construct vector- valued function F whose components are defined by the functions fi We can then obtain vector fields which annihilate the components of F by calculating nullspace vectors of J, the Jacobian matrix of F More vector fields may annihilate the functions than can be identified by the nullspace of J, though our method is inspired by this idea. First, we construct vector field whose components are arbitrary linear combination of pre-determined features, as a general additive model, though we primarily use polynomial features herein. The coefficients of this linear combination reside in matrix W, with the columns W corresponding to the coefficients for a single vector field. We then estimate W in the following equation:
MW = O,
(13)
where M = MM, is the Extended Feature Matrix computed using the array I of Jacobian matrices at each point, B is the feature matrix, and O is the zero matrix. The matrix M is computed via Algorithm
Algorithm Constructing the Extended Feature Matrix M
m < number of features
t dimension of space
N < number of points in the dataset
J < Jacobian (aaobiaa(F)(x)) (x:) for i in range(NN) do rown + padded B:)
3-d array of Jacobian matrices at each point
> m(n-1) O's are appended to the 7th row of B
for in range(1,m-1) do
row, +- roll(rowo.) m) end for
Each subsequent row is displaced m entries to the right.
by + -Matria(roouo.... ... rowm mat; 3 Jíb,
Multiply the Jacobian of F at z; by the matrix 6,
end for M + SStckkka eatckveetiaal(mmatt. mattaato.......tt....
This matrix has size nN nm.
As with Equation (TO). solution to Equation selected loss function, for which we turn to McTorch [16]. Using this pytorch-compatible library, one choose from variety of supported loss functions and (manifold) optimization algorithms. As with level set estimation, we sequentially increase the number of columns W until an "elbow" in cost function values is reached.
1 is estimated using constrained optimization of
However, we find the same unwanted freedom as with level set estimation, since X()) = implies that for a smooth function h. hX(() 0. Since we are assuming that the components of X are polynomial functions, is possible that a such a pair X and hXX may exist in the search space. The means of dealing with this unwanted freedom can depend the particular experiment: if there no affine symmetries, for example, vector fields with components degree 2 can be approximated without regard to this issue.
To define a model which is invariant with respect to the flows of pre-computed vector fields X,, we seek features h (separate of the functions f.) such that ((()) = 0. This we do by estimating in the following equation:
(14) where M2 M2[J, B, B) is the Invariant Function Extended Feature Matrix computed via Algo- rithm [ B is the feature matrix of W (already estimated), and is the feature matrix of the unknown function.
M2U O,
In fact, if F is given in closed form, this can be done using symbolic software and without referencing the original data.

---

Algorithm 2 Constructing the Invariant Function Extended Feature Matrix M2
m < number of features
n f dimension of space
N < number of points in the dataset
J(B) (z:) <- Jacobian matrix of features of B at Dq
P size m X n. derivatives of raw features.
J < (z;)]
> Array of Feature Jacobian matrices at each point. b. Extended Feature matrix computed via Algorithm
M = M(J.8)
M₂ MW
This is not O, since the arguments of M are different than in 13
This method assumes that the invariant functions can be expressed in terms of prescribed features, which, in our case, are typically polynomial functions. The quantity e can be estimated from Equation 14 in much the same way that we estimate other quantities, in that we constrain v to be orthogonal, then optimize a selected loss function using McTorrh [16]. We should note that while this method may produce invariant features, they are not guaranteed to be functionally independent. For example, in our polynomial feature space, the vectors corresponding to x and x² are orthogonal, though the functions themselves are not functionally independent.
3.2.1 Comparison of Discovered Symmetries to Ground Trruh Symmetries
To evaluate symmetry discovery methods general, is important to quantify the ability of a given method to recover ground truth symmetries in controlled experiments. For a ground truth vector field
X - f. and estimated vector field X -E f.,,, similarity score between them
defined to be
N (Ja,,AM
sim (on(X,X) X)
(15)
N
[flll 1l
where
(e.. - J J.ddM.
|| - (f.,
and with Ω being defined by the range of a given dataset-herein we take the full range of the dataset. We note that under a change of coordinates induced by a diffeomorphism, the similarity is invariant, owing to the Jacobian which would be present in the transformed definite integrals. Further refinement of this similarity score is left as future work.
4 Experiments
Our first experiment will compare our method to LieGAN in an effort discover affine symmetry. Our next experiment will demonstrate our method's 's ability to identify symmetries which are not affine. The third experiment demonstrates that symmetry can occur in real data, and that our method can detect approximate symmetry in real data. Our last main experiment deals with pre-determined models which are not polynomials-in particular, when the ground truth symmetry is not expressible in terms of polynomial coefficient functions. Additional experiments appear in Appendix D some highlights are symmetry of a induued probability distribution, discrete symmetry estimation, and estimation of infinitesimal isometries.
4.1 A Alline Symmetry Detection Comparison
Our first experiment compares the performance of our method LieGAN, which is the current state of the art in symmetry detection. We use simulated data where the "true" symmetries are known. Our three datasets for this experiment consist of N random samples generated from 2-dimensional normal distribution with mean (1,1) 1) and covariance matrix [ 0 where N [200, 2000, 20, 000}. In each case, target function values are generated as f f= (x 1)2 + 4(y 1)2 for each point (x. v) in the dataset. The function exhibits symmetry: X(f) O. with X = - (4y 4)0, + (x - 1)0,, (or a scaling thereof). the methodology employed by LieGAN, the ground truth infinitesimal generator is expressible in terms of 3 x 3 matrix, with generated transformations acting on the = 1 plane in three dimensions. Thus, the case of LieGAN, the data is generated with mean 1, 1) and covariance given with the upper 2 x 2 matrix as before, with all other values besides the bottom right
7

---

Table 1: Affine Symmetry Detection Comparison on a Gaussian Distribution
N
LieGAN similarity
Our similarity
LieGAN time (s)
Our time (s)
200
0.7636 + 0.2862
1.0000 + 5.6 10-?
2.1717 0.0481
2.1778 + 0.0125
2000
0.9990 + 0.0014
1.0000 + 1.7 10-7
18.1289 + 0.2528
2.6412 + 0.0208
20000
1.0000 + 1.5 10-5
1.0000 3.2 10-s
195.5748 + 2.6080
5.6177 + 0.0585
value being O, which bottom right value is 0.0001 so as to approximately on the plane. Our own method uses the 2-dimensional data.
Our method first employs polynomial regression approximate the function f from the data. Then. with model using affine coefficients for the estimated vector field, we estimate W in Equation (13) using the L1 loss function and the Riemannian Adagrad optimization algorithm [6] with learning rate 0.01 for 5000 epochs.
We compare the results of our method to the results of LieGAN using two metrics, namely similarity to the ground truth defined by Equation (13) and computation time. The results are shown in Table [ and use 10 trials for each N. These results suggest that if sufficient amount of data is given, LieGAN is able to correctly identify the ground truth symmetry. However, our method exhibits a clear advantage for N 200 while offering comparable similarity scores LieGAN for larger sample sizes. Perhaps the most striking feature of Table [ however, is the dramatic difference in time needed correctly identify the symmetry. Our method offers a clear advantage in terms of computation time as the number of samples increases.
4.2 Non-Affine Symmetry Detection Comparison
We now demonstrate the ability of our method to discover symmetries which are not affine. We generate 2000 data points from normal distribution with mean (0, 0) and covariance matrix [] Q.. For each data point, we generate target values according to f = x y². ground truth vector field which annihilates this function is 2y0, + 3x
We first use polynomial regression to approximate f= y². Then, we estimate W in Equation ( using the L, loss function, selecting the Riemannian Adagrad optimization algorithm [[6] with a learning rate of 0, training for 5000 epochs. Our method then obtains a similarity score of 0.9983 + 0.0006, having aggregated across 10 independent trials. Meanwhile, LieGAN obtains similarity score of 0.0340 + 0.0142 on 10 independent trials, being unable to discover symmetries of this complexity.
4.3 Symmetry Detection using Realistic Data
Our next experiment uses real data that is publicly available, which dataset we refer to as the "Bear Lake Weather" dataset 177. The dataset gives daily weather attributes. It contains 14, 610 entries with 81 numeric attributes including the daily elevation level of the lake. The dataset contains precisely 40 years' worth of data from October of 1981 through September of 2021.
We believe an understanding of the behavior of the weather in the past is relevant to this problem. Therefore, we first construct time series of length in dimensions by means of a sliding window of length days: the first time series is constructed using the first 1461 days (the number of days in four years). The next time series is constructed using the second day through day 1462, and so forth.
After converting the raw data to time series data, we apply a transformation on the data meant to extract time-relevant features of the data known the Multirocket transform (18). We select 168 kernels in our transform. The Multirocket transform transforms the data from 13, 149 time series of length 1461 in 81 variables tabular data: 13, 149 entries in 1344 variables.
For such high-dimensional data, we turn to PHATE, a state of the art data visualization method Using PHATE, we reduce the dimension to 2, so that our new dataset has 13, 149 entries in variables. The resulting data appears to approximately lie on a circular shape and is shown in Figure
19
8

---

Embedded Bear Lake Weather Data Colored by Invariant Function Value
0.0003
0.02
0.01
0.0001
0,0000
-0.01
-0.0001
-0.0002
-0.03
-0.02
-0.01
0.00
0.01
0.02
0.03
0.04
Figure 1: The ROCKET-PHATE embedded Bear Lake Weather dataset, colored by invariant function value.
Figure I suggests that the data is largely periodic. In fact, further experimentation reveals that the points for a given calendar year correspond to a traversal around the circular-like shape. Thus, for the analysis of non-seasonal weather patterns, it may be of use to examine features of this embedded dataset which are invariant under the periodic transformation, the approximate symmetry.
Indeed, our method reveals an approximate invariant function given by
0.33592x. + 0.9418999 - 10
This result was obtained using level set estimation, assuming function expressible az? by² using the Mean Square Error loss function with Equation (1O), optimized using Riemannian stochastic gradient descent with learning rate of 0.001 and trained for 5000 epochs. This experiment shows that symmetry can occur in real data, and that our method can detect symmetry and estimate invariant functions for real data.
4.4 Symmetry Detection with Non- Non-Polynomial Ground Truth
This experiment deals with case in which the ground truths not strictly polynomial functions. We generate 2048 numbers x; and 2048 numbers Yj each from UO, 2T). Next, for each pair (r,, yc). obtain 2, by means of z; sin(x:) cos(y:). that ground-truth level set description of the data is given as sin(x) + cos(y) = O.
We first apply our level set estimation method to estimate this level set. We optimize the coefficients of the model
ap + ar + a2y + 032 + a4 cos(x)) + a5 cos(y) + ag cos(z) + ar sin( + ag sin(y) + ag sin(=) =
subject to E.= a3 In light Equation 10 the matrix B has a row for each of the 2048 tuples (z,, Ur. 2), and 10 columns, which columns correspond to the 10 different feature functions in our pre-determined model. The vector w contains all 10 parameters {a, {a.)%
Using the L, loss function and the Riemannian Adagrad optimization algorithm rate 0.01, our estimated level set description is
[6 with learning
0.577372 - 0.57713 cos + 0.57756 = 0,
which is approximately equivalent to the ground truth answer up to a scaling factor.
Having demonstrated that our method easily extends non-polynomial pre-determined features, now wish to examine case in which a polynomial pre-determined model of symmetry does not contain the ground truth vector field. For this, use the same dataset, albeit we define 2₁,

---

that we are approximating symmetries of f(z y) = sin(x) - cos(y). A ground truth vector field
which characterizes the symmetry of is given as
X (yn c0s
Applying our method with pre-determined model of degree 2 polynomial coefficients gives estimated (using Equation
vector field X of
X = -0.7024 - 0.1874r - 0.2203y + 0.0121 +.0242yy 0
+ (-0.5783 + 0.2665x + 0.1236y - 0.0311x 0.01500y 0g.
This result was obtained using the L1 loss function, the Riemannian Adagrad optimizer [66 with learning rate 0.1, training for 5000 epochs. We also note that sim(X, X) = 0.62, using the similarity score defined in Equation (15). This demonstrates that our method can be applied even when a ground truth vector field is not covered by the vector field pre-determined model.
5 Conclusion
While current state of the art experiments focus, in the context of continuous symmetry detection, on affine symmetries, we have outlined a method to estimate symmetries including and beyond the affine symmetries using vector fields. Our method relies on estimating machine learning functions such probability distribution, continuous targets for regression, or functions which characterize an embedded submanifold (level set estimation). Our method also requires, in the case of discrete symmetry detection, that the transformation (sub)-group be parameterized, so that the "best-fit" parameters can be obtained by optimization. In the case of continuous symmetry detection, vector fields are constructed as arbitrary linear combinations of chosen basis of vector fields, and the parameters that are the coefficients of the linear combination are optimized. In both discrete and continuous symmetry detection, the parameter space is often constrained. Herein, we have chosen to express the coefficients of our vector fields in terms of polynomial functions.
When searching for affine symmetries in low dimensions, our method appealing when compared with current state of the art methods. This is due primarily to the relative ease of implementation. Additionally, when compared with the state of the art, our method appears offer computational advantage while at least competing and often outperforming in terms of accuracy, especially for more complex symmetries than affine ones.
Our infinitesimal generators are a step removed from the symmetry group than what is typical of symmetry detection methods. We have proposed building invariant architectures based on the identification of features which are invariant with respect to the discovered vector fields, and we have demonstrated the estimation of these invariant features. This results in an invariant model, since the information fed to the model is approximately invariant with respect to the symmetry group.
We have also discussed potential novel applications of symmetry discovery, including model validation and symmetry- -based feature engineering. Further potential applications are discussed in Appendices Aland D.4 Our novel approach for detecting symmetry in data, the computational advantages of our method, and the demonstrated and potential applications of our method make our method of interest to the machine learning community.
Future work includes the following. First, we plan to extend our method express the vector fields in greater generality beyond polynomial functions. A natural extension of this would be to express the invariant features greater generality as well. Additionally, as discussed the appendices, the detection of isometries relies the ability to compute Killing vectors, for which, in the case of non- Euclidean geometries, no machine-learning compatible methods have been apparently implemented. Also discussed in the appendices is the notion of metric tensor estimation, which we leave, in large part, as work for the future.
Acknowledgments
This research was supported part by the NSF under Grants 2212325 [K.M.], CIF-2212327 [A.M.], and CIF-2338855 [A.M.].
10

---

References
[1] C. Lyle, M. Wilk, M. Kwiatkowska, Y. Gal, and B. Blooem-Reddy, "On the benefits invariance in neural networks," 2020. [Online]. Available: httpes:/aaxii.rrr/////0000000010
[2] D. L. Bergman, "Symmetry constrained machine learning." in Advances in Intelligent Systems and Computing. Springer International Publishing, aug 2019, pp. 501-512. [Online]. Available: https///doi.rg/10010000
[3] S. Craven, D. Croon, D. Cutting, and R. Houtz, "Machine learning a manifold," Physical Review D, vol. 105, no. 9, may 2022. [Online]. Available:
[4] B. Tahmasebi and S. Jegelka, "The exact sample complexity gain from invariances for kernel regression," Advances in Neural Information Processing Systems, vol. 36, 2023.
[5] J. Yang. R. Walters, N. Dehmamy, and R. Yu, "Generative adversarial symmetry discovery," Proceedings of the 40th International Conference on Machine Learning, vol. 202, no. 1647, pp.
39 488 39 508, 2023. [Online]. Available: https://ariivoorg/bb//22222222
[6] A. Zhou, T. Knowles, and C. Finn, "Meea-learning symmetries by reparametterizzaioon CoRR, vol. abs/2007 2es/2007.00233, 2020. [Online]. Available: http://arriv.orryabb220000..0099
[7] N. Dehmamy, R. Walters, Y. Liu, D. Wang, and R. Yu, "Automatic symmetry discovery with lie algebra convolutional network," Advances in Neural Information Processing Systems, vol. 34, pp. 2503-2515, 2021. [Online]. Available:
[8] R. P. N. Rao and D. L. Ruderman, "Learning lie groups for invariant visual perception," Sload Center for Theoretical Neurobiology. 1999.
[9] Sohl-Dickstein,
C. Wang, and B. A. Olshausen, "An unsupervised algorithm for learning lic group transformations," CoRR, vol. abs/1001.1027, 2010. [Online]. Available: http://arxiv.or//bbb// 1027
[10] Cohen and M. Welling, "Learning the irreducible representations of commutative lie groups." 2014.
[11] F. Anselmi, G. Evangelopoulos, L. Rosasco, and T. Poggio, "Symmetry-adapted representation learning," " Pattern Recognition, vol. 86, pp. 201-208, 2019. [Online]. Available: www.:///wwiiiitttoo///eeeeeetaaa 1200
[12] G. Benton, M. Finzi, P. Izmailov, and A. G. Wilson, "Learning invariances in neural networks," 2020.
[13] K. Desai, B. Nachman, and J. Thaler, "Symmetry discovery with deep learning," " Phys. Rev. D, vol. 105, p. 096031, May 2022. [Online]. Available: https///inkaap..rrddo//0.......0 http:///ddo.....111///////ee [05.096031]
[14] J. M. Lee, Introduction to Smooth Manifolds. Springer New York, NY, 2012.
[15] J. Townsend, N. Koep, and S. Weichwald, "Pymanopt: A python toolbox for optimization on manifolds using automatic differentiation," Journal of Machine Learning Research, vol. 17, no. h7m.
137, p. 1-5, 2016. [Online]. Available: http:///mm.oor//papers/////
[16] M. Meghwanshi, P. Jawanpuria, A. Kunchukuttan, H. Kasai, and B. Mishra, "Metorch, a manifold optimization library for deep learning," arXiv preprint 2018.
Tech. Rep.,
[17] B. D. Shaw, S. Jarman, B. Bean, K. R. Moon, W. Zhang, N. Butler, T. Bolton, A. Knight, E. Haroldsen, A. Funk, and R. Higbee, "Supplementary files for: 'interactive modeling of bear lake elevations in a future climate"," " 2024. [Online]. Available: https:/diggaaalommonn..uu...... www.snsuuuuuue.... edu/all datasets/230
[18] C. W. Tan, A. Dempster, C. Bergmeir, and G. I. Webb, "Multirocket: Effective summary statistics for convolutional outputs in time series classification." CoRR, vol. abs/2102.004577, 2021. [Online]. Available: https//aarii..rrgaa///0....0
[19] K. Moon, D. van Dijk, Z. Wang. S. Gigante, D. Burkhardt, W. Chen, K. Yim, A. Elzen, M. Hir, R. Coifman, N. Ivanova, G. Wolf, and S. Krishnaswamy, "Visualizing structure and transitions in high-dimensional biological data," Nature Biotechnology, vol. 37, pp. 1482 1492, 2019. [Online]. Available: https://doi.or// 10. 11tp:/00000000011111000000000000

---

[20] A. R. Gover and T. Leistner. "Invariant prolongation of the killing tensor equation." Annali di Matematica Pura ed Applicata (1923 -). vol. 198, no. 1, p. 307-334, Jul. 2018. [Online]. Available: http://dx.doi.org/// 10.110007///11111118-800000
[21] N. Miolane, A. L. Brigant, J. Mathe, B. Hou, N. Guigui, Y. Thanwerdas, S. Heyder, O. Peltre, N. Koep, H. Zaatiti, H. Hajri, Y. Cabanes, T. Gerald, P. Chauchat, C. Shewmake, B. Kainz, C. Donnat, S. Holmes, and X. Pennec, "Geomstats: A python package for riemannian geometry machine learning," 2020.
[22] I. Anderson and C. Torre, "The differential geometry software project," 2022. [Online]. Available: https:/diggialcommonn. edu/dg/
[23] A. Pinkus, "Weierstrass and approximation theory," Jourmal of Approximaation Theory, vol. 107, no. 1, pp. 1-66, 2000. [Online]. Available: http:///wwwsseeeecccccccccoccccc
$0021904500935881
[24] A. M. Horst, A. Hill, and K. B. Gorman, palmerpenguins: Palmer Archipelago (Antarctica) penguin data, 2020, r package version 0.1.0. [Online]. Available: https:
Wallisonhorst Vallisonhorss.github.oooooo wtwphnub.oopplmmmppnnngiii
[25] Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825-2830, 2011.
12

---

A Killing Vectors, Isometries, and Metric Tensor Estimation
Certain vector fields have flows corresponding to isometric transformations. i.e. transformation which preserves manifold distances. Such infinitesimal isometries are called Killing vectors, named after Wilhelm Killing. The search for Killing vectors can be done in much the same way as with ordinary vector fields, although instead of searching the parameter space defined by arbitrary polynomial coefficients, one constructs an arbitrary linear combination of the Killing vectors of the space of interest, optimizing the coefficients.
However, this method requires knowing the Killing vectors for the space of interest. In general, Killing vectors can be calculated if Riemannian (or Aseudo-Riemannian metric is given [20]. In machine learning, data with continuous-valle but on the Riemannian manifold R" equipped with the Euclidean metric. Within this context, the Killing vectors form proper subspace of the generators of the affine transformations, and are thus obtainable by previous methods. Nevertheless, our methods have the following advantages over existing ones. First, our methods can detect the infinitesimal generators of affine transformations and, as our experiments suggest, may offer a large computational advantage in certain situations to existing methods. Second, the assumption that suitable Riemannian metric on the data space is the Euclidean metric appears to be an assumption only. Generally, inner products of (tangent) vectors are computed using the dot product without consideration of structure which may naturally exist in the data. In fact, one discovered symmetry of LieGAN is an infinitesimal generator corresponding to a subgroup of the Lorentz group [5] which generator is an isometry of the Lorentz metric and not the Euclidean metric. In that particular context, the Euclidean assumption is replaced with the assumption that the poseudo-Riemannen relation to physic Thyss This point is nontrivial, since the decision of which Riemannian metric to endow differentiable manifold with impacts not only which vector fields generate the isometry group, but also suggests the way in which (tangent) vectors should be "dotted."
features are usually assumed not only to lie in R",
metric is the Lorentz metric, which assumption is appropriate given the
We are then led to the question of what assumptions can be made about the form of the Riemannian metric for a data space modeled as lying on a Riemannian manifold. This question is not fully addressed herein, though we will describe practical scenario in which the Riemannian metric can be assumed to be non-Euclidean.
Consider a data set D whose data points lie in R". There are several dimensionalliy-redueee techniques that may seek to map points in D to form a data set D whose points lie in R", where m < n. These techniques are thought to provide a mapping &b : 12" + ""." However, such an algorithm provides bijective mapping from the finite set {a;} (z; D) to the finite set { (T; E D). Therefore, we can seek to learn functions (²) via regression, then use the learned functions as the components of a function θ Rmm -> RR.. Insofar as the Euclidean metric assumption is valid on R", we can obtain a metric y on I by means of the so-called "pullback" of θ, then calculate the Killing vectors of g: this allows one to calculate the infinitesimal isometries of a distribution or level set on RT the components of which will not generally be affine functions.
Obtaining a metric tensor by means of θ operates under the assumption that the data points x; lie on manifold of dimension m embedded in RR, and that the coordinates of the data points o; are suitable manifold coordinates, at least locally. The strength of the model metric tensor thus depends on the extent to which these assumptions are valid. An example of relatively strong model would be to begin with a data set in R", construct new data set D consisting of points R. which points are projected onto the first m principal components, and then let D be the data set constructed by writing each point in 'D in the basis defined by the m principal components. An example of a relatively weak model would be to use the coordinates (æ, v) to describe points in D, where the points in D lie on the surface of a sphere in IR3.
We should make a couple of points clear at this point. First, if one wishes, the Killing vectors (any smooth vector field, for that matter) of the metric tensor g can be expressed as vector fields on : that is, in terms of the original coordinates. In the original coordinates, the flow of any Killing vector will represent an affine transformation, since the metric on 8" is the Euclidean metric, begging the question of whether the symmetry should have been estimated in the original coordinates where only a search among affine symmetries is needed. One potential reason for this is due to the curse
For full details, we refer the reader to [5]
13

---

of dimensionality; as our estimation of infinitesimal generators uses (constrained) regression, the performance of our methods may suffer in higher dimensions, suggesting that, in certain cases, may be more practical to estimate more complicated symmetries in lower dimensions than simpler symmetries in higher dimensions. However, this point requires experimentation in addition to the experiments conducted herein.
The second point we wish make clear that the procurement of Killing vectors for a given metric is no small task. In fact, generic Riemannian metries admit few, if any, Killing vectors: the defining equations for Killing vectors form an overdetermined system of differential equation On that point, while the python library geomstats [21] offers the ability to estimate the metric tensor using function such as O appears that the estimation of Killing vectors is not yet supported. Thus, our experiment which seeks to identify isometries as described in this section uses symbolic software to compute the
Killing vectors [22]
Metric tensor estimation has the potential to make a large impact on the way in which certain aspects of machine learning are approached. Apart from adjusting the metric tensor according to transformations as discussed above, metric tensor and Killing vector estimation could impact manifold alignment. In general, manifolds whose points are put into bijective correspondence (in a smooth manner) should admit the same number of Killing vectors. Estimating the infinitesimal isometries may thus help to bring added clarity manifold aligmment. And, of course, a choice of mon-Euclidee metric tensor will have a large impact on the way inner products and manifold distances are computed.
B Similar Methodology for Discrete Symmetries
While much of this paper focuses on the identification of continuous symmetry, we also present an analogous method by which discrete Shantity-preserving transformations can be identified. Given a quantity of interest denoted by the function f. a transformation $ is said to define a symmetry of the data if fx;; = f((())) for each point in the data set. For example, if f = y x² and the equation f is satisfied for each point in the data set, the data set symmetric under a reflection about the line x = O, which is discrete transformation.
To estimate discrete symmetries, we assume that symmetry belongs to a parametric family of transformations, which parameters we denote by w. We define, for each uy in a given dataset,
ú = ((x)) w) fzz),
y4 = 0,
(16)
and we seek to minimize L(g., 34) for a smooth loss function L. typically by means of constrained optimization. For example, we consider the 2-dimensional task of determining a line of reflection about which a function f(z y) is symmetric. The reflection of a point (z, y) about a line in the form of a. by can be written
1
a²
1
2ab a²
y( a² 6 22b
(17)
3--
y
Minimizing L(i yt) for smooth loss function then estimates the parameters (a, b) which character- ize the line for which f(x, y) is approximately reflectively symmetric. Since the line aæ + by = 0 equivalent to car + cby- 0 for some constant c, our method best accomplished by means of constrained optimization, which, in this example, is easily accomplished by means of a² + b2 = 1.
For continuous symmetries, invariant quantities beyond those found be of significance to the data set itself-the functions ht-were used to define an "invariant feature space." " That is, a set of features were given such that when the data points were expressed in terms of these features, any machine learning task performed on this new feature space would necessarily be invariant with respect to the given symmetry group. For discrete symmetry detection, the transformations can be given explicitly, so that cxisting methods for building invariance into model that require the symmetry group to be given beforehand can be employed. However, it may also be pessible-perhaps depending on the discovered symmetries-to calculate features h that are invariant with respect to the discovered discrete symmetries. For example, the function h y is invariant to reflection about the x = line.
'We should also note that efforts to compute Killing vectors without solving the Killing equations directly have been successful 20
14

---

The identification of features which are invariant with respect to fixed collection of symmetries would allow one transform points from the original feature space the invariant feature space, so that any machine learning task would be invariant with respect to the given symmetry group.
We also note that our assumption that the discrete symmetries sought originate in parametric families of transformations is limiting. In particular, despite previous work conducted with permutation groups 66) it does not appear that our method can be applied to permutation groups.
C Limitations
Our method is an improvement to state of the art methods for continuous symmetry detection, both for computational reasons and due to the fact that our method can detect continuous symmetries which are not affine transformations. However, there are limitations to our approach. First, as we have mentioned previously, is the issue which arises when vector fields X and fX are both present in the search space. We have suggested that symbolic approach may resolve this, though non-symbolic approaches such as ours must grapple with this issue. Our approach of searching for polynomials of lower order first partially resolves this issue. Another promising resolution is to restrict the search space of vector fields to particular types of symmetries, such as isometries, which is experimented with in Appendix D.4
Another limitation is that our method requires an assumption about the coefficient functions of the vector fields. Herein, we primarily use polynomial functions, and although polynomials are universal approximators [23], the number of polynomial terms for a given coefficient function can increase rapidly, particularly when the number of dimensions is high. We believe this limitation is best resolved, as with the previous limitation, by restricting the search space to particular types of symmetries.
Though our method comparatively efficient, the reliance on the elbow curve to determine both the number of discovered symmetries and, when level set estimation is used, the number of coefficients in the level set, is limitation. This limitation is of a more persistent nature than the previous limitations, appearing in analogous form as the dimension hyperparameter in manifold learning methods, as the number clusters in k-means clustering. is possible that our method would be improved by another means of selecting this number, however.
We also mention the limitations inherited via the choice of optimization of equations (13) and [4].
Cost function selection, optimization algorithm selection, hyperparameter selection, together with the potential for poor parameter initialization all constitute limitations our method. We believe the optimization task is much more ttraihht-forart than training current state of the art symmetry detection models, however, as other methods require fine-tuning neural networks.
Lastly, our notion of similarity of vector fields, while adequate for our experiments, is limited. Primarily, when a collection of vector fields is estimated and taken to constitute the infinitesimal generators for the symmetry group, it is not clear how to compare the estimated collection of vector fields ground truth collection. The similarity score also depends on the range of given dataset, and the aggregation of component-wisg wise similarities via the arithmetic mean is naïve, leaving room for further refinement. In particular, in the presence of a metric tensor g, one may seek to define the
similarity score by means of E (X. where the inner product (X, XX)g is defined by the metric on the tangent space of the manifold, and where the expected value is computed by means of the underlying distribution of the data. However, exploration of such refinements are left as future work.
D Additional experiments
D.1 Invariant Feature Construction
We have claimed that, given vector fields X,, one can find certain number of features (x) such that X,,h,) = O. For a single vector field, it is generally true that n - 1 functionally independent functions can be found, where n is the dimension of the space. However, when more vector fields are considered, the number of functions invariant with respect to all vector fields decreases. Additionally, it is not clear whether, for given machine learning task, it better to identify invariant features or features which correspond to the flow parameters of the vector fields: for example, the angle 0
15

---

10
10
33
09
-15
-15
2.0
-2.
-55
-1.0
-0.5
5.0
85
1.5
2.0
-20
-1.5
-0.5
50
05
1.5
Figure 2: A visual representation of the kernel density estimate of the circular dataset.
Figure 3: A representation of the esti- mated invariant function the symme- try of the circular probability distribu- tion.
corresponds the flow parameter of rotational transformation, while the radial feature is invariant-a given machine learning task may favor one above the other.
Despite the apparent challenges with both estimation and interpretation, we will estimate an invariant feature in the simplest possible example: we will try to recover the radial feature for circular dataset. The data is generated by uniformly sampling 0 randomly on [0, 2xr. Then, we choose x cos(() and = (0). The probability distribution is estimated using kernel density estimation, and single vector field is estimated by applying X O. where p is the probability distribution, and where the components of X are assumed to be affine. Finally, we optimize Xhh) = 0, where we assume takes the form
h= a=a² + a2y² + Qyty + agæ + O5y + 46.
with ==1 of 1. After estimating h, we plot grid of function values of h. The result is pictured alongside the visualization of the estimated probability distribution of the data (Figs. and
D.2 Level Set Estimation
We now seek find an invariant function of circular dataset using level set estimation. alluded to previously, certain datasets may have an underlying distribution which is not symmetric, and yet data points themselves may lie on an embedded surface which does exhibit symmetry.
We return again to the circle, though we initially embed the circle in three dimensions. We sample 2000 values from a standard normal distribution, then obtain angles 0 via 0 t,mod2n, We then obtain 3-dimensional data by means of the following:
T = cos(O),
(0), =
(18)
As noted previously, level set estimation with polynomials can lead to degenerate expressions for the level set. Therefore, initially restrict our search space to affine functions. We optimize according to sequuennial first taking the number of (orthonormal) columns of W to be 1, then 2. The chosen loss function value with the number of columns being is approximately 7 10-12 while the loss function value with two columns is approximately 7 10-2 Therefore, we choose W to have one column, which approximately represents the following function, ignoring the and y coefficients, which are less that 10-5;
0.70710 + 0.70710
(19)
We wish continue looking for suitable functions that, when combined with 19 describe the manifold on which the data lics. Thus, will expand search space to quadratic functions. Before doing so, in an attempt to avoid discovering equations degenerate with respect to [9 we will project the 3-dimensional data onto the space defined by our discovered alfine functions-in this case, the approximate (x, y) plane (though technically = = 1). plot of the result is obtained in 4 We then apply 10 on the reduced space, with quadratic features. The discovered component is given
0.5779y² 0.0044zy + 0. 0.0007y 0.0018x 0.5770 0,
(20)
16

---

8%
2.0
10
09
3.0
-15
-1.00
2.0
-1.00
-0.5
-0.50
-0.25
0.03
0.25
0.50
575
140
-20
-1.5
-0.5
000
0.5
1.5
Figure 4: The circular dataset, not uni- formly distributed, projected onto the (x, y) plane.
Figure 5: A visual representation of the function values of the estimated invariant function of X as given in 21
bearing striking resemblance to the ground-truth expression x:2 + y² 0.
In the search for infinitesimal generators which annihilate both 19 and 20 accordance with 13 we perform four (relatively fast) searches: first, among vector fields with constant components; second among vector fields with linear components (no constant terms); third, among vector fields with affine components; fourth among vector fields with quadratic components. In this case, since the effective dimension of the dataset we limit our searches to those where the number columns W 1. Our best result is obtained in the second case, and the estimated vector field is
X = (-0.00662 + 0.7047y) 0, + (-0.7095x + 0.0038y)0,.
(21)
which is similar to the expected outcome of -yd, + 2d.. Lastly, search for function / such that XJ) = O, assuming / can be approximated by quadratic polynomial (with no constant term). Our estimate is given by
0.0088x 0.0886y 0.7392x 0.0185ry 0.6673y²,
(22)
to which the reader can compare with x:2 y A plot of the function values of f is given in
appears that Figure 5 is more faithful to the ground-truth invariant function than We do believe this indicates the inferiority of symmetry detection using probability density estimation. Rather, we believe this may have been caused by the estimation of gradients of the probability distribution: the gradients of the level set functions were computed exactly. This may indicate that parametric approach to density estimation may allow for better approximation of gradients.
D.3 Discrete Rotation
This experiment motivated by the simulated discrete rotation experiment given in [5]. Conveniently. code is provided to reproduce the results of their experiment, which entails the generation of 20, 000 points (x, y, z) resulting multivariate normal distribution centered the origin. Then, target values for a hypothetical regression task are generated via the following function for a fixed integer k:
f(z, 2) 1 + +arctan(1// mod 2 kk)
(23)
The task with this dataset is correctly predict the angle corresponding to the alleged discrete symmetry: an integer multiple of 2n /k. In [5] k = 7 is chosen, so we will do the same. Our methods require that we estimate f from the data, then parametric description of the symmetry group, after which one can estimate the parameters to approximate suitable group element. However, the function given difficult estimate via regression. Thus, we will estimate the density of the dataset using kernel density estimation, using the function values weights. As kernel density estimation is known to perform poorly more than two dimensions, we will restrict the size of the experiment by setting 1.
17

---

-2
Figure 6: Pictured are the 20,000 points generated by replicating experiment of (5] restricted to two dimensions.
Figure 7: A visualization of the density estimated using weighted kernel density estimation.
Though we mentioned using the function values as weights for the density estimation, we actually apply - f (clement-wise), that the differences in weights more exaggerated, This changes the function to be estimated, but the symmetries f and f are thought to be similar
We plot the data in Figure 6 and Figure 7 gives visualization of the weighted kernel density estimation. Notably, the density estimation has "smoothed over" the jump discontinuities of the original function.
With the density p(a; y) estimated, experiment, we will search only among the planar rotations, so that the group can be parameterized by the angle 0:
parameterize the transformation group. In this
5()) = cos ()
sin (0)
(24)
sin (0) cos (0)
We thus minimize the following function L for the parameter 0, subject to 0 > x/6:
(())= (P(S -
(25)
where P is the vectorization of p. We choose 0 > π / to find transformations away from trivial 0 O. Our optimization, performed using python's scipy package, yields result of 0 Rz 0.8972, which is 2 to four decimal places, and our similarity to ground truth is 1.0000. The total computation time, including the estimation the density, is approximately 48 seconds.
We repeat the experiment using only 1000 data points, recovering 0 ~ 0.8333, in approximately 0.2 seconds. This yields similarity to ground truth of 0.9979. On the other hand, when the amount of data for the experiment with LieGAN is moderately reduced, the model performance suffers greatly. The intended generator is
0
1
0
0
(26)
0
0
1
while the obtained generator is
0.0869
-0.0244
0.1371
0.2666
-0.4933
-0.4303
(27)
0.0841
0.1345
0.0377
The similarity these two generators is 0.1956. Thus, our approach seems to have greater potential handle discrete symmetry with smaller datasets in two dimensions, in addition having friendlier interface.
'In fact. identifying the symmetries of probability distribution p(x) using our methods is equivalent to identifying the symmetries of pxx)/(x)), provided that a function / can be given such that whenever X (p) = 0, X (/) = 0.
18

---

D.4 Infinitesimal Isometries
We will now conduct a controlled experiment detect symmetries beyond the complication obtainable by current state of the art methods. Specifically, we will detect infinitesimal isometries: that is, Killing vectors. We set up the experiment by creating an artificial data set D in I we name the coordinates u, v, and w. This data set is created by drawing three random samples of size 2 (one sample for each of the three coordinates) from 1)) Then, we create data set 'D with four features, namely y, and I, by applying the following function for each data point in D:
u² + +o² - w, Next, we generate a list of function values, one for each point in D, according to f 9u² + u² + wt the goal of the experiment is to recover infinitesimal isometries of f.
1= 2u..
(28)
The experiment begins under the assumption that the starting data set is D, and that the data set D has been produced from D by means of a dimensionality reduction algorithm. The computation begins by learning functions æ(u, 0, w), y(u, v, w), z(u, e w). and i(u, v, w)) with polynomial regression. This we do, with results varying only, it seems, due numerical approximation error, so that take 28 to be our estimation.
We will need to compute the Killing vectors for the 3-dimensional space, for which we will need the Riemannian metric. We assume that the metric in the original 4-dimensional space is the Euclidean metric, and we use 28 to pull back this Euclidean metric. The result can be expressed the following matrix in the coordinates (u, v, w):
4u² + 5
4ue
-2u
g =
4wn
4n
-2 22
(29)
2u
-2
We have computed this metric using Maple [22]. even though the geomstats package for python 2I appears to be able to perform this, primarily because geomstats does not appear to currently support the calculation of Killing vectors. We calculate a basis for the Killing vectors in Maple, and we note the six basis elements be
X1 (u= -2
w) 0, + (2u² + 2uv² -2w + 5u) Oy..
(30)
X₂ (u² + 1²2 10--o ww) 0. (2u2r u + 203 2vw v) Ow:: X5 = 0, + 2vdw.,
(31)
X3 = 8
X4 = - v.. + 5ud,, + Sund
X6 = -, + 2u0,.. (32)
We then return to python to learn f via polynomial regression. The result of this regression task is almost exactly f 9u² + 222 + w, so that we take this as our function for which we will seek to estimate symmetries. To identify the symmetries, we perform the following constrained optimization task:
6
6
min
(f).
Σ
11
(33)
This is equivalent to the formulation given in 13 with the feature matrix being defined by the components of the known Killing vectors and with W having six rows and a single column. We handle this particular task using McTorch [16), which is manifold optimization library compatible with pytorch. The result is a similarity score of 1.0000 with respect to the ground truth of
dj 02 = a3- = as = ag = 0, a1 ==
(34)
Thus, our methods have discovered the correct infinitesimal isometry of f in the coordinates (u, v, w).
D.5 10-Dimensional Reduction using Level Set Estimation
We will now conduct an experiment for a dataset with 10 features. The goal of this experiment is to apply level set estimation to reproduce the correct number and complexity of components of the ground truth level set function. First, we sample N = 26 numbers [t+ from a uniform distri- bution on [-2 2]. Then, three additional independent samples are made of the same size: {a: J {us N 1, and f2, N. Thus, the dataset D {(tr, x,, Vr, 2)) NN,, is hypercube of dimension 4. Then, a 10-feature dataset 'D is constructed as follows for each point (1, U, 2) E D:
21 t D2 2, Σ3 =
I4 2, 25 = 2t,
(35)
19

---

Bloow curve for the rumber of offine components
Fihowr the nimber cundratie
0.43
0.0
0.30
0.30
025
3 0.00
0.10
043
003
680
components
Figure 8: An elbow curve for selecting the number of affine components for the 10-dimensional simulation.
Figure 9: An elbow for selecting the number of quadratic components for the 10-dimensional simulation. value of is placed at 0 on the horizontal axis for easier visual comparison of the rela- tively small value obtained by selecting one component.
26 x 22 t, I7 I7= 4, 2g 0, 3g = 2, 210
The equations given in 35 could be used to define function θ R R. allowing Riemannian metric tensor to be given on Rd by pulling back the Fuclidean metric tensor on IR much like what we demonstrated in a previous example. However, our goal here is level set estimation. Our dataset, by construction, admits two different and equally valid level set descriptions. The first is given by the following equations:
I5s
2x1
0,
I6 22 23 + 0, 27 47-4- 0,
2g
0,
Zg 21 1 =0
210
0.
(36)
Five of the equations given above define affine functions. If this was known from the data, one could project the data onto the hyperplane defined by these equations, obtaining, in coordinates (x11, 22, 33, I4, ns), the level set description of
36
33
0.
(37)
We begin applying level set estimation by assuming that all component functions are affine. Beginning by identifying approximate affine components is recommended in effort to avoid obtaining degenerate equations when expanding the search space to quadratic components. In any case, we iteratively optimize according to I0, constraining W to be orthogonal matrix, starting the number of columns of W at 1, then increasing by steps of 1 to the number Our optimization routine uses the MSE loss and (Riemannian) gradient descent with a learning rate of 0.01. A plot of the cost function values as a function of the number of columns of W given in Figure g From this figure, evident that the elbow occurs at 5 columns.
Again, to avoid degenerate components, we use the affine components discovered in the previous step to project onto the associated 5-dimensional plane. On the new 5-dimensional space, apply level set estimation, looking now for quadratic components. This optimization routine uses the loss function and (Riemannian) Adagrad optimizer with learning rate of 0.01. A plot of the number of columns of W is given in Figure 9 It seems to be most reasonable to select as the number of approximate quadratic components. The single quadratic component defines a level set estimate of the reduced data space.
D.6 Invariant Features of a Model-Induced Class Probability Density
In a classification setting, certain models can output class probability predictions for given points.
the distribution function differentiable, we may also seek vector fields which annihilate this
function, allowing us to give the symmetries. In this experiment, we analyze the symmetries of
model-induced probability distribution arising from the Palmer Penguins dataset (24). This dataset
consists of three distinct classes of penguin species with various features, with the total number of
20

---

Palmer Penguin Density Map
Penguin Invariant Feature Map
bill depth (normalizzdd lizee!
bill depth (nomallized)
0-
-2-
-2-
-4
-2 bill length (normalized)
-4
-2
2
bill length (normalized)
Figure 10: A visual representation of the SVM-induced probability that a test point is labeled "Adelie".
Figure 11: A representation of the esti- mated invariant function of the symmetry of the SVM-induced distribution.
observations being 344. We retain the features called "bill length" and "bill depth," and we train support vector machine using scikit-learn [25] We take as our probability density the model-induced probability that test point will be predicted as belonging to class "Adelie." For reference, 80% of the data was used to train the model, and the accuracy of the model when applied to the test set was 0.94.
plot of the probability density is given in Figure [0] From the probability density, we estimate a vector field with quadratic components for which the density is invariant. Then, we use the estimated vector field to learn approximate polynomial invariant function. A plot of the estimated invariant function is given in Figure [T] Visually, the contour lines of the probability distribution approximately align with contour lines of the estimated invariant function, which function estimated as 0.4335z - 0.901ly. The feature defined by this estimated function could be used in subsequent machine learning tasks. Suppose, for example, that we encountered additional data points with the task of classifying whether the points belonged to the class "Adelie." One could reduce the dimension of the data 1, with the single feature being given by the approximate invariant function, which feature would be highly correlated with the probability function for that class.
D.7 Symmetry-Assisted Regression
The symmetries of a distribution shape on which data lies may correspond with another machine learning function such as target values in a regression problem. In this experiment, we will show that symmetry-baue -based feature engineering can improve model performance on a regression task. The data for this experiment has been analyzed previously for the purpose of discovering discrete rotational symmetry [5) though we set 2 for the purpose of demonstrating symmetry-informer feature engineering. We also search for discrete rotational symmetry for this dataset in Appendix D.3
We generate 2000 data points from 2-dimensional standard normal distribution. For each data point, function values are generated according to
1
(38)
1 arctan(y/x mod 2T. 27)
Using scikit-learn (25), we train a random forest regressor on this data using a random train/test split with the test size proportion being 0.2. The R2 scores for this experiment are given in Table 2
Without consideration of targets, the dataset is approximately rotationally symmetric. We apply kernel density estimation on the unlabeled data, then test for symmetry of the estimated distribution. The estimated infinitesimal generator is given by
(0.015 + 0.0069r - 0.6867y) + (-0.0012 + 0. 0.7267. + 0.0080y)
(39)
We estimate the invariant feature for this infinitesimal generator using our method. The flow parameter is approximately the polar angle, since the vector field given in Eq. 39 has similarity of 0.9994 to the vector field corresponding to rotation the plane about the origin. We thus transform the original dataset to coordinates (E, (), where & is the estimated invariant feature and where 0 is the
21

---

Table 2: R2 Scores for Symmetry-Enhanced Regression with Random Forests
Dataset
R score (Train)
R² score (Test)
Original
0,9488
0.5965
Transformed
0.9975
0.9965
polar angle. We train random forest model using the transformed data, and our results given in Table 2 Based on the obtained R² scores in table appears that the regression task is more easily accomplished in the symmetry-basee feature space than in the original feature space.
E A Note about the Freedom Present in Identifying Vector Fields
Given a vector field X and a vector field fX, a function h is X-invariant if and only if it is fX- invariant. If h is X X-invariant, the flow of X is symmetry of h, so that the flow of fX is also symmetry of h: this follows from our discussion of vector fields in section 2
For example, circle centered at the origin, characterized by F= 0 where V(r, = x² + p2
for some real number r, exhibits rotational symmetry described by X = yy + zŪ,- However, (F) = 0, so that the flow of the vector field - + = ayy where defined, also a symmetry of the circle.
This may seem to introduce a theoretical problem requiring greater care when detecting non-affine symmetry. However, our proposed method of constructing models which are invariant with respect to the symmetry group requires only the identification of functions which are invariant with respect the vector fields. Thus, both X and f f.X are valid answers for "ground truth" symmetries, since function h is X -iivvariant if and only if it is f f X-invariant.
We note that this lack of uniqueness does present a challenge when estimating suitable vector fields. When estimating the symmetries via constrained regression, X and fX may or may not both be present in the search space. As mentioned in section 3 one potential workaround is owing to the fact that symmetry estimation can be done symbolically. Consider the example of F = 0 given previously in this section. The Jacobian matrix of F is given as
f2x
2y
22).
and a basis for the nullspace of J (using functions as scalars) is
{
y
which vectors correspond to tangent vectors -yd, + 2.gy and -20, + y0z, respectively. The flows of these vector fields correspond with rotations (about the origin) in the (a., y) and (y. planes, respectively. However, only two vector fields can be recovered using this method, which in this case has neglected another symmetry, namely =z0, + x&2, corresponding to rotations in the (x., z) plane. In fact, each of the three vector fields corresponding rotations about coordinate axis are isometries of the sphere described by F We should note, however, that the generator for rotations in the (z, 2) plane can be expressed linear combination of the other rotations using functions as scalars:
(- xd,) + x 20, yd.) =0,- + x... y
y
Therefore, it is plausible that applications exist in which a symbolic approach continuous symmetry discovery may be sufficient. Our approach does not rely on symbolic software, however.
We also mention that the issue of unwanted freedom can be addressed by identifying "special" vector fields and reducing the search space of vector fields to linear combinations of these special vector fields known as Killing vectors. This would eliminate the uniqueness problem and is experimented with in Appendix D.4 This approach is a special case of symmetry detection where only isometries be detected.
Another point which may help to address concerns about our handling of the difficult nature of non-affine symmetry detection is in the relationship between the flows of X and fX generally. The
22

---

trace of the flow of X through the point is characterized by the level set h; = Ct, where {hr is complete of X X-invariant functions. Since function X X-invariant if and only if is f Xiiivarriant, complete set of invariant functions for f'X can be taken to be {h;) without loss of generality. Thus, the level set = G also characterizes the trace the flow of fXX through the point p. This argument assumes that the flows of X and fX, as well as the vector fields themselves, are well-defined in an open neighborhood about the point p.
In fact, in our example above with F = I + 32 - r², the flow of - + 20,, assuming > 0, is given
(t. y)) = x t, t, V2 + 2tr - f2
which trace is (part of a) circle (we note that the flow parameter is x). The trace of this flow through a point is equivalent to the trace of X through the same point.
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we claim to be able to estimate non non-affine symmetries, which we do in theory and application. We also show that our method compares favorably with the current state of the art method of detecting symmetries, showcasing easier implementation and faster computation times.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss that level set estimation limited in that it may return degen- erate set of component functions. We also discuss that our method cannot guarantee that discovered invariant features are functionally independent. We give additional limitations in separate limitations section in the appendices.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattio, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper describes the information needed reproduce equivalent results for each experiment, including data sources or methods used to synthetically generate the data.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
25

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code used for the main experiments is available at https://ggiii //gtuhbb com/ /tt///vvinnooaaalaee SymmetryML Additionally, all the data we have generated for the main experiments is made available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/ lttips..ccppublicuuu.eeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper explains hyperparameter selection where appropriate in the Experi- ments section. The paper also indicates when and how data is split.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Where applicable, our experimental results contain error bars. This is indicated in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper indicates that the experiments were conducted a personal
puter. Computation times are also given, where applicable to particular experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: There ethical concerns with work. The datasets source made available to the public. Moreover, the datasets are cited in the paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses ways in which our work will lead to improved machine learning models. We do not anticipate any negative impacts from work.
Guidelines:
27

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve the use or release of models or data that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification; The software used in this paper cited throughout the paper. The data sources were also cited. All code and data were properly respected, being used only to obtain our experimental results.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
28

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, palasets.papersstt paperswi thcode. com con/dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: All simulated datasets are described in detail.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research did not involve human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper has no associated study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "4fPh3HdYzK", "reviewer": "Reviewer_S4uK", "summary": "The authors are trying to solve a challenging problem to discover symmetry for given data, regarding such symmetry may include non-affine transformations. They observes one-parameter family of symmetric transformations can be represented as a vector field. In the proposed method, they first find machine learning functions (level set estimation) for given data, and then find vector fields which annihilate the machine learning functions.", "questions": "Discuss the following relevant reference:\n\n* Desai, Krish, Benjamin Nachman, and Jesse Thaler. \"Symmetry discovery with deep learning.\" *Physical Review D* 105.9 (2022): 096031.\n\n\n\nThe cosine similarity metric used in Section 4 is computed by coefficients of the pre-determined model which determine the overall vector field rather than an average of cosine similarities for vectors at each point, right? If so, how about the following case: In Sec. 4.2, even if we got a vector field $2y f \\partial_x + 3x^2 f \\partial_y$, then it is also a correct vector field describing symmetry of given data. If you are trying to propose a metric to evaluate non-affine symmetry, then your metric should produce zero value also for this case. Please discuss about this circumstance.\n\n\n\nWhich pre-determined model for vector field are you using in Sec. 4.2? It must be clarified.\n\n\n\nThe result in Equation (16) seems able to be also produced by LieGAN, since the equation indicate an affine symmetry. Why are not you comparing with LieGAN?\n\n\n\nIssues on clarity of exposition.\n\n* Attach color bars for Figures 1-2\n* It would be better to clarify to which values $B$ and $w$ in Eq. (1) and $M$ and $W$ in Eq. (13) corresponds for each experiment in Section 4.\n\n\n\nMinor typoes\n\n* Line 61: \"exampe\" -> \"example\"\n* Equation (2): $\\partial_{x^i}$ -> $\\partial x^i$ at the denominator", "limitations": "Following limitations should be emphasized, if my understanding is not wrong.\n\n* Based on pre-determined model of one-parameter symmetries, this work find the most suitable one-parameter symmetry among the pre-determined model", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors address an interesting problem: symmetry detection beyond affine transformations. Their observation to connect transformations and vector fields technically sound.", "weaknesses": "While the title of the manuscript is quite general, their method seems to highly rely on choice of pre-determined models for machine learning functions (Sec. 3.1) and vector fields (Sec. 3.2). However, no experiment addresses impact of choices of those pre-determined models.\n\nIn addition, all of the experiments assume we already know a proper pre-determined model of vector fields. The case that pre-determined parametric model of vector fields does not cover the GT vector field used for data generation is not discussed.\n\nTo address the challenging problem about non-affine symmetry, we should more carefully handle it. Please see the question about the cosine similarity in \"Questions\" below.\n\nResults of affine symmetry detection (Sec. 4.1) seem worse than LieGAN. While LieGAN produces results which clearly converge to GT in terms of both bias and variance, but the proposed method does not in terms of both. The proposed method have benefit at low samples and speed, but this contribution seems too marginal. For non-affine symmetry experiment (Sec. 4.2), experimental details are missing. I cannot consider it is a good result unless I can see more case studies about different choices of pre-determined models.\n\nThese weaknesses make me doubtful about results and benefit of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We have responded below to the concerns raised.\n\n**Pre-determined models**: We believe that we are the first to detect continuous symmetries in the context of machine learning beyond the affine group, despite work on the subject spanning decades. Our method of estimating vector fields is regression with general additive models, where the coefficients of the model features are constrained: we specialize to polynomials for the purpose of illustration and better comparison with existing methods for symmetry detection. We have showcased examples in which polynomial-based models are most appropriate, and although experiments which make use of other types of smooth functions can be devised, the only aspect of our method which changes in these cases is the components of the feature matrix used to estimate the coefficients. Therefore, our method can be trivially adapted to non-polynomial models. Later in this response, we will present the results of an experiment which uses non-polynomial functions.\n\nA suitable pre-determined model for the detection of manifold symmetries would rely on the nature of the manifold itself. However, it appears that although the manifold assumption is widely used in machine learning, very little work has been done to characterize the manifolds on which data is assumed to lie. Therefore, our choice to specialize to polynomial-based models is no less valid than other specializations. Additionally, polynomials can be used to approximate any continuous function [R1] and thus, in theory, can characterize any continuous symmetry, at least locally.\n\n**Covering the GT vector field**: If the degrees of the polynomial functions of a vector field are sufficiently high, the vector field can approximate a suitable ground truth vector field. We also reiterate that current SOTA methods, if recharacterized in terms of vector fields, would assume that the coefficients of the vector fields are degree 1 polynomials. Our presentation of symmetry detection in terms of vector fields opens up a great many possibilities for types of functions used, although we use polynomials for better comparison with existing methods and due to their universal approximation property [R1].\n\n**LieGAN comparison**: For every value of $N$, our method usually outperformed LieGAN for most trials. However, the average score of our method was brought down by outlier trial runs for our method, likely due to poor initialization of the model parameters. However, such a shortcoming could be easily overcome by fine-tuning. In the spirit of a fair comparison, we did not perform any (albeit simple) hyperparameter tuning for our method, since we did not perform any hyperparameter tuning for the LieGAN method.\n\nIt is also evident that the error bars for the scores in both methods overlap, except for $N=200$ where our method vastly outperforms LieGAN. Thus, the experimental results seem to suggest that our method is comparable to LieGAN in terms of accuracy, except when $N$ is low. Ultimately, the purpose of this experiment was to show that our method competes with SOTA in terms of accuracy when detecting affine symmetries, while offering a computational advantage. However, using the median as the estimate and the IQR for the error bars gives the results in Table 1 of the rebuttal attachment.\n\nWe now present an additional experiment that uses both linear terms and a few sinusoidal terms as a pre-determined model. First, we generate 2048 numbers $x_i$ and 2048 numbers $y_j$, each from $U(0,2\\pi)$. Next, for each pair $(x_i,y_i)$, we obtain $z_i$ by means of $z_i = \\sin(x_i)-\\cos(y_i)$, so that a ground-truth level set description of the data is given as $z-\\sin(x)+\\cos(y) = 0.$\n\nWe first apply our level set estimation method to estimate this level set. We optimize the coefficients of the model\n    \\begin{equation*}\n        a_0 + a_1 x + a_2 y + a_3 z + a_4 \\cos(x) + a_5 \\cos(y) + a_6 \\cos(z) + a_7 \\sin(x) + a_8 \\sin(y) + a_9 \\sin(z)=0\n    \\end{equation*}\n    subject to $\\sum_{i=0}^9 a_i^2 = 1$. In light of Eq. (10) in the paper, the matrix $B$ has a row for each of the 2048 tuples $(x_i,y_i,z_i)$, and 10 columns, which columns correspond to the 10 different feature functions in our pre-determined model. The vector $w$ contains all 10 parameters $\\\\{a_i\\\\}_{i=0}^{9}$.\n    \nUsing the $L_1$ loss function and the (Riemannian) Adagrad optimization algorithm with learning rate $0.01$, our estimated level set description is\n\n\\begin{equation*}\n        -0.57737 z - 0.57713 \\cos(y) + 0.57756\\sin(x) = 0,\n\\end{equation*}\nwhich is approximately equivalent to the ground truth answer up to a scaling factor.\n\n**Relevant reference:** We will include a brief discussion of the paper by Desai et al. We note that LieGAN outperformed this work in their experiments, and thus a comparison is unnecessary.\n\n**Details about Section 4.2 experiment**:  For this experiment, the search space for level set estimation was limited to cubic polynomials, while the search space for vector fields was limited to quadratic coefficients. In this setting, the only valid symmetries are constant multiples of the given ground truth symmetry. It is a good point that $fX$ may not have a favorable cosine similarity. Thus, it seems that the cosine similarity metric can only be applied where the issue of uniqueness can be controlled, as in this particular example. We will discuss this limitation in the revision. The purpose of this experiment is to show that our method can detect non-affine symmetries, where current SOTA cannot. See also our overall rebuttal for further discussion.\n\n[R1] A. Pinkus, “Weierstrass and approximation theory,” Journal of Approximation Theory, vol. 107, no. 1, pp. 1–66, 2000.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hPkWjI0pLW", "reviewer": "Reviewer_fxa1", "summary": "The paper uses standard ideas from differential geometry to find symmetries in datasets. \nThe procedure is the following:\n1. Estimate a parameterization of the dataset (what the authors call machine learning functions). This step looks like manifold learning.\n2. Find a vector field under which the machine learning functions are zero.\n3. Find a coordinate system for the invariant space for the vector field in 2.\n\nThough the employed techniques are not exactly the same, the ideas remind me of this paper https://arxiv.org/abs/2008.04278", "questions": "In addition to the questions above:\n- What is the rationale behind assuming that the h functions are polynomials? Could this be done by implementing the h functions with MLPs?", "limitations": "The paper doesn't explicitly state the technical limitations of their approach (see weaknesses). The paper could be improved significantly by stating the mathematical assumptions under which the algorithms work.\n\nAdditional discussions on the dependence on the dimensionality of the data, dimensionality of the symmetry group, number of samples needed, etc, would improve the paper as well. Even if just empirical.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The problem the paper addresses is very interesting and can have many applications.\n- Several numerical examples are presented.", "weaknesses": "- The paper should state the assumptions under which the algorithms work. \n  - In particular it seems that one necessary assumption for the first step is that the data lies in a manifold that can be parameterized by a single chart (the f). Is that correct or can this assumption be bypassed somehow?\n  - Another seemingly needed assumption is that the group of symmetries one can learn is a 1-parameter group. If one has a 2-parameter group then it would be given by another vector field, and it is not obvious how to make sure that the 2 vector fields are compatible. Is this a necessary assumption too?\n\n- The paper mentions that the techniques they develop work also for discrete groups, and it gives a not very detailed discussion in Appendix B. I don't see how that's the case. Say that the data has a symmetry with respect to an unknown action of an unknown permutation group. How can this method find it? It is not obvious how the algorithm would work in this case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We have responded below to the concerns raised.\n\n**Assumptions**: The single chart assumption is not necessary. Consider $f(x,y,z) = -x^2-y^2+z^2-1$. The surface $f=0$ is a hyperboloid of two sheets. We can estimate a continuous symmetry of this hyperboloid--a rotation in the $(x,y)$ plane--though a single coordinate chart cannot be given. Level set estimation does not deal with coordinate charts, which is a benefit of using a level set to characterize an embedded manifold.\n\nWe are also not assuming that the symmetry group has dimension 1. We do assume that the group has 1-parameter subgroups. There may be several 1-parameter subgroups, which our method can handle. The discovered vector fields are compatible in the sense that they admit a common set of invariant functions. This is described in lines 112-120, where we seek vector fields (each representing a 1-parameter subgroup) which annihilate the machine learning functions.\n\n**Discrete transformations**: If one can express the discrete transformation parametrically, this method can be applied to discrete transformations. Many examples stem from continuous symmetries with a fixed continuous parameter, such as a rotation in the plane by a fixed but arbitrary angle $\\theta$, where $\\theta$ is the parameter to be optimized.\n\nAnother example is a 2-d reflection about a straight line passing through the origin. Such a line can be characterized by the equation $ax+by=0$, and a formula for the reflection can be written as\n\n\\begin{equation*}\n        S(x,y;a,b) = \\dfrac{1}{a^2+b^2} \\begin{bmatrix}\n            a^2-b^2 & -2ab\\\\\\\\\n            -2ab & b^2-a^2\n        \\end{bmatrix}\n        \\begin{bmatrix}\n            x\\\\\\\\\n            y\n        \\end{bmatrix}.\n\\end{equation*}\nOptimizing the parameters $a$ and $b$ in $f(S(x)) = f(x)$ for some function $f$ would yield the \"best fit\" line of reflection under which $f$ is symmetric.\n\nPermutations may not be expressible parametrically, and we will state this explicitly as a limitation of the method, since previous work with permutation groups has been done and is of interest.\n\n**Polynomials vs. MLPs**: Since MLPs are universal approximators (as are polynomial functions), our method can be applied to MLPs as well. The main reason MLPs are not used here is because of the difficulty in using them for level set estimation: the MLP would need to output 0 for every input and yet not be the zero function. Meanwhile, we assume that the invariant functions are polynomials primarily for two reasons. First, in the case of affine symmetry, the invariant functions are typically expressed in terms of polynomials. Secondly, polynomials can approximate smooth functions with an arbitrary degree of accuracy [R1] and are easy and transparent to manipulate.\n\n**Additional Discussion**: We will include a discussion on the number of parameters needed to estimate the symmetries in dimension $m$ using degree $n$ polynomials. This point is discussed in response to reviewer uBWo. The number of parameters relates to the dimension of the symmetry group and can also provide insight on the number of samples needed.\n\n[R1] A. Pinkus, “Weierstrass and approximation theory,” Journal of Approximation Theory, vol. 107, no. 1, pp. 1–66, 2000.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m7SeW0C3k8", "reviewer": "Reviewer_uBWo", "summary": "The paper presents a method for continuous symmetry detection under the manifold assumption.\nCrucially, the symmetries that are discovered by this method can extend beyond the affine ones.\nThe method is tested and compared against the state of the art (LieGAN), and is found to outperform it in scarce data regime, and be competitive in the large data regime.", "questions": "Can we get a discussion of the computational complexity/scaling of this method? Seems like there are combinatorial explosions hiding just beneath the surface of the method. A proper discussion of this may result in an increase in the rating from this reviewer.", "limitations": "The authors have properly discussed the limitations of their model in terms of the experiments they have run. Not enough has been said in terms of the various null space estimations, which require knowledge of adequate “basis functions”, which can not just be polynomials in most cases.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The paper connects symmetry to invariance of dynamical systems to particular functions. Instead of explicit identification of the symmetry, the focus is on identifying the infinitesimal generators of the flows which correspond to the symmetries themselves. Once found, the vector field can be used to create symmetry-invariant features.\n* The exposition is overall very clear. Many examples are provided throughout.\n* The paper outlines a (sometimes) computationally cheap alternative to methods such as LieGAN.", "weaknesses": "* Depending on the machine learning function of interest, the method requires level set estimation, which is done by assuming that level sets themselves can be found by linear combination of polynomials. This seems to be very optimistic\n* The elbow method to identify the right amount of polynomial terms to keep seems a bit brittle\n* The scaling of a polynomial based method is very unfavourable beyond very toy-like examples.\n* More generally, who chooses the feature functions that form the basis for the linear combinations?\n\nIf these weaknesses are addressed, the rating from this reviewer may increase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We have responded below to the concerns raised.\n\n**Use of Polynomials**: In general, level set estimation can be applied using linear combinations of any set of smooth functions, though our experiments do specialize to polynomial functions. However, smooth functions can be approximated by polynomials of sufficiently high degree and are thus universal approximators [R1].\n\n**The elbow curve**: Instability can arise, and the results of the elbow curve can be misleading. This seems to primarily occur when the problem of degenerate expressions, discussed first in lines 182 - 204, is not properly controlled. For level set estimation, if both $f=0$ and $hf=0$ are discoverable expressions based on the pre-defined search space (for example, $x-y=0$ and $x^2-xy=0$), the loss function values may not generally follow an elbow curve, and any perceived elbow may not occur at the correct number of components for level set estimation. We discuss some workarounds for this problem, both for level set estimation and for vector field estimation, in lines 182-204, as well as 226-231.\n\n**Scaling beyond toy examples**: Vector fields with polynomial coefficients can characterize highly complex and non-trivial symmetries. Especially due to their universal approximation properties [R1], we expect that many symmetries that occur in the real world are expressible or approximately expressible in this form. Additionally, our method advances the current SOTA methods, as current SOTA methods consider a more restricted class of symmetries. Thus, we believe our contribution is an important advance. See our overall rebuttal for more details on our contributions. See also our response to reviewer U1Sk for an experiment on real data.\n\n**Feature function choice**: The features are chosen, in general, based on the types of symmetries sought. For example, if only affine symmetries are being sought, the feature functions for the vector fields are chosen to be affine functions. If no assumption on the form of the symmetries can be made, one can choose the features to be arbitrary polynomials, due to their universal approximation property [R1].\n\n**Computational complexity**: The computational complexity of polynomial regression is comparatively low, especially in a single dimension [R2]. Moreover, manifold optimization algorithms on the matrix manifolds used herein are also of low computational complexity, with a recent manifold optimization algorithm obtaining computational complexity of $\\mathcal{O}(\\log(T)/\\sqrt{T})$, where $T$ is the number of iterations [R3]. In contrast, current SOTA methods employ large neural networks, which networks are known to take more computational resources to train.\n\nThere is no combinatorial explosion in our method. In fact, we can obtain a precise count of the number of parameters our method requires. Each vector field in $m$ dimensions using polynomials of degree $n$ has the following number of coefficients:\n\n\\begin{equation*}\n\\dfrac{(m+n)!}{n!(m-1)!}.\n\\end{equation*}\n\nThe coefficients can increase quickly, particularly in higher dimensions, but for every fixed value of $m$, the number of coefficients increases as a polynomial function of $n$ rather than combinatorially. However, each coefficient in our model increases the power of our method to detect symmetries. In contrast, while SOTA methods employing large neural networks must have parameters like ours which characterize the symmetries, they also have additional parameters. The number of parameters in our method increases according to the number of vector fields in a basis of our search space, whereas in methods such as LieGAN, these parameters are required in addition to other parameters present in the neural network architecture.\n\nTo address the potential issue of having a “large” number of parameters, which number is dwarfed by the number of parameters present in neural networks that take $m$ inputs, we suggest limiting the search space to infinitesimal isometries: Killing vectors, that is. We discuss this restriction further in Appendix A.\n\n\n[R1] A. Pinkus, “Weierstrass and approximation theory,” Journal of Approximation Theory, vol. 107, no. 1, pp. 1–66, 2000.\n\n[R2] L. Li, “A new complexity bound for the least-squares problem,” Computers \\& Mathematics with Applications, vol. 31, no. 12, pp. 15–16, 1996.\n\n[R3] H. Kasai, P. Jawanpuria, and B. Mishra, “Riemannian adaptive stochastic gradient algorithms on matrix manifolds,” 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c4GankgJPw", "reviewer": "Reviewer_U1Sk", "summary": "This paper proposes a method for finding a transformation that is invariant to a given function. The transformation is restricted as governed by a single parameter, so it is written as a flow described by a specific vector field. The method estimates the vector field by solving the polynomial regression. The authors evaluated the performance on three toy datasets and a classification dataset.", "questions": "1. What is the transformation class that is (theoretically) handled by the method?\n1. How did you solve Eq. (10), (13), (14) in the experiments?\n1. What would happen if x contains some noise?\n1. When does the proposed method fail?", "limitations": "Limitations are not provided.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. A flow-based symmetry detection is novel and an interesting research direction.", "weaknesses": "1. There is a large room to improve the presentation. First of all, the terminology seems not standard in ML (e.g. I've never heard the term \"machine learning function\"). Second, the contents are not self-contained --- some unfamiliar notions appear without explanation (e.g. elbow curve). Third, it is not clear what the proposed method actually does. For example, it is said that the method solves Eq.(10), but it is not likely that we can always solve it. I mean, for some data, the equation may not have a solution. Such a case is not mentioned in the paper.  \n2. The capabilities and limitations of the proposed method are not fully mentioned. \n3. The experiments are mainly on toy datasets and are not strongly convincing that the method is applicable to real problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We have responded below.\n\n**Terminology**: With no intent to confuse the reader, we used the term \"machine learning function\" to cover very general types of functions which may appear within the context of machine learning. Such a function could be a regression function, a classification function, or a function offering a manifold description of given data, as in level set estimation or in the presence of a metric tensor.\n    \nThe elbow curve notion refers to a relatively sudden increase in function values, perceived as an elbow shape, since the function values immediately preceding the elbow point are usually close together. This is sometimes referred to as finding a \"kink,\" as in [R1] within the context of selecting the \"best\" number of clusters $k$ for $k$-means clustering.\n    \nIn general, Eq. (10) may not have a solution, as is the case with OLS regression. Eq. (10) is estimated/fitted by constrained optimization of a selected loss function. We will clarify this in the revision.\n\n**Capabilities and Limitations**: We have clarified many points regarding the capabilities and limitations of our method in our response to the reviewers. We will include a discussion of these points in the revised version.\n\n**Only using toy datasets**: Current SOTA papers in symmetry detection commonly use simulated data. An experiment with simulated data allows for the identification of ground truth symmetry, allowing one to quantify the accuracy of a given method. Symmetry detection on real-world data can be applied, but it is harder to evaluate the ability of new methods to properly detect symmetry if ground truth symmetry is not known, as is generally the case with real data. Thus our experiments are consistent with other SOTA papers in this area.\n    \nNevertheless, we present an additional experiment on real data that we will include in the revision. This comes from a publicly-available dataset that deals with weather for four decades in the vicinity of Bear Lake, Utah. The dataset, along with a report describing how the data was sourced, is publicly available [R2].\n\nThe dataset gives daily weather attributes. It contains $14,610$ entries with 81 numeric attributes including the daily elevation level of the lake. The dataset contains precisely 40 years' worth of data from October of 1981 through September of 2021.\n\nWe believe an understanding of the behavior of the weather in the past is relevant to this problem. Therefore, we first construct $13,149$ time series of length $1461$ in $81$ dimensions by means of a sliding window of length $1461$ days: the first time series is constructed using the first $1461$ days (the number of days in four years). The next time series is constructed using the second day through day 1462, and so forth.\n\nAfter converting the raw data to time series data, we apply a transformation on the data meant to extract time-relevant features of the data  known as the Multirocket transform [R3]. We select $168$ kernels in our transform. The Multirocket transform transforms the data from $13,149$ time series of length $1461$ in $81$ variables to tabular data: $13,149$ entries in $1344$ variables.\n\nFor such high-dimensional data, we turn to PHATE, a SOTA data visualization method [R4]. Using PHATE, we reduce the dimension to $2$, so that our new dataset has $13,149$ entries in $2$ variables. The resulting data appears to approximately lie on a circular shape and is shown in Figure 1 of the rebuttal attachment.\n    \nFigure 1 in the rebuttal attachment suggests that the data is largely periodic. In fact, further experimentation reveals that the points for a given calendar year correspond to a traversal around the circular-like shape. Thus, for the analysis of non-seasonal weather patterns, it may be of use to examine features of this embedded dataset which are invariant under the periodic transformation, the approximate symmetry.\n\nIndeed, our method reveals an approximate invariant function given by \n    \n\\begin{equation*}\n    f(x,y) = 0.33592 x^2 + 0.94189 y^2 - 2.9743 \\cdot 10^{-4} = 0.\n\\end{equation*}\n\nIn Figure 2 of the rebuttal attachment, we replot the embedded data, colored this time by the value of the approximate invariant function. This experiment shows that symmetry can occur in real data, and that our method can detect symmetry and estimate invariant functions for real data.\n\n**Transformation class**: The transformations our method can handle are 1-parameter subgroups of transformation groups (Lie groups). We will clarify this in the revision.\n\n**Solving method**: Solutions to Eqs. (10), (13), and (14) are estimated using constrained regression. The specific loss function and hyperparameters vary by experiment, but we will clarify these details in the revision.\n\n**Noise**: Noise is common in regression problems, and it affects the quality of least squares estimates. We have noise present in the experiment that compares our method with LieGAN, since the dataset is not technically rotationally symmetric, but rather approximately rotationally symmetric.\n\n**When our method fails**: It may fail if the number of independent parameters exceeds the number of datapoints. The success of the method depends on optimization and loss function choices, as well as hyperparameters. We will clarify these points in the revision.\n\n[R1] T. Hastie, R. Tibshirani, and J. Feiedman, ``The Elements of Statistical Learning: Data Mining, Inference, and Prediction,'' 2nd ed. Springer, 2009.\n\n[R2] B. D. Shaw et al., “Supplementary files for: ‘interactive modeling of bear lake elevations in a future climate’,” 2024.\n\n[R3] C. W. Tan, A. Dempster, C. Bergmeir, and G. I. Webb, “Multirocket: Effective summary statistics for convolutional outputs in time series classification,” CoRR, abs/2102.00457, 2021.\n\n[R4] K. Moon et al., “Visualizing structure and transitions in high-dimensional biological data,” Nature Biotechnology, vol. 37, pp. 1482 – 1492, 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all of the reviewers for their very helpful comments. We first wish to emphasize here our novel contribution to symmetry detection, particularly in light of current SOTA methods. To date, the most successful method of detecting continuous symmetry is LieGAN, which is a large neural network that can successfully detect affine symmetries in low dimensions. In contrast, our method uses polynomial regression, with constrained coefficients, to detect not only affine symmetries, but symmetries of much higher complexity. The computational advantage alone, owing to our novel use of vector fields, offers a highly non-trivial advantage to current SOTA methods. Moreover, we appear to be the first to detect any continuous symmetries beyond affine transformations using only training data. The current SOTA method (LieGAN), after decades of work on the subject, uses a GAN to detect rotational symmetry in two dimensions, where such a transformation is likely the most rudimentary of all symmetries, from a mathematical perspective.

Our experiments use polynomial regression, though we note that our method can be trivially adapted to accommodate different functions. We use polynomials not only for better comparison with current SOTA, (which, if presented in terms of vector fields, would assume linear polynomial vector field components) but also due to the universal approximation property of polynomials, as polynomials can approximate any smooth function to an arbitrary degree of accuracy [R1]. We have also included an experiment which uses a non-polynomial basis in response to a specific reviewer's concern for this.

We also wish to address a technical question relating to the potential need for careful handling of non-affine symmetry detection. Given a vector field $X$ and a vector field $fX$, a function $h$ is $X$-invariant if and only if it is $fX$-invariant. If $h$ is $X$-invariant, the flow of $X$ is a symmetry of $h$, so that the flow of $fX$ is also a symmetry of $h$: this follows from our discussion of vector fields.

For example, a circle centered at the origin, characterized by $F=0$ where $F(x,y) = x^2+y^2-r^2$ for some real number $r$, exhibits rotational symmetry described by $X=-y\partial_x + x\partial_y$. However $\frac{1}{y} X (F) = 0$, so that the flow of the vector field $-\partial_x + \frac{x}{y} \partial_y$, where defined, is also a symmetry of the circle.

This may seem to introduce a theoretical problem requiring greater care when detecting non-affine symmetry. However, our proposed method of constructing models which are invariant with respect to the symmetry group requires only the identification of functions which are invariant with respect to the vector fields. Thus, both $X$ and $fX$ are valid answers for "ground truth" symmetries, since a function $h$ is $X$-invariant if and only if it is $fX$-invariant.
    
This lack of uniqueness does present a challenge when estimating suitable vector fields. When estimating the symmetries via constrained regression, $X$ and $fX$ may or may not both be present in the search space. As we allude to on page 5, symmetry estimation can be done symbolically, and this would eliminate the possibility of both $X$ and $fX$ appearing in the set of discovered vector fields. A non-symbolic technique is also discussed, with the challenge being partially addressed in lines 226-231, though this issue may also warrant a discussion in a new ``limitations'' subsection of the methods section.

We also briefly mention that this issue can be addressed by identifying "special" vector fields and reducing the search space of vector fields to linear combinations of these special vector fields known as Killing vectors. This would eliminate the uniqueness problem and is experimented with in Appendix A. This approach is a special case of symmetry detection where only isometries can be detected.
        
Another point which may help to address concerns about our handling of the difficult nature of non-affine symmetry detection is in the relationship between the flows of $X$ and $fX$ generally. The trace of the flow of $X$ through the point $p$ is characterized by the level set $h_i=c_i$, where $\{h_i\}$ is a complete set of $X$-invariant functions. Since a function is $X$-invariant if and only if it is $fX$-invariant, a complete set of invariant functions for $fX$ can be taken to be $\{h_i\}$ without loss of generality. Thus, the level set $h_i=c_i$ also characterizes the trace of the flow of $fX$ through the point $p$. This argument assumes that the flows of $X$ and $fX$, as well as the vector fields themselves, are well-defined in an open neighborhood about the point $p$.
    
In fact, in our example, the flow of $-\partial_x + \frac{x}{y} \partial_y$, assuming $y>0$, is given as $\Phi(t,(x,y)) = \left(x-t, \sqrt{y^2+2tx-t^2} \right)$, which trace is a (part of a) circle (we note that the flow parameter is $-x$). The trace of this flow through a point is equivalent to the trace of $X$ through the same point.


[R1] A. Pinkus, “Weierstrass and approximation theory,” Journal of Approximation Theory, vol. 107, no. 1, pp. 1–66, 2000.

Meta-review: This paper introduces a novel method for detecting symmetries in data using vector fields, extending beyond traditional affine transformations. The approach is innovative and addresses an important problem, with potential for significant impact. While one reviewer raised concerns about the clarity of the presentation and reliance on toy datasets, other reviewers acknowledged the method's strengths, particularly its effectiveness in small-sample scenarios and its computational efficiency compared to existing methods like LieGAN. Despite the presentation issues, the paper's contribution is valuable, and I recommend accepting it.
Decision: Accept (poster)