Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery
Authors: Yoshitomo Matsubara
Naoya Chiba
Ryo Igarashi
Tatsunori Taniai
Yoshitaka Ushiku
Venue: Submitted to ICLR 2023 (2023)
Keywords: symbolic regression for scientific discovery
physics
datasets
benchmarks

Abstract: This paper revisits datasets and evaluation criteria for Symbolic Regression, a task of expressing given data using mathematical equations, specifically focused on its potential for scientific discovery. Focused on a set of formulas used in the existing datasets based on Feynman Lectures on Physics, we recreate 120 datasets to discuss the performance of symbolic regression for scientific discovery (SRSD). For each of the 120 SRSD datasets, we carefully review the properties of the formula and its variables to design reasonably realistic sampling range of values so that our new SRSD datasets can be used for evaluating the potential of SRSD such as whether or not an SR method can (re)discover physical laws from such datasets. As an evaluation metric, we also propose to use normalized edit distances between a predicted equation and the ground-truth equation trees. While existing metrics are either binary or errors between the target values and an SR model's predicted values for a given input, normalized edit distances evaluate a sort of similarity between the ground-truth and predicted equation trees. We have conducted experiments on our new SRSD datasets using five state-of-the-art SR methods in SRBench and a simple baseline based on a recent Transformer architecture. The results show that we provide a more realistic performance evaluation and open up a new machine learning-based approach for scientific discovery. We provide our datasets and code as part of the supplementary material.

Under review as a conference paper at ICLR 2023
RETHINKING SYMBOLIC REGRESSION DATASETS AND BENCHMARKS FOR SCIENTIFIC DISCOVERY
Anonymous authors Paper under double-blind review
ABSTRACT
This paper revisits datasets and evaluation criteria for Symbolic Regression, task of recovering mathematical expressions from given data, specifically focused on its potential for scientific discovery. Focused set of formulas used in the existing datasets based on Feynman Lectures on Physics, we recreate 120 datasets to discuss the performance of symbolic regression for scientific discovery (SRSD). For each of the 22 SRSD datasets, we carefully review the properties of the formula and its variables design reasonably realistic sampling ranges of values that new SRSD datasets can be used for evaluating the potential of SRSD such as whether not an SR method can (re)discover physical laws from such datasets. As an evaluation metric, we also propose to use normalized edit distances between a predicted equation and the ground-trute equation trees. While existing metrics are either binary or errors between the target values and an SR model's predicted values for given input, normalized edit distances evaluate a sort of similarity between the ground-truth -ouundttuthhha and predicted equation trees. We have conducted experiments on our new SRSD datasets using five state-of-the-art SR methods in SRBench and a simple baseline based we provide a more realistic performance evaluation and open up a new machine learning- based approach for scientific discovery. We provide our datasets and code as part of the supplementary material.
recent Transformer architecture. The results show that
INTRODUCTION
Recent advances in machine learning (ML), especially deep learning (DL), have led to the proposal of many methods that can reproduce the given data and make appropriate inferences on new inputs. Such methods are, however, often black-box, which makes it difficult for humans to understand how they made predictions for given inputs. This property will be more critical especially when non-ML experts apply ML to problems in their research domains such physics and chemistry.
Symbolic regression (SR) is the task producing a mathematical expression (symbolic expression) that fits a given dataset. SR has been studied in the genetic programming (GP) community (Hoai
et al. 2002 Keijzer 2003 Koza & Poli 2005 Johnson 2009 Uy et al. 2011 Orzechossk et al.
2018 and based SK has been attracting more attention from the ML/DL community (Petersen
ct al.
2020 Landajuela et al. 2021 Biggio et al., 2021] Valipour et al. 2021] La Cava et al.) 1
Kamienny et al. 2022). Because of its interpretability. various scientific communities apply SR to
advance research in their scientific fields e.g., Physics (Wa & Tegmark] 2019; & Tegmark
2020: Udrescu et al., 2020 Kim al.] 20200 Cranmer al., 2020) Liu & Tegmark [202]] Liu
et al. 2021b), Applied Mechanics (Huang et al. (2021), Climatology (Abdellaoui & Mchrkanoon
2021). ) Materials (Sun al.) 2019 Wang al.
2019 Weng et al.. 2020) Loftis et al., 2020), and
Chemistry (Batra et al. 20220).
Given that SR has been studied in various communities, La Cava al. (202I) propose SRBench. unified benchmark framework for symbolic regression methods. In the benchmark study, they combine the Feynman Symbolic Regression Database (FSRD) (Udrescu & Tegmark (020) and the ODE-Strogatz repository (Strogatz 2018) to compare number of SR methods, using a large-scale heterogeneous computing cluster"
They used hosts with 24-28 core Intel(R) Xeon(R) CPU ES-2690 v4 2.60GHz processors and 250GB RAM.

---

Under review as a conference paper at ICLR 2023
To discuss the potential of symbolic regression for scientific discovery (SRSD), there still remain some issues to be addressed: oversimplified datasets and lack of evaluation metric towards SRSD. For symbolic regression tasks, existing datasets consist of values sampled from limited domains such as in range of 1 to 5, and there are no large-scale datasets with reasonably realistic values that capture the properties of the formula and its variables. Thus, it is difficult to discuss the potential of symbolic regression for scientific discovery with such existing datasets. For instance, the FSRD consists of 120 formulas selected mostly from Feynman Lectures Serieg) (Feynman et al. 1963a c) and are core benchmark datasets used in SRBench (La Cava et al., 2021). While the formulas indicate physical laws, variables and constants used in each dataset have no physical meanings since the datasets are not designed to discover the physical laws from the observed data in the real world. (See Section [31
Moreover, there is a lack of appropriate metrics to evaluate these methods for SRSD. An intuitive approach would be to measure the prediction error or correlation between the predicted values and the target values in the test data, as in standard regression problems. However, low prediction errors could be achieved even by complex models that differ from the original law. In addition. SRBench (La Cava ctt al., 2021)) presents the percentage of agreement between the target and the estimated equations. But in such cases, both 1) equations that do not match at all and 2) that differ by only one term are equally treated as incorrect. As a result, it is considered as a coarse-resse rersuustrsootiion evaluation method for accuracy in SRSD, which still needs more discussion towards real-world applications. A key feature of SR is its interpretability. and some studies (Udrescu et al. 2020: La Cava et 2021) use complexity of the predicted expression as an evaluation metric (the simpler the better). However, is based on big assumption that simpler expression may be more likely to be a hidden law in the data (scientific discovery such as physics law), which may not be true for SRSD, Therefore, there are no single evaluation metrics proposed to take into account both the interpretability and how close to the true expression the estimated expression is.
To address these issues, we propose new SRSD datasets, introduce a new evaluation method, and conduct benchmark experiments using representative SR methods and a new Transformer-basee SR baseline. We carefully review and design annotation policies for the new datasets, considering the properties of the physics formulas. Besides, given that a formula can be represented as a tree structure, we introduce normalized edit distance the tree structure to allow quantitative evaluation of predicted formulas that do not perfectly match the true formulas. Using the proposed SRSD datasets and evaluation metrie, we perform benchmark experiments with set of SR baselines and find that there is still significant room for improvements in terms of the new evaluation metric.
2 RELATED STUDIES
In this section, we briefly introduce related studies focused on 1) symbolic regression for scientific discovery and 2) symbolic regression dataset and evaluation.
2.1 SRSD: SYMBOLIC REGRESSION FOR SCIENTIFIC DISCOVERY
pioneer study on symbolic regression for scientific discovery is conducted by /Schmidt & Lipson (2009), who propose a data-driven scientific discovery method. They collect data from standard experimental systems like those used in undergrad physics education: an air-track oscillator and a double pendulum. Their proposed algorithm detects different types of laws from the data such as position manifolds, energy laws, and equations of motion and sum of forces laws.
Following the study, data-driven scientific discovery has been attracting attention from research communities and been applied to various domains such as Physics (Wu & Tegmark 2019 Udrescu & Tegmark 2020 Udrescu et al. 2020 Kim et al. 22020; Cranmer et al. |0200] Liu & Tegmark 1021
Liu et al. 2021b), Applied Mechanics (Huang et al. 2021). Climatology (Abdellaoui & Mehrkanoon
2021), Materials (Sun al. 2019 Wang et al. 2019 Weng et al., 2020
Loftis et al. 2020), and
Chemistry (Batra et al. 2020).
Udrescu & Tegmark (2000) extract 20 of the 120 equations as "bonus" from other seminal books Goldstein
ct al.
Jackson 1999 Weinberg 1972 Schwartz (014). those differ by constant scalar, Bench treats the estimated equation as correct for solution rate.
2

---

Under review as a conference paper at ICLR 2023
These studies leverage symbolic regression in different fields. While general symbolic regression tasks use synthetic datasets studies collect data from the real world and discuss how could leverage symbolic regression toward scientific discovery.
with limited sampling domains for benchmarks, many of the SRSD
While SRSD tasks share the same input-output interface with general symbolic regression (SR) tasks (i.e., input: dataset, output: symbolic expression), we differentiate SRSD tasks in this study from general SR tasks by whether or not the datasets including true symbolic expressions are created with reasonably realistic assumptions for scientific discovery such as meaning of true symbolic expressions (whether or not they have physical meanings) and sampling domains for input variables.
2.2 DATASET AND EVALUATION
For symbolic regression methods, there exist several benchmark datasets and empirical studies. The Feynman Symbolic Regression Database (Udrescu & Tegmark) (020) is one of the largest symbolic regression datasets, which consists of 100 physics-inspired equations based on Feymman Lectures on Physics (Feynman et al., 1963a blc). By randomly sampling from small ranges of value, they generate the corresponding tabular datasets for the 100 equations. Inspired by HHoi et al.) (2002); Keijzer
(2009), [uy et (201T) suggest 10 different real-valued symbolic regression problems (functions) and create the corresponding dataset (a.ka. Nguyen dataset). The suggested functions consist of either or 2 variables e.g. f(x) x. + I x² + x + x² + z and f(z v) = sin(x) + sin(y"). They generate each dataset by randomly sampling 20- 100 data points.
La Cava et al. 12021) design symbolic regression benchmark, named SRBench, and conduct prehensive benchmark experiment, using existing symbolic regression datasets such as the Feynman Symbolic Regression Database (Udrescu & Tegmark] 2020) and ODE-Strogatz repository (La Cava et al. 2016). In SRBench, symbolic regression methods are assessed by 1) an error metric based on squared error between target and estimated values, and 2) solution rate that shows a percentage of the estimated symbolic regression models that match the true models (equations).
However, these datasets and evaluations for scientific discovery. In Sections 3.1 and 4.1 we further describe potential issues in prior studies.
necessarily designed discuss symbolic regression
3 DATASETS
In this section, we summarize issues found in the existing symbolic regression datasets, and then propose new datasets to address them towards symbolic regression for scientific discovery (SRSD).
3.1 ISSUES IN EXISTING DATASETS
As introduced in Section 2.2
there are many symbolic regression datasets. However, we consider that novel datasets are required to discuss SRSD for the following reasons:
1. No physical meaning: Many of the existing symbolic regression datasets (Hoai al. 2002
Keijzer 2003 Johnson 2009 Uy et al. 2011) are not necessarily chysicc-inspirgg but instead randomly generated e.g.. f(x) log(z) Ø f(z y) ry + sin((z 1) - 1)). To discuss the potential of symbolic regression for scientific discovery, we need to further elaborate datasets and evaluation metrics, considering how we would leverage symbolic regression in practice.
2. Oversimplified sampling process: While some the datasets are physics-inspired such as the Feynman Symbolic Regression Database (FSRD) (Udrescu & Tegmark (0200) and ODF-Strogatz repository (La Cava et al. 2016), their sampling strategies are very simplified. Specifically, the strategies do not distinguish between constants and variables e,g., speed of light" is treated as variable and randomly sampled in range of to 5. Besides, most of the sampling domains are far from values could observe in the real world e.g, II.4.23 in Table SI (the vacuum permittivity values are sampled from range of 1 to 5). When sampled ranges of the distributions are narrow, cannot distinguish Lorentz transformation from Galilean transformation e.g. 1.15.10 and 1.16.6 in Table S3 1.48.2 in Table S5 1.15.tt, 1.15.3x, and I.34.14 14 in Table S7 or the black body
We treat speed of light constant (2.998 10 108m m /s) this study.

---

Under review as a conference paper at ICLR 2023
radiation can be misestimated to Heepann-Boltznnn law or the Wien displacement law e.g. 141.16 in Table S8
3. Duplicate equations:
Due to the two issues above, many of the equations in existing datasets turn out to be duplicate. e.g., as shown in T F J,N, (I.12.1) 1) and F = E (1.12.5) in the original Feynman Symbolic Regression Database are considered identical since both the equations are multiplicative and consists of two variables, and their sampling domains (Distributions in Table I are exactly the same. For instance, approximately 25% of the symbolic regression problems in the original FSRD have 1 duplicates in that regard.
4. Incorrect/ Incorrect/mappropriit nappropriaae formulas:
The Feynman Symbolic Regression Database (Udrescu & Tegmarkk 2020) treat every variables as float whereas they should be integer to be physically meaningful. For example, the number of phase difference in Bragg's 's law should be integer but sampled as real number (1.30.5 Table ST). Furthermore, they don't 't even give special treatment of angle variables (I.18.12, 1.18.16, and 1.26.2 in Table Tp. Physically some variables can be negative whereas the original Feynman Symbolic Regression Database (Udrescu & Tegmark 2020) only samples positive values (e.g. 1.8. 14 and 1..19 in Table (S3). We also avoid using aresinfarccos in the equations since the use of arcsin/arccos in the Feynman Symbolic Regression Database (Udrescu & Tegmark 2020) just to obtain angle variable is not experimentally meaningful (1.26.2 in Table 1.30.5 in Table SI and B10 in Table SII). Equations using aresin and arccos in the original annotation are I.26.2 (Snell's law), 1.30.5 (Bragg's s law), and B10 (Relativistic aberration). These are all describing physical phenomena related to two angles, and it is an unnatural deformation to describe only one of them with an inverse function. Additionally, inverse function use implicitly limits the range of angles, but there is no such limitation the actual physical phenomena.
3.2 PROPOSED SRSD DATASETS
We address the issues in existing datasets above by proposing new SRSD datasets based the equations used in the FSRD (Udrescu & Tegmark 2020). i.e., Section 3.1 summarizes the differences between the FSRD and our SRSD datasets. Our annotation policy carefully designed simulate typical physics experiments so that the SRSD datasets can engage studies on symbolic regression for scientific discovery in the research community.
3.2. ANNOTATION POLICY
We thoroughly revised the sampling range for each variable from the annotations in the FSRD (Udrescu & Tegmark 2020). First, we reviewed the properties of each variable and treated physical constants (e.g., light speed, gravitational constant) as constants while such constants are treated as variables in the original FSRD datasets. Next, variable ranges were defined to correspond to each typical physics experiment to confirm the physical phenomenon for each equation. We also used (of Japan 2022) as a reference. In cases where a specific experiment is difficult to be assumed, ranges were set within which the corresponding physical phenomenon can be seen. Generally, the ranges are set to be sampled on log scales within their orders as 102 in order to take both large and small changes in value as the order changes. Variables such as angles, for which a linear distribution is expected are set to be sampled uniformly. In addition, variables that take specific sign were set to be sampled within that range. Tables T (and[ST [ST show the detailed comparisons between the original FSRD and our proposed SRSD datasets.
3.2.2 COMPLEXITY-AWARE DATASET CATEGORIES
While the proposed datasets consist of 120 different problems, there will be non-trivial training cost required to train symbolic regression model for all the problems individually (La Cava al., 202T) i.e., there will be 120 separate training sessions assess the symbolic regression approach. To allow more flexibility in assessing symbolic regression models for scientific discovery, we define three clusters of the proposed datasets based on their complexity: Easy, Medium, and Hard sets, which consist of 30, 40, and 50 different problems respectively.
We define the complexity of problem, using the number of operations to represent the true equation tree and range of the sampling domains. The former measures how many mathematical operations compose the true equation such add, mul, pow, exp, and log operations (see Fig. 2 The latter
4

---

Under review as a conference paper at ICLR 2023
Table 1: Easy set of our proposed datasets (part 1). C: Constant, V: Variable, F: Float, I: Integer, P: Positive, N: Negative, NN: Non-Negative, ut Uniform distribution, Uog: Log-Uniform distribution.
Properties
Distributions
Eq. ID
Formula
Symbols
Original
Original
F
Force of friction
V.F
V.E.P
NIA
N/A
112.1
F pN
Coeflicient of friction
V.F
V.E.P
4015)
(10 10") Mho (10 -2002,,000 10%)
N
Normal force
V.F
V.E.P
40.5)
Magnitude charge
V.F
1.1244
E'
Jmer
01
Distance
V.F
V.F
WEP
M015)
((1.5)
Mox
Ming 10")
Vacuum permittivity
V.F
C.P.P
4(1.5)
8.854 x 10
Furce
V.F
V.F
NIA
N/A (10 (10-8,10)))) 110-1) (10 , 10
1.1.55
F = E
Electric charge
V.F
V.F
W(1,5)
5)
E
Electric field
V.F
W.F
4(1.5)
Potential energy
V.F
V.E.P
V.F.
M015)
Mmne(102 100%)
1.4.3
U mgz
Gravitational acceleration
V.F
C.P.P
M(155)
9.807 10
Height
V.F
V.F
4(1.5)
Ving (10 10")
Elastic energy
V.F
V.P.P
N/A
N/A (10 10") Ming (10-2, 10")
1.144
Rapinn
Spring constant
V.F
W.P.P.
4(1,5)
Position
V.F
V.
40.5)
Torque
V.F
VE
NIA
NIA Min 10²)
Distance
V.F
4(1,5)
1.18.12
T rF sin 9
Force
V.F
(11
Min 10²)
Angle
V.F
V.
M(0,5)
2410, 2mr
Angular momentum
V.F
V.F.
N/A
N/A (10 10*)
m
Mass
V.F
V.P.P.
4(1
118.16 L= mru sin 8
Distance
V.F
V.P.P.
4(1,5)
M10 1::00-11000000 10*)
Velocity
V.F
V.P.P.
40155)
Miog (10-1111, 10!)
Angle
V.F
V.E.N
4(1.5)
u10, 2xr
Vollage
V.F
V.
NIA
N/A 200 10m3)
1.25.13
V &
Electric charge
V.F
u(1.5)
Electrostatic Capacitanee
V.F
M(L
Mox (10 -0 10-3)
Relative re refructive index
V.F
WEP
1)O,
N/A
1.2622
11
Ain P2
sio #1
81
Refraction angle
V.F
V.F
N/A
21(0. F)
82
Refraction angle
V.F
V.F
4(1,5)
24(0. 2)
Focal length
V.F
V.F
N/A
N/A
1.7.66
8
d1
Distance
Refractive index
V.F
V.
V.P.P.
V.E.P.
M(1,5)
4015) 5)
M1x
2hox (10
11110-11000 10
10-
V.F
((1,5)
M1g 9.10
considers magnitude of sampling distributions (Distributions column in Tables increases the complexity when sampling values from wide range of distributions. We define the domain range as follows:
and ST [nd
S and
f Frange (S) logio max
min
#ES
where S indicates set of sampling domains (distributions) for a given symbolic regression problem.
As we will show Section 5.3 these clusters represent problem difficulties at high level. For instance, these subsets will help the research community shortly tune and/or perform sanity-check new approaches on the Easy set (30 problems) instead of using the whole datasets (120 problems). Figure shows the three different distribution maps of our proposed datasets. Easy, Medium, and Hard sets consist of 30, 40, and 50 individual symbolic regression problems, respectively.
4 BENCHMARK
Besides the conventional metrics, propose new metric to discuss the performance of symbolic regression for scientific discovery Section Following the set of metrics, we design an evaluation framework of symbolic regression for scientific discovery.

---

Under review as a conference paper at ICLR 2023
30
25
x
X
Domain Range
20
Easy
15-
Medium
10
x
x
x
Hard
X
5
x
x
X
0
x
x
x
0.0
2.5
5.0
7.5 Number of Operations
10.0
12.5
15.0
17.5
20.0
Figure 1: Distribution map of our proposed datasets based on three different subsets with respect to our complexity metrics. Data points at top right/bottom left indicate more/less complex problems.
ftrue AT
Convert to an equation tree
3.14159265318979
Mul
1. Substitute values
6.626 10-34
L = (8.3264771690743) X 10 JBB
C
Figure Example of preprocessing a true equation (III.7.38 in Table in evaluation session. When converting equation tree, replace constant values and variables with specific symbols e.g. 8.326477169074339 10 + CI, X1, X2.
4.1 METRICS
In general, it would be difficult define "accuracy" of symbolic regression models since will compare its estimated equation to the ground truth equation and need criteria to determine whether or not it is "correct". La Cava et al. (2021) suggested reasonable definition of symbolic solution, which designed to capture symbolic regression models that differ from the true model by constant or scalar. They also used R2 score (Eq. 2 and define accuracy the percentage of symbolic regression
problems that a model meets R² T, where r is threshold e.g., 0.999 in (La Cava et al. 2021).
ΣS (Spea (X,) Sinee (X,))² Σ (fuve (Xx)
R²
8) -))
where N indicates the number of test samples (i.e., the number of rows in the dataset), and is mean of target outputs produced by fawe. fped and fave are trained SR model and true model, respectively. However, these two metrics are still binary (correct not) or require a threshold and does not explain how structurally close to the true equation the estimated one is. While key feature of symbolic regression is its interpretability. there are no single evaluation metrics to take into account both the interpretability and how close to the true expression the estimated expression is.
To offer more flexibility and assess estimated equations such a way, we propose the use of edit distance between estimated and ground truth equations, processing equations as trees. Although edit distance has been employed in different domains such as machine translation (Przybocki et al.) has been to study the search process for genetic programming approaches (O Reilly 1997 Burke et al. 2002), Nakai et al. 2013). Different from metric of solution quality for SRSD.
2006) (text-based edit distance), its primary
prior work, propose of tree-based edit distance

---

Under review as a conference paper at ICLR 2023
For pair of two trees, edit distance computes the minimum cost to transform to another with sequence of operations, each of which either 1) inserts, 2) deletes, or 3) renames a node. In this study, node can be either mathematical operation (e.g.. add, exp symbols), variable symbol, constant symbol. For the detail of the algorithm, we refer readers to (Zhang & Shasha 1989).
As illustrated in Fig. Planck constant to the expression, and 2) converting the resulting expression to an equation tree that represents the preorder traversal of the equation with simplified symbols. It should be worth noting that before generating the equation tree, we simplify and convert equations to approximations by sympy Meurer al. (2017), a Python library for symbolic mathematics. It helps us consistently map given equation to the unique equation tree and compute edit distance between the true and estimated equation trees since our evaluation interest is in simplified expressions of the estimated equations rather than how SR models produced the equations. For instance, "x: + I + x", x² will be simplified by sympy to "3 x= and considered identical.
preprocess equations by 1) substituting constant values e-g., and
x. x², and "Ir
For edit distance, we use method proposed by Zhang & Shasha (189). Given that the range of edit distance values depends on complexity equations, we normalize the distance in range to I as
a( firue) min (1, d (Spreds Saue)
[
where fpred and fime are estimated and true equation trees, respectively. d(ppred J is an edit distance between fpred and fime- [mme|] indicates the number of the tree nodes that compose an equation fane- We note that this metrie is designed capture similarity between estimated and true equations, thus coefficient values themselves (e.g., value of C₁ in Fig. 2 should not be important.
4.2 EVALUATION FRAMEWORK
For real datasets (assuming observed datasets), only tabular data are available for training and validation. (In practice, a test dataset does not include the true equation.) For benchmark purposes, true equations are provided test data besides test tabular data.
For each problem, we use the validation tabular dataset and choose the best trained SR model fjred from J, a set of the trained models by a given method respect to Eq.
fprea(X.) - Stree (X;)
spied = arg min E
(4)
finue (X,)
where X, indicates the i-th row of the validation tabular dataset X.
Notice that while proposed in Section 4.1 normalized edit distance between estimated and true equation trees, such true equations will not be available in practice, especially when using symbolic regression methods for scientific discovery. For this reason, predicted values against validation tabular dataset to choose the best model obtained through hyperparameter tuning. Using the best model per method, compute the normalized edit distance to assess the method.
the geometrical distance between
5 EXPERIMENTS
5.1 BASELINE METHODS
For baselines, we use the five best symbolic regression methods in SRBench (La Cava et al. 2021). Specifically, we choose gplearn (Koza & Poli 2005). AFP (Schmidt & Lipson (011)).
AFP-FE (Schmidt & Lipson 2009). , AI Feynman (Udrescu et al. 2020), and DSR (Petersen et al.)
2020), referring to the rankings of solution rate for the FSRD datasets in their study. We note that
La Cava et al. (202I) also benchmark symbolic regression methods for black-box problems, whose
true symbolic expressions are unknown, and other symbolic regression methods e.g., Operon (Kom-
menda et al., 2020). SBP-GP /Virgolin et al. 2019). FEAT (La Cava et al. 2018 EPLEX (La Cava al. 2019). and GP- GOMEA (Virgolin et al. 2021 outperform the five baseline methods we choose from their study, in terms of driveiven accuracy. However, we find solution rate more aligned with
edit distance, thus choose the five best symbolic regression methods in terms of solution rate
7

---

Under review as a conference paper at ICLR 2023
empirically shown for the FSRD datasets in SRBench (La Cava al. 2021). In addition to the five existing symbolic regression baselines, we introduce Symbolic Transformer, a new baseline model with Transformer (Vaswani al. 2017):
1. gplearn (Koza & Poli 2005
genetic programming based symbolic regression method
published as Python package gplearn.
2. AFP (Schmidt & Lipson 2011): Age-fitness pareto optimization.
3. AFP-FE (Schmidt & Lipson
2009):
AFP optimization with fitness estimates.
4. A1 Feynman (Udrescu et al.
2020 an iterative approach to generate symbolic regression to
seek to fit data to formulas that are Pareto-optimal.
5. DSR (Petersen et al. 2020): reinforcement learning based deep symbolic regression.
6. Symbolic Transformer (ST): our Transformer- based symbolic regression baseline.
For the details of the baseline models, we refer readers the corresponding papers (Koza Poli & Lipson 2011 2009 Udrescu et al. 2020 Petersen et al.) 2020). We provide the details of Symbolic Transformer in Section including our pretraining strategy. While Symbolic Transformer itself is a new model, we note that the main contribution of this work lies in the datasets and benchmark of symbolic regression for scientific discovery. Our Transformer-based baseline method is simply inspired by recent advances in deep learning, specifically Transoormer-bsee gegghperformance. modern, and flexible models such as (Vaswani et al.] 2017 Devlin et al. 2019) Dosovitskiy et al. (2020). Thus, the new model is not necessarily designed to show improvements over the existing Transformer-basee symbolic regression models (Valipour et al. 2021 Biggio et al. 2021 including contemporary work such as (Kamienny al. 2022).
5.2
RUNTIME CONSTRAINTS
The implementations of the baseline methods in Section 5.1 except Symbolic Transformer do not use any GPUs We run 600 high performance computing (HPC) jobs in total, using computing nodes in an HPC infrastructure, which have 5 20 assigned physical CPU cores, 30 - 120 GB RAM, and 720 GB local storage. Due to the properties of our HPC resource, we have some runtime constraints:
1. Since each HPC job is designed to run for up to 24 hours due to the limited resource, we run a job with a pair of target tabular dataset and a symbolic regression method.
2. Given pair of a dataset and method, each of our HPC jobs runs up to 100 separate training sessions with different hyperparameter values.
5.3 RESULTS
In this section, discuss the experimental results of our baseline methods, using the proposed SRSD datasets. Tables 2 and 3 show the performance of the symbolic regression baseline methods in terms of R2-driven accuracy 2 > 0.999) and solution rate respectively, and both the metries used in SRBench (La Cava et al. (021). According to the metrics, DSR significantly outperforms all the other baselines we considered. The DSR results also indicate difficulty levels of the three categories of our SRSD datasets, which looks aligned with our complexity-wwan aware dataset categorization (Section 3.2.2).
Now we discuss the results using the normalized edit distance. Thble the results of the baseline methods in terms of the normalized edit distance. Interestingly, while the Symbolic Transformer performed the worst in Table 2 in terms of R² based accuracy, achieved the best normalized edit distance for all the SRSD Easy, Medium, and Hard sets and significantly improved DSR. This trend also implies that the R2- -aaedd accuracy does not always indicate how well the SR model can produce an equation that is structurally close to the true equation. We also confirmed that the difficulty level of each set of SRSD datasets is reflected to the overall trend Tables
We also performed user study and how aligned with human judges the existing SR and new SRSD evaluation metrics are. The results show that the SRSD evaluation metric (NED) is more aligned with human judges than R22 score. We refer readers to Section G for more details.
'e used NVIDIA GeForce RTX 3070 Ti for pretraining Symbolic Transformer.
8

---

Under review as a conference paper at ICLR 2023
Table 2: Baseline results: accuracy (R² > 0.999) defined by
La Cava et al. (022))
SRSD Datasets
Method
gplearn
AFP
AFP-FE
Al Feynman
DSR
ST
Easy (30 problems)
6.67%
20.0%
23.3%
33.3%
60.0%
0.00%
Medium set (40 problems)
7.50%
2.50%
2.50%
5,00%
42.5 %
0.00%
Hard (50 problems)
2.00%
4.00%
4.00%
8.00%
30.0% %
0.00%
Table 3: Baseline results: solution rate defined by
La Cava et al.
2021pp
SRSD Datasets
Method
gplearn
AFP
AFP-FE
AI Feynman
DSR
ST
Easy set (30 problems)
6.67%
20.0%
20.0%
30.0%
43.3%
16.7%
Medium set (40 problems)
2.50%
2.50%
2.50%
2.50%
10.0 %
5.00%
Hard set (50 problems)
0.00%
0.00%
0.00%
2.00 %
2.00 %
0.00%
Table 4: Baseline results: our proposed normalized edit distances (the smaller the better).
SRSD Datasets \
Method
gplearn
AFP
AFP-FE
AI Feynman
DSR
ST
Easy set (30 problems)
0.876
0.703
0.712
0.646
0.551
0,435
Medium set (40 problems)
0.939
0.873
0.897
0.936
0.789
0.556
Hard set (50 problems)
0.978
0.960
0.956
0.930
0.833
0.704
6 LIMITATIONS
6.1 IMPLICIT FUNCTIONS
Symbolic regression generally has limitation in inferring implicit functions, as the model infers a trivial constant function if there are restrictions variables. For example, fxx y) = 0 is inferred as Vx, y. This problem can solved by applying the constraint that an inferred function should depend on at least two variables e.g. inferring [x, y) 0 with Øf ou * 0 and By + 0, or by converting the function to an explicit form e.g., y = g(x). We converted some functions in the datasets into explicit forms and avoided the inverse trigonometric functions described in Section 3.1
6.2 DUMMY VARIABLES AND NOISE INJECTION
When applying machine learning to real-world problems, it is often true that 1) not all the observed features (variables in symbolic regression) are necessary to solve the problems, and 2) the observed values contain some noise. While we follow La Cava et al. (0211 and show experimental results for SRSD datasets with noise-injected target variables in Section E these aspects are not thoroughly discussed in this study, such discussions can be separate paper built on this work and further engage studies of symbolic regression for scientific discovery.
CONCLUSION
In this work, we pointed out issues of existing datasets and benchmarks of symbolic regression for scientific discovery (SRSD). To address the issues, we proposed 1) 120 new SRSD datasets based on set of physics formulas in FSRD (Udrescu & Tegmark 2020) and 2) a new evaluation metric for SRSD to discuss the structural similarity between the true and estimated symbolic expressions (equations). We note that this study argues that the normalized edit distance is metric not take the place of existing SR metrics but to incorporate such metrics (e.g., Tables 2 and 4). Besides the main contribution above, proposed Transfonmer-bast Transformer that achieved the best normalized edit distance for the proposed SRSD datasets. To encourage the studies of SRSD,, we provide datasets and code with MIT License.
symbolic regression baseline, named Symbolic

---

Under review as a conference paper at ICLR 2023
(OPTIONAL) ETHICS STATEMENT
In this study, we recruited 23 human subjects and performed a user study (see Section G with approval from an ethics review board. We performed the user study to discuss how meaningful our new SRSD evaluation metric is, compared to existing SR metrics. Subjects were asked to assess in scale of 1-to-5 how close the true equation the estimated equation is, and in the user study we used only the human-base based evaluations against equations. For this reason, we found no ethical concerns in the user study.
(OPTIONAL) REPRODUCIBILITY STATEMENT
For reproducibility, we provide the following items in this paper and/or the supplementary material:
1. our datasets and code repository with MIT License and instructions to scripts, including all the five existing and one new basclines (See the supplementary material),
2. annotation policy and details of the 120 proposed datasets (See Section 3.2.1 and Tables
3. architecture design and hyperparameters of Symbolic Transformer, introduced (See Section C and the supplementary material),
SRSD baseline
4. computing resources and runtime constraints in this study (See Section 5.2 and Footnote and
5. hyperparameters used for the six baseline methods (See Sections B and
REFERENCES
Ismail Alaoui Abdellaoui and Siamak Mehrkanoon. Symbolic regression for scientific discovery: an application to wind speed forecasting. In 2021 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 01-08. IEEE, 2021.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM 2019.
SIGKDD international conference on knowledge discovery data mining, pp.
Rohit Batra, Le Song, and Rampi Ramprasad. Emerging materials intelligence ecosystems propelled by machine learning. Nature Reviews Materials, pp. 1-24, 2020.
Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo. Neural Symbolic Regression that Scales. In International Conference on Machine Learning, pp. 936 945. PMLR, 2021.
Edmund Burke, Steven Gustafson, Graham Kendall, and Natalio Krasnogor. Advanced Population Diversity Measures in Genetic Programming. In International Conference Parallel Problem Solvingfroo Nature, pp. 341-350. Springer, 2002.
R. Qi Charles, Hao Su, Mo Kaichun, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Miles Cranmer, Alvaro Sanchez Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering Symbolic Models from Deep Learning with Inductive Biases. Advances in Neural Information Processing Systems, 33:17429- 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference Learning Representations, 2020.
10

---

Under review as a conference paper at ICLR 2023
Richard P Feynman, Robert B Leighton, and Matthew Sands. The Feynman Lectures Physics, Vol. The New Millennium Edition: Mainly Mechanics, Radiation, and Heat, volume 1. Basic books, 1963a.
Richard P Feynman, Robert B Leighton, and Matthew Sands. The Feynman Lectures on Physics, Vol. I1: The New Millennium Edition: Mainly Electromagnetism and Malter, volume 2. Basic books, 1963b.
Richard P Feynman, Robert Leighton, and Matthew Sands. The Feynman Lectures on Physics, Vol.
HI: The New Millennium Edition: Quantum Mechanics, volume 3. Basic books, 1963c.
Herbert Goldstein, Charles Poole, and John Safko. Classical Mechanics. Addison Wesley, 2002.
Nguyen Xuan Hoai, Robert Ian McKay, D Essam, and R Chau. Solving the Symbolic Regression Problem with Tree Adjunct Grammar Guided Genetic Programming: The Comparative Results. In Proceedings of the 2002 Congress Evolutionary Computation. CEC'02 (Cat. No. 02TH8600), volume 2, pp. 1326-1331. IEEE, 2002.
Zhanchao Huang, Chunjiang Li, Zhilong Huang, Yong Wang, and Hanqing Jiang. Al-Timoshenko: Automatedly Discovering Simplified Governing Equations for Applied Mechanics Problems From Simulated Data. Journal of Applied Mechanics, 08(10)101000 2021.
John David Jackson. Classical Electrodynamics. Wiley, 1999.
Colin G Johnson. Genetic Programming Crossover: Does it Cross Over? In European Conference on Genetic Programming. pp. 97-108. Springer, 2009.
Fierre-Alexandre Kamienny, Stéphane d' Ascoli, Guillaume Lample, and François Charton. End-to- end symbolic regression with transformers. arXiv preprint 21XX1:2204410532 2022.
Maarten Keijzer. Improving Symbolic Regression with Interval Arithmetic and Linear Scaling, In European Conference on Genetic Programming, pp. 70-82. Springer, 2003.
Samuel Kim, Peter Y Lu, Srijon Mukherjee, Michael Gilbert, Li Jing, Vladimir Čeperić, and Marin Soljačić. Integration of Neural Network Based Symbolic Regression in Deep Learning for Scientific Discovery. IEEE Transactiony Neural Networks and Learning Systems, 2020.
Michael Kommenda, Bogdan Burlaeu, Gabriel Kronberger, and Michael Affenzeller. Parameter
identification for symbolic regression using nonlinear least squares. Genetic Programming and Evolvable Machines, 2105):471-501. 2020.
John R Koza and Riccardo Poli. Genetic Programming. In Search methodologies, pp. 127-164. Springer, 2005.
William La Cava, Kourosh Danai, and Lee Spector. Inference of compact nonlinear dynamic models by epigenetic local search. Engineering Applications of Artificial Intelligence, 55:292-306, 2016.
William La Cava, Tilak Raj Singh, James Taggart, Srinivas Suri, and Jason H Moore. Learning concise representations for regression by evolving networks of trees. In International Conference on Learning Representations, 2018.
William La Cava, Thomas Helmuth, Lee Spector, and Jason H Moore. Probabilistic and Multi- Objective Analysis of Lexicase Selection and e-Lexicase Selection. Evolutionary Computation, 27 (3):377-402, 2019.
William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabricio Olivetti de Franca, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason H Moore. Contemporary Symbolic Regression Methods and their Relative Performance. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep rein- forcement learning. In International Conference on Machine Learning, pp- 5979 5979.5989. PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. Scheduled Sampling Based on Decoding Steps for Neural Machine Translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3285-3296, 2021a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv: 1692, 2019.
Liu, B Wang. Q Meng, Chen, M Tegmark, and TY Liu. Machine-learning nonconservative dynamies for new-physics detection. Physical Review: E, 1044522)05500--053302 2021b.
Ziming Liu and Max Tegmark. Machine Learning Conservation Laws from Trajectories. Physical May 2021.
Review Letters, 126:180604
Christian Loftis, Kunpeng Yuan, Yong Zhao, Ming Hu, and Jianjun Hu. Lattice Thermal Conductivity Prediction Using Symbolic Regression and Machine Learning. The Journal of Physical Chemistry A, 43355):435-450, 2020.
Yoshitomo Matsubara. torchdistill: A Modular, Configuration-Driver Framework for Knowledge Distillation. In International Workshop Reproducible Research in Pattern Recognition, pp. 24-44. Springer, 2021.
Aaron Meurer, Christopher P Smith, Mateusz Paprocki, Ondfej Čertík, Sergey Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K Moore, Sartaj Singh, et al. SymPy: symbolic computing in Python. PeerJ Computer Science, 3:e 1031 2017.
Shohei Nakai, Tetsuhiro Miyahara, Tetsuji Kuboyama, Tomoyuki Uchida, and Yusuke Suzuki. Acquisition of Characteristic Tree Patterns with VLDC's by Genetic Programming and Edit Distance. In 2013 Second IIII International Conference on Advanced Applied Informatics, pp. 147-151. IEEE, 2013.
National Astronomical Observatory of Japan. Handbook of Scientific Tables. World Scientific, 2022.
U-M O'Rell. Using a distance metric on genetic programs to understand genetic operators. In 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, volume 5, pp. 4092-4097. IEEE, 1997.
Patryk Orzechowski, William La Cava, and Jason H Moore. Where are we now? A large benchmark study of recent symbolic regression methods. Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1198-1190, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances neural information processing systems, 32:8026-8037, 2019.
Brenden K Petersen, Mikel Landajuela Larma, Terrell N Mundhenk, Claudio Prata Santiago, Soo Kyung Kim, and Joanne Taery Kim. Deep symbolic regression: Recovering mathemat- ical expressions from data via risk-secking policy gradients. In International Conference Learning Representations, 2020.
Mark Przybocki, Gregory Sanders, and Audrey Le. Edit Distance: A Metric for Machine Translation Evaluation. Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06), 2006.
Michael Schmidt and Hod Lipson. Distilling Free-Form Natural Laws from Experimental Data. Science, 324((523):811-5) 2009.
Michael Schmidt and Hod Lipson. Age-Fitness Pareto Optimization. In Genetic programming theory and practice VIII, pp. 129-146. Springer, 2011.
Matthew D Schwartz. Quantum Field Theory and the Standard Model. Cambridge University Press, 2014.
12

---

Under review as a conference paper at ICLR 2023
Steven H Strogatz. Nonlinear Dynamics and Chaos with Student Solutions Manual: With Applications to Physics, Biology, Chemistry, and Engineering. CRC press, 2018.
Sheng Sun, Runhai Ouyang, Bochao Zhang, and Tong-Yt Yi Zhang. Data-driven discovery of formulas by symbolic regression. MRS Bulletin, 44(7):559-564, 2019.
llya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to Sequence Learning with Neural Networks. Advances in neural information processing systems, 27, 2014.
Silviu-Marian Udrescu and Max Tegmark. AI Feynmann A mehysics-inspire method for symbolic regression. Science Advances, 6(16):eaay2631, 2020.
Silviu-Marian Udrescu, Andrew Tan, Jiahai Feng, Orisvaldo Neto, Tailin Wu, and Max Tegmark. AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. Advances in Neural Information Processing Systems, 33, 2020.
Nguyen Quang Uy Nguyen Xuan Hoai, Michael O' O'Neill, Robert McKay, and Edgar Galván-López. Semantically-based Crossover in Genetic Programming: Application to Real-valued Symbolic Regression. Genetic Programming and Evolvable Machines, 12(2):91-119, 2011.
Mojtaba Valipour, Bowen You, Maysum Panju, and Ali Ghodsi. SymbolicGPT: A Generative Transformer Model for Symbolic Regression. arXiv preprint
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. Advances in neural information processing systems, 30, 2017.
Marco Virgolin, Tanja Alderliesten, and Peter AN Bosman. Linear scaling with and within semantic backpropagation-bun based genetic programming for symbolic regression. In Proceedings of the genetic and evolutionary computation conference, pp. 1084-1092, 2019.
Marco Virgolin, Tanja Alderliesten, Cees Witteveen, and Peter AN Bosman. Improving Model-Based Genetic Programming for Symbolic Regression of Small Expressions. Evolutionary computation, 29(2):211 1--22,,11,,, 2021.
Yiqun Wang, Nicholas Wagner, and James M Rondinelli. Symbolic regression in materials science. MRS Communications, 93))793-805, 2019.
Steven Weinberg. Gravitation and Cosmology: Principles and Applications of the General Theory of Relativity. Wiley,, 1972.
Baicheng Weng, Zhilong Song, Rilong Zhu, Qingyu Yan, Qingde Sun, Corey G Grice, Yanfa Yan, and Wan-Jian Yin. Simple descriptor derived from symbolic regression accelerating the discovery of new perovskite catalysts. Nature Communications, 11(1):1-8, 2020.
Ronald J Williams and David Zipser. A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural computation, 1(2):270-280, 1989.
Tailin Wu and Max Tegmark. Toward an artificial intelligence physicist for unsupervised learning. Physical Review E, 100(3):03331, 2019.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Neural Information Processing Systems, 2017.
Kaizhong Zhang and Dennis Shasha. Simple Fast Algorithms for the Editing Distance between Trees and Related Problems. SIAM journal on computing, 18(6):124--1200 1989.
13

---

Under review as a conference paper at ICLR 2023
A OUR SRSD DATASETS: ADDITIONAL INFORMATION
This section provides additional information regarding our SRSD datasets. We created the datasets to discuss the performance of symbolic regression for scientific discovery (SRSD). Each of SRSD) datasets consists 10,000 samples and has train, val, and test splits with ratio of 8:1:1. We refer readers to Section 3 for details of the datasets. Tables - SII comprchensively summarize the differences between FSRD and our SRSD datasets. Note that the table of Easy set (part 1) provided as Table in Section 3.1 As described in Section [3..2] we categorized each of the 120 SRSD
datasets into one of Easy, Medium, and Hard sets. The datasets and the documentations are provided as part of the supplementary material.
Table S1: Easy set of our proposed datasets (part 2). C: Constant, V: Variable, F: Float, I: Integer, P:
Positive, N: Negative, NN: Non-Negative, Integer treated float due to the capacity of 32-bit integer, U: Uniform distribution, Mog: Log- Uniforrf distribution.
Properties
Distributions
Eq. ID
Formula
Symbols
Original
Ours
Original
Ours
Interplanar distance
V.F
V.E.P.
4(2 5)
N/A Mog
Wavelongth of X-tay
V.E.P
2(12))
..00))
1.30.5
d =
number phase difference
20,5)
(00 10?)
Incidense/Retteciion angle
V,F
V.
NAA
(1(-2m,mm N/A
11
Velocity
V.F
V.
N/A
lonic conductivity
V,F
V.
4(155)
M1n 9, 10 4)
144.16
pg &
Electric charge of lons
V.F
V.F
4(1,5)
Mon(10-11 M10010011110-)) 10-
Voltage
V.F
V.
4(1
(10 101)))
Distance
V.F
V.P.P. 4(155)
U0e 10-1)
Velocity sound
V.F
V.P.P NIA
NIA
147.23
c=
7
Atmospheric pressure
capacity
VEP 20155)
VEP
20(.5)
M(005
24112)
5, 4(1.2)
10 10-5)
Density of air
V,F
VEP 2(1.5)
J
Energy difference
V.F
VE NA
NAA Mog .100)
Thermal conductivity
V.F
VEP 4(155)
T, N(T;
$
Th
Temperature
Temperature
V.F
V.F
V.P.P 4(155)
V.P.P 4(155)
Umg(100 10a)
Umg(10' 10a)
Area
V.F V.P.P. 401.5)
41e 4, 10-2)
d
Length
V.F V.P 20L.)
Mog 10%)
W
Work
V.F
VF 211.5)
(100 10?)
r
Distance
V.F
MEP 201.5)
Uoog (1002,
0
Electric potential
V.F
VF
NAA
N/A Umn "10-",100000 1,101)
Electric charge
V.F
V. 40155)
Φ
Vacuum permittivity
V.F
C.P.P
4(1.5)
8,851 x 10
Distance
V.F
V.P.P.
40L5)
M1x 10")
Energy
16
permintivity
C.F.P
20155)
x 10-12
Magnitude electric
V.F
V.P
UIL5))
40g 102)
Electric field
V.F
V.F
N/A
N/A (10 8 10-1)
11.109
E Ztm
Surface charge
Vicuum permitivity
V.F
V.F
C.P.P.
V.
40.5)
40.5)
8.854 x 10-12
Electric susceptibility
V.F
V.P.P
4(1.))
Unx 10?)
The magnitude of the magnetic field
V.F
V.
NYA
N/A 8.854 10-12
Vacuum permintivity
21(15)
I1.13.17
Speed of light
C.E.P
200.5)
2.998 10"
Electric current
200.5)
Radius
V.F
M.P
201.5)
Zing (10 10 -1
U Energy from maguetic field
V.F
V.F
NIA
N/A Mog 10-20) 23)
Magnetic dipale moment
V.F
V.F
40.5)
11.15.4 U=
Magnetic field strength
V.F
V.F
4(1.5)
U1og(1 10 10
0 Angle
V.F
W.E.N
(((1.))
24(0, 2m)
14

---

Under review as a conference paper at ICLR 2023
Table S2: Easy set of our proposed datasets (part 3).
Properties
Distributions
Eq. ID
Formula
Symbols
Original
Ours
Original
Ours
U
Energy
IL15.5
U
P. cos
P
E
Flectric dipole moment
Magnitude of electric field
V,F
V,
V.F
M(1,5)
M(1,5)
Moo 22 20)
(10' 10")
Angle
V.F
V.F.
M(1,5)
24(0, 2m)
S
Radiant intensity
V,F
V.F
NIA
NIA 8.854 10
16.27.16
= ecE
Vacuum permittivity
V.F
C.F.P
2(1.5)
Speed of light
V.F
C.F.P
2(1.5)
2.998 10
E Magnitude of electric field
N.F
V.P.P.
4(1.5)
Mog (10 10²)
Energy deusity
V.E.P.
N/A
N/A 12
11.27.18
F
permittivity
C.F.P
M(1.5)
E3
Magnitude of electric field
V,F
W.P.P
4(1.5)
Ling (10 1, 1
Angular frequency
V.F
V.F.
N/A
N/A
factor
V,F
V.F 4(155)
u(-1,1)
1134.11
Electric churge
V.F
V.F 4(1.5)
Uog Uno
(10-11.1009)
B
Magnetic field strength
V.F
V.F 4(1.5)
(1e 10110-0,000 10-T, (10 10 25)
Mass
V.F
W.P.P 4(1.5)
U
Energy
V.E.P N/A
N/A 1(-1,1) 9.2740100783
9
factor
M(1.5)
11.4..9%
B
14
Bohr magneton
C.F.P
M(1.5)
24
13
Magnetic lield strength
V.F
V.F 4(1.5)
2log (10 10-1)
J.
Element of angular momentum
V.F
V.F.
4(1.5)
2iog 210 (10 .0102
1
Planck constant
V.F
C.E.P
2(1.5)
6.626 x 10 34
F
Foree
V.F
V.F
NIA
N/A 41a (100-11000) 10
Y
Young's modulus
V.F
V.E.P
4(1.5)
11.38.3
F=
P=Y
A
Area
V.F
V.P.P. M(1.5)
(10 10
81
Displacement
V.F
V.F. M(I.5)
Me
Length
V.E.P 4(1.5)
Kiig (200010220)) 10%)
14
Rigidity modulus
V.F
V.P.P. N/A
NAA Mog Mog 10001,1000 10²)
14
Y
Young's modulus
V.F
W.P.P. 2(1.5)
Poisson coefficient
V.F
V.P.P.
2(115)
Uiog 2 10
Precession frequency
V.F
V.F
N/A
N/A Mm0 10-9)
4mp B
Magnetic momeat
V.F
V.F
M(1.5)
13
Magnetic flux densiry
V.F
V.F
M(1.5)
Mox Mo
(107) 10-1)
Planck constant
V.F
C.F.P
M(I.5)
6.626 10
V.F.
N/A
N/A Mhe M1eg 102)
111.12.43
m
771
Spin state
V.F
V.LNN
M(1.5)
Planck constant
V.F
C.E.P
4(1.5)
6.626 x 34
h:
Wivenumber
V.F
V.F
N/A
NIA 2hog (109 102)
Parameter of state
V.
V.I
4(15))
11..15.7
N Number of atoms
V.F
V.
((1.5)
(10% 102)
Lattice constant
V.F
V.E.P
(4(1.5)
(10 10"1 10-*)
15

---

Under review as a conference paper at ICLR 2023
Table S3: Medium set of proposed datasets (part 1).
Properties
Distributions
Eq. ID
Formula
Symbels
Original
Ours
Original
Ours
222
Position
V.F
211,5)
Mox (10 100)
18.14
d
(22 (22-
21) + (3/2 yr
251
Position
V.F
V.F
4(1,3)
Ung ,00)
5/2
Position
V.F
V.F
2(1,3)
Unn (200(1111100 .00)
M/
Position
V.F
V.F
2(1,5)
Mox (10 1, 1.1)
178
Mass
V.F
V.P.P.
N/A
NIA (10 .01)))
Invariant mass
V.F
V.P.P.
2(1155)
L10.7
Velocity
V.F
W.P
21(1,2)
Uig (10?; 10")
Speed Tight
V.F
C.F.P.
M(3.
2.998 10
product
V.F
251 Element voctor
V.F
V.F
(1),3)
M0n 1, 1
y/
Element a vector
V.F
V.F
21(1.)
(100 ,10")
111.19
A 2191
x232 + #aMs
22
Element vector
V.F
V.F
2)
21x .10)
Element voctor
V.F
V.F
21(15)
2oo (10 -.10")
Element of vector
V.F
V.F
21(1,5)
Mhu (10 1.10")
Element of vector
V.F
V.F
21(155)
Mhxx (10 1,10
F
Flectmstatic force
V.F
V.F
N/A
N/A Uing (10 3 11
91
Electric charge
V.F
2(1.5)
L12.2
F
92
Ellcttric charge
V.F
2(1,5)
Min 10
Distance
V.F
V.P.P.
21115)
Mix M10 10
Vacuum permittivity
V.F
C.E.P
2(1,5)
8.854
12
F
Force
V.F
V.F
N/A
N/A
Ellecticc charge
V.F
V.F
21(15)
E
Fllectric field
V.F
V.F
21(155)
2hox (10 101001,10 10²)
(E++ (0))
B
Magnetic tield strength Velocity
V.F
V.E.P
((1,5)
Ulog (10 1,10²)
V.F
W.P.P
(1(155
thog M10'1111 10!)
Angle
V.F
V.R.NN
21(1.)
M(0,
K
energy
V.F
V.P.P.
N/A
NIA 21og 2 2,00)
m
Mass
V.F
V.P.P.
2(1.5)
L13.4
K 3m((² (002 + +
Element velocity
V.F
W.P
201.5)
2hu 10²)
"
Flement velocity
V.F
W.P.P.
21()5)
Mox
1.10)
w
Element of velocity
V.F
V.P.P.
2((15)
2he (10 1
U
Potential energy
V.
V.E.P
NA
N/A 6.674 10-
G
Gravilational constant
V.F
C.E.P
(1(1.5
Mass (The Earth)
V.F
V.E.P.
21(1
2oo
100))
L13.12
Gmm m2
Mass
V.F
V.E.P.
21(15)
Mox (10 1.10)
72
V.F
21(15)
Miz (10 10")
71 Distance
V.F
20(15)
Mox 2010%)
Relativistic mass
V.F
W.P.P.
N/A
N/A 21n 2, 100)
L15.10
mig
Rest Mass
V.F
VEP
2(115)
Velocity
V.F
V.F
2(112)
Uing M0n 107)
Speed of light
V.F
C.F.P
4(33 10)
2.998 10
Velocity
V.F
V.F
N/A
NIA (109, 108)
Velocity
V.F
2((15)
L.6.6
Ioww/ct
Velocity
V.F
21(155
Uing M0g (10 10
Speed light
V.F
C.E.P
20(15)
10
Center of gravity
V.F
W.F.
N/A
N/A Mox 1, 10t)
mi
Mass
V.F
W.P.P.
21(15)
L18.4
TB 18222
Position
V.F
V.F
2((1.5)
21og(10
10')
m0] mgg
Mass
V.F
V.P.P.
27(1,5)
21001001 1100)
Position
V.F
V.F
24(1, 5)
Mox(10 1,10)
16

---

Under review as a conference paper at ICLR 2023
Table S4: Medium set of proposed datasets (part 2).
Pruperties
Distributions
Eq. ID
Formula
Symbols
Original
Ours
Original
Ours
Eiergyy
V.E.P.
Mass
V.E.P
4(133)
Mux (10 100)
E mm(w w
Angular velocity
V.F
V.F
4(1,3)
ting (10 10")
We
Angular velocity
W.F
W.F. 4(1,3)
tling (10 10")
a2
Position
V.F
211,3) V.F.
Moo (10 10*)
Wavenumber
V.F
V.P.P.
N/A
NIA Mor(10", 10
1.29.4
==
Froquency clectromagnetic waves
V.F
V.P.P.
2011, 10)
Speed of light
V.F
C.E.P.
20,100 10)
2.998 10
Radiant energy
V.E.P.
Fllctric charge
V.F.
4(1,5)
M1x 10-1) 1
2 Breeh
Magnitude of direction vector
W.F
V.E.P 2(1.5)
Mmg 107)
Vacuum permittivity
WE
C.E.P ((1,5)
8.854 x 10-12
Speed of light
V.F
C.F.P 201,6)
2.998 10
Angular velucity
V.F
V.F.
N/A
N/A Mor(10 10 10-9)
Eloctric charge
V.F
V.F.
4(1.5)
1.34.8
Velacity
V.F
V.F. 401,5)
10")
13 Magnetic field
V.F
V.F.
(11,5)
(10'
1 102)
Angular
((1.5) V.F.
Mor 10
Frequency electromagnetic waves
V.F.
V.P.P.
We Frequency electromagnetic waves
W.F.
V.P.P 4(1.5)
M10 10
1.34.10
1- 1
Velacity
V.F
V.F. 4(1,2)
Mog Mog(10 107)
Speed of light
V.F
C.E.P
213, 10)
2.998 10
W Energy
V.F
V.E.P. NIA
NIA 6.626 x 10
1.34.77
W 27
Planck constant
V.F
C.E.P
4(1,5)
Frequency electromagnetic
V.F.
W.P.P.
((1.5)
Mog (10?, 1021)
V.E.P.
permittivity
V.F
C.F.P
((1.5)
x 12
1.3..1
r= drt (h/(2-)) mag
Planck constant
W.F
C.E.P 2(1.5)
6.626 x 10 34
m
Mass
W.F
V.P.P. 241,5)
2oog (30 10 10-20)
Electric charge
V.F
V.F. 4(1.5)
Mag (10 (10-11 10
U Internal energy
V. W.P.P. N/A
N/A (10 10") 2o (10x 10-3)
1.
==
P
Pressure
V.F V.P.P. 4(1,5)
V
Volume
V.F.
W.P.P. (11,5)
Einergy
V.F.
1.39.11
U= PV
Heat capacity
Pressure
V. W.P.P. (((25))
V.F V.P.P. 4(1.5)
Mmag100 102)
(4(1,)
Volume
WE W.P 211,5)
Zing (10-10 10-3)
D Diffusion coefficient
V.F. V.E.P. N/A
N/A (1g 1015)
Viscosity
V.F V.E.P. (015))
1.43.11
Boltzmann constant
V.F C.E.P 411,5)
1.381 x 10
Temperature
V.F
V.P.P 44(15)
(0g 10²)
17

---

Under review as a conference paper at ICLR 2023
Table S5: Medium set of proposed datasets (part 3).
Properties
Distribations
Eq. ID
Formula
Symbols
Original
Ours
Original
Ours
conductivity
capacity
V.
V.P.P
2(2,5)
24(1,2
Bolizmann constant
V.F
C.E.P
2(1,3)
1.381 10-23
Velocity
V.
WEP
2(1,3)
Uing (10", (10",1"")
Molocular collision cross section
V.F
V.R.P
211,5)
Mhox (10 2, 10 19)
E
Energy
V.F
V.R.P.
N/A
N/A Mog (10 10-277
Mass
V.F
V.R.P
2(1,3)
1.4.22
E
V1
Speed light
V.F
C.E.P
4(3,10)
10)
2.998 10
Velocity
V.F
V.P.P.
20012)
(10%, 108)
Electric potential
V.F
Vacuum permittivity
V.F
C.E.P
211,3)
8.854
IT.6.11
P.
Electric dipole moment
V.F
W.F.
2111,3
M10
10-20)
Angle
V.
V.
21113)
2(0, 2r)
Distance
V.F
V.E.P.
2(1.3)
Miex (10 10 10
Energy
V.F
V.F
NIA
N/A 11 10-
11.8.7
02 U=
Electric charge
V.F
V.F.
21(1.5)
(10
Stren
Vacumm permittviity
V.F
C.E.P
21(1.5)
8.854
V.F
211,5)
Mine (10
10 10
Position
V.F
N/A
Electric charge
V.F
W.F.
2111.3
Mlng
10-
E
Magnitude of electric lield
V.F
WEP
21(1.3)
Ming (1008 10 10-
I1.11.3
2
miup
Y73
Mass
V.F
V.P.P.
2(113)
Uma
10 10-20)
Angular velocity
V.F
V.F
21(3.5)
Mho (10" 10
Angular velocity
V.
VF
21(1,2)
(10 1022)
Flectric potential
V.F
V.F
NIA
N/A Mixx 1,10 1)
Flectric charge
V.F
W.
21(1,3)
permittivity
V.F
211,5)
8.854 10-12
11.21.32
4warl w/6)
Distance
V.F
V.R.P.
201,8)
Mix (109, 10
Velocity
V.F
V.R.P
211.2)
Mog (109, 10
Speed light
V.F
C.E.P
14(3,
2.998 x 10
Magnetic moment
V.F
V.F
N/A
NA (10 11, 10-9) -0
Fllctric charge
V.
V.
24(1.5)
1134.2
Velocity
V.F
W.F.
21113)
thox (10 10")
Radius
V.F.
W.P.P
21113)
M10 10-9)
Electric
V.F
Electric charge
V.E
211,3)
210 11 11009)
1134.2a
2
Velocity
V.F
W.F.
U(1.3)
Miog (10", 10")
Redius
V.F
V.R.P
U(1.3)
Vheg Mieg(10- (10-11 10-
Bohr magneton
V.F
V.F
N/A
N/A U1u 10
Flectric charge
V.F
V.F
21(1.5)
11.4.29
Planck constant
V.F
C.E.P
21115)
6.626
10
Mass
V.F
W.E.P
2111.5
Uing (10 10 28)
Emergy magnetic field
V.F
N/A
Magnetic
V.F
21115)
Ming (10 25
11.37.1
== x)B
Volume magnetic suscoptibility
V.F
V.F
2(1.3)
Miog (10", 10
Magnetic field strength
V.F
V.F
21113)
Uing (10 1,000 10
18

---

Under review as a conference paper at ICLR 2023
Table S6: Medium set of proposed datasets (part 4).
Properties
Distributions
Eq. ID
Formuta
Symbols
Original
Ours
Original
Ours
Average photons
V,F
Planck constant
V,F
C.F.P
2(155)
6.626
Prequency
V,F
V.E.P
4(1,5)
Ming (10%, 10
Bolizmann constant
V,F
C.E.P.
((1,5)
1.381
10023
Temperature
V.F
V.F.P
2(1,6)
Mox Mox 10%)
[C
Probability
V.F
V.NNN N/A
N/A 00ex
11...54
|C sin² (2
Energy
V.F
V.F.
4(1,2)
101 10-10) 18)
Time
V.F
VENN
(1(12)
[1eg
10-100
Planck constant
V.F.
C.F.P.
201,4)
6.626
10
Speed
V,F
Energy
V.F
W.F. 4(1.5)
Mog(10
18 10 16
18.13.18
2
Latlice constant
V,F
V.P.P ((1,5)
Ling (10 10
k
Wavenumber
V.F
V.P.P. 2((,
Umg (10"".10") 10')
h
Planck constant
V.F
C.E.P
4(1,5)
6.026 10
Electric Current
V.F
V.F
N/A
N/A 10
To
Electric current
V.F
V.F.
4(1,5)
M1
Flectric charge
V.F
V.E.P.
21(12)
M1e 22 20)
14 7 To (exp (gAV/eT)
1)
A
Voltage
4(1.2)
Mlg 1, 1,10')
Belizmann
4(1,2)
1,381 10-23
Temperature
V.F
V.P.P
4(1.2)
Moo 1
Energy
V.F
V.P.P
N/A
N/A 10-19)
Amplitude
V.F
V.P.P
4(15))
II.15.12
== 21
cas (kd))
k
Propagation coefficient
V.F
V.P.P
401,5)
(102(10-1.101
d
Lattice constant
V.F
W.P
40155)
M1x 1000000,10000 10 1)
Fifecttive
V.F
V.F.P.
N/A
N/A 6.026
Planck constant
V.
C.F.P
4(1,5)
14
Ag2
Amplitude
4(1,5)
21og
10
Lattice constant
V.F
W.P.P.
4(1.6)
Mix (10 10 10
Distribution
V.F
V.F.
N/A
N/A Uee 18 10 10 Meg
Variable
V.F
V.E.P
4(1,5)
II.17.37
88(1 cns
Variable
V.F
V.F.
4(1,5)
16)
Angle
V.F
V.NN
40155
21(0,25)
Fieergy
V.F
V.P.P. N/A
N/A
Mass
V.F
V.F.P.
4(1,5)
M1ex 30 10 28
11.19.51
E 2(4ma) (N/(2)))
g
Electric charge
Vacuum permittvvity
2((15)
4(1,5)
Ming
10-12
9)
Planck constant
V.F
C.
4(1,5)
6.626 10-84
Number of protons
V.F
V.PP
4(1.5)
thog(10" 10
Variable
V.F
V.E.P
N/A
N/A
F
Fllectromaneticc energy
V.F
V.E.P.
2(1,3)
2102110 10 10-22
B8
U
Flectron mass
V.F
C.F.P
4(1,3)
9.9
10
mi
Speed of light
V.F
C.E.P
41(13)
2.998 10
Incidence angle
V.F
V.F
401,3)
24(
Variable
Gravitational constant
V.F
C.E.P
4(1.5)
6.674 11
B18
BFC
H²
ly
Variable
Speed of light
V.F
V.F
C.E.P
V.F. 21(15)
4(1.5)
Mo0 1 10")
2.998 x 10"
Distance
V.
V.F.P
21(15)
(10 1020)
Variable
V.F
V.F
(1(15)
M0g 10a)
19

---

Under review as a conference paper at ICLR 2023
Table S7: Hard set of our proposed datasets (part 1).
Properties
Distributions
Eq. ID
Formula
Symbols
Original
Ours
Original
Ours
Prubahility density
V.
N/A
16.20
f exp
20²
/2
tion
Position
V.F
V.F
4(13) Mox (10 1,00-11.0: 10+)
Standard deviation
V.F
V.P.P
4013)
Miog (100 10') 10')
16.20a
/ = exp
12
/v2x
Prubahility density
tion
Position
V.
V.
4(13)
Mox (10 1, 10*))
Probability density func-
V.F
V.F
NIA
N/A
1.6.006
f = exp
18-9 20
/v2no
tion
Position
V.F
V.
40.3)
Muu (10 10²)
0 Position
V.F
VE
4013)
Mux 10a)
Standard deviation
V.F
V.P
4013)
Ling (wu-1,
F Force gravity
N/A 10-11
G
Gravitational constant
V.F
2(11) 2)
V.F
V.P.P M(11)
M1g(110, 10
m2 Mass
V.F
V.P.P 20(1)
M1g 108)
F
19.18
Gma mgg
22
Prsition
Prsition
V.F
V.
W.F. 24(3.4)
W. 24
102(119,100 10
(102
(22 21)² + (y2
21)² + (=2 21)
Position
V.F
W.F 21(1.2)
Unng(10", 10")
Position
V.F
V.F. 2413,4
Ume(110 "0000"_1000" 10')
Position
V.F
V.F 21
2 10*
Position
V.F WE 14(3,) 4)
(100(10) 10²)
(1 Time
V.F
N/A 2hog
V.F V.E.NN
20 5)
1.15.3
t-nz/d2
Velocity
V.F
M((12)
Mix (100 107)
Vi-utj22
4 Position
V.F
W.F 2((1,5)
Mhog (10%, 11100110000 102)
Speed of light
V.
C.F.P
21(3, 10)
2.998 x 10"
Position
V.F
W.F.
NIA
NIA (100 10a)
Position
V.F
V.F. 21 10)
L15.3x
#1
V1 21-u2/00
Volocity
V.F
V.F. 44(12)
(10g(10; 108)
Time
V.F
W.
20122
Meg 1010" 10-4)
Speed light
V.F
C.E.P (13,20)
10
Wavelength
V.F.P.
NIA
Wavelength
V.F V.P.P 2111
5)
24ng (10 10-)
129.16
22
Wavelength
V.
V.P 24(15) 5)
Mong 40o
(10-1,1 10²)
m + + + 2m ara cos (0: 0a)
81
Angle
V. V.E.NN 24(1.) 5)
u((0, 2n)
82
Angle
V.F
VN 24(15)
u((0, 2n)
Amplitude of combined wave
V.
V.F
NIA
N/A
1.3.3
To sin ((0022)
(8/2(
To
Amplitude of wave V.
V.F.P. u(11 5)
M0e 10 1
The number of waves
V.
V.I.P
2(1.5)
10a)
Phase difference
V.F
KF
40.5)
41-2r, 2m))
P Energy
V.
V.P.P.
N/A
N/A
Vacıum permitivity
V.
C.E.P
20(1)
8.854 10
Speed light
V.F
2(1,2)
10
132.17
2 P (zecE2)
{
E Magnitude of field
4(1,2)
Ming(10 160)
Radius
V.F
V.E.P.
(4(1.)
Uiog (10-2, 10
Frequency of electromag- netic waves
V.F
V.F.
24(12)
Mux Mox 10
Wp Frequency of electromag-
V.F
V.F.
24(3, 5)
Mon (100 10
waves
W Frequency of electromaa-
V.F
W.F.
N/A
NA
14w/o
netic waves
134.14
373
Velocity
V.F
u(1,2) 2)
Ueg(100; 108)
Speed of light
V.F
C.
24(3, 10)
10
Up Frequency of electrmag-
V.F
F.F.P
M(1,5)
Mug 10"!)
netic waves
20

---

Under review as a conference paper at ICLR 2023
Table S8: Hard set of our proposed datasets (part 2).
Properties
Distributions
Eq. ID
Formula
Symbols
Original
Ours
Original
Ours
Amplitude
V.E.P
112 = +
11
Amplitude wave
V
V.E.P
40158 5)
Mog 1 10-3)
1.3.4
+ +2/5222 c058
12
Amplitude wave
V.F
W.F.P
u(1, 5)
Ming Ming(10 .010
Phase difference
W.F
W.F
4(15))
24(U, m)
P Pressure
V.F
V.E.P
NIA
N/A M102 10
Number of molocules
V.F
V.I.P
W0, 5)
1.39.22
P= T
Boltzmann constant
V.F
C.R.P 4(15))
1.381
10
Temperature
V.F
V.E.P 4(15))
(10' 10")
V Volume
V.F.
V.E.P. M(1.5)
Mugg (10 10
density
W.P.P.
no
Molecular density
V.F V.F 4(1.5)
Mog 10
Mass
V.F
W.P.P 41,, 5)
Mmeg(10221 10 10-22)
1.40.1
11 ng exp mggz (
Gravitational acceleration
V.F
C.F.P 4(15))
9.807 x 10
Height
V.F
V.E.P 4(1.))
(10 10%)
J:
Boltzmann constant
V.F
C.E. 4015)
1.381 x 10
T
Temperature
V.
V.F.P. 40.5)
Mug 109)
Lua Rudiation per frequency
V.F
V.P.P. N/A
N/A 6.626
Led h
Planck
C.E.P
M(1,5)
Frequency electromagnetic
4(1,5)
Uig M10 (.0))
141.16
wave
(cc((xxp(hx/221k 1)
Speed of light
V.F
C.E.P 4(1.5)
2.998 10
Bolizmann
V.
C.F.P
M(15))
1.381 10
Temperature
V.F
V.E.P.
M(1.5)
Mog (10', 109)
Q
Energy
V.F
W.F.
N/A Mex 1026)
Number of molecules
V.F
V. M(1.)
144.4
Q nkT In( Va
Boltzmann constant
V.F
C.E.P 4015))
1.381 x
T
Temperature
V.
V.P.P 4115))
Mog 10a)
V2
Volume
V.F
V.E.P u(1.5)
5)
(10 -. 10-2)
Vi
Volume
V.
V.F.P 40.5)
Moe (10 10
Amplitude
V.F. V.F NAA
N/A 10')
K
Amplitude
V.F. V.F.P M(1,3
(1x
1.50.26
(cos cos? wt)
Angular velocity
V.F
V.F. M(1,3)
M1x 102)
V.E.NN
401,3)
M1o
1001)
Variable
V.F
V.F.
u(1.3)
Miog (10 -3
E Electric field
V.F.
V.
N/A
N/A
Electric dipole moment
V.F
V.F.
W11, 3)
Meeg(10 ?2, 10
Vacuum permittivity
V.F
C.E.P
u11 3)
8.854 x 10-12
116.15a
E
Position
V.
V.F. 4(1,3
3)
Mux 10, 10 8)
Distance
V.F
V.E.P W(1,3
Mix (10 10
Pusition
V.F
V.F 401,
21og (10
(10-10 10-5)
Position
V.F.
V.F.
M(1.3)
Mine
E Electric
N/A
Electric dipole moment
V.F
V.F.
M(1,
Ming Mieg(10 (10g(1022 22, 10-20) 8.834
11.6.15b
P. coo
Vacuum permittivity
V.
C.E.P
2(1,3)
10-12
Angle
V.
V.F.
4(1,3)
14(0, F)
Distance
W.F
V.F.P
40133 3)
Uix (10 10 10 -8
Number of polar molecules
V.F
V.F.
NIA
N/A
per angle per unit volume
Number of malecules per unit volume
V.
V.F.P.
40,3)
Mu (10P7, 1022)
11.11.17
11+
po
dipole field
4(1,3)
Mog 40g
Magnitude Angle
4013)
Moz 102)
V.F
V.E.NN
141,3)
21(0, 2m)
Boltzmann constant
V.F
C.E.P
24113)
1.381 10
T
Temperature
V.F
V.E.P
4(1.3)
Mox (10* 10a)
P
Polarizahility
V.
V.F.
NIA
N/A Miz (1023 1025)
Number of atom
V.F
V.I.P.
u(1.5)
5)
Po Electric dipole
V.
V.F.
4(1
(10 12-100000000000000 10-2
1L.1.20
E
Magnitude electric field
V.F
V.F.
M(1,5)
M1v 102)
Boltzmann
C.E.P
4(1,5)
1.381 10 10-28
Temperature
4(1,5) 5)
Mng 103)
21

---

Under review as a conference paper at ICLR 2023
Table $9: Hard set of our proposed datasets (part 3).
Properties
Distributions
Eq. ID
Formula
Symbols
Original
Ours
Original
Ours
Polarizability
Number
V.T,,
M(0, 1)
(1x 10
TT.11.27
Molecular polarizability Q
V.F
V.E.P
u(00, 1)
Ming 10-311
Vacuum permitiviity
V.
C.E.P
2(1,2)
8.834 x 10-12
E
Magnitude of electric field
V.F
V.F.P
2(1,2)
Mox(10",,
K Electric dipele momennt per unit volurne
V.F
V.F
NIA
N/A
11.11.28
Nux
N Number dipoles
electric
V.F
V.I.P
U(0, 1)
Mog 1025)
Molecular polarizabiiity
V.F
V.E.P
u(0,1) 1)
(10
31
Electric charge density
V.F
V.E.P
NIA
NIA
1113:23
V1 c/
Po
Electric charge density Velocity
V.F
V.E.P.
2(1,2
4(1.5)
(10 1022)
M10 10a)
Speed of light
M(3,
10)
10
Electric current
V.F
V.F
N/A
IL13.34
Vi-u2/22
P0W
Po
Electric charge density Velncity
V.F
V.F
W.P
W.P
20122
4(1.5)
Muu ( 10
Mox (109, 109)
Speed of light
V.F
C.E.P
443, 10)
2.998 10
Wavenumber
V.F
V.E.P
NIA
NIA Miog (10", 10"))
11.24.17
== Vw3/0²= 02/0²
Angular velocity
V.F
V.F.
4(4.6)
Speed of light
C.R.P
2(12)
2.998 10
Length
V.E.P
4(2, 4)
Mog (100 10-1)
Number atoms with the equivalent magnetic moment
V.F
V.I..P
NIA
N/A
11.35.18
N
N Number of atoms per unit volume
V.B
V.I..
u(1,3)
(10 1100s,100000 10005
exp(uB/RT) exp(-xBB/kk B/kTTT
Magnetic moment V.
V.F.P.
P
u(1.3)
Moo 22 10 10-23)
B
Magnetic flux deasity
V.P.P.
4(1.3)
Mane 110-1)
Boltzmann constant
V.
C.E.P
4(133)
1.381 x 10023
T
Temperature
V.F
W.P.P.
201,3)
Moo Moo 10?)
M Number of magnetized atoms
V.F
V.
N/A
N/A
11335.1
M = Nytanni (eB
Number of alom Magnetic moment
V.F
V.F
V.I..P
W.P.P.
2(1.5)
2(1,5)
Moy 10-23)
210 1028)
B
Magnetic flux deasity
V.B
V.E.P.
4(1,5)
Mine(1 3, ..101)
k
Boltzmann consiant
V.
C.E.P
4(1, 5)
1.381 x 10-23
T
Temperature
V.
V.F.P.
20,55
Mo0' 102)
Parameter of magnetiza-
V.F
V.F
N/A
N/A
tion
Magnetic moment
V.F
V.F
4(1,3)
Mog 10 10-23) Mox
Magnetic field strenghh
V.
V.F
20,3)
10 1010011 1)
11.36.38
pl
k
Boltzmann constant
V.
C.R.P
2(133)
1.381 x 10 28
cc
T
Temperature
V.F
V.E.P
u(1.3)
Mnn 10²)
Constant
V.E.NN
(01.3)
14(0, 8.854 10 12
Vacuom permittivity
C.E.P.
u(1.3)
Speed of light
C.E.P
201,3)
10
Number magnetized atoms
V.T.P
20,,3
Mug Mug 1625)
E Energy
V.F
V.F.P
N/A
4(1.5)
111..33
F 2rloxptha/3 25
Frequency
2015))
Mog M10 (10
10
Boltzmann constant
V.F
C.E.P
4(1.5)
1.381 x 10 28
Temperature
W.
W.P.P.
4(1.5)
Mone 10
P1-11 Probability
V.F
V.EN
N/A
NIA 10-20
Electric dipole moment
V.F
V.F
4(13))
Moe
Pr
E
Magnitude of electric field
V.F
V.F
2(1.3)
Uon o, 102)
111.9.52
(2npEt
sin? ((w wa) t/2)
(w w/) 1/2)2
Time
V.F
V.E.NN 2(1,3)
M10 18, 10
Planck constant
V.F.
C.R.P P
2(1,3)
6.626 x 10 34
Frequency
V.F
V.E.P.
2(1.5)
2ing (10%, 10
Resonant frequency
V.F
V.E.P.
2(1,
Mox Mor(10 10
22

---

Under review as a conference paper at ICLR 2023
Table S10: Hard set of our proposed datasets (part 4).
Properties Original
Distribations
Eq.
Formuln
Symbols
Original
Ours
E Energy
V.F.
V.F
NIA
NAA
Magnetic moment
V.F
V.F
M(1, 5)
toe (10 10-200
19.10.19
E E=
B
B. Element magnetic field
V.F
V.F
M(1,5)
Mox Mox (10 3, 9.10
B3 magnetic field
V.F
M(1,5)
5) Mogg (10
1
13 Ellement magnetic field
M(1.))
Mwy(110 (10
90100-
Electric Current
V,F
V.F
N/A
NIA Ming Ml02r 10
Electric charge density
V.F
V.EN
u(1, 5)
11.21.20
J -
Electric charge
V.F
V.N
4(1,5)
Ling (100
11009
A Magnetic vector potential
V.F
V.F.
W(1 5)
2hox (10x110-)
10100-110000 10-1
Mass
V.F
V.P.P
W(1, 5)
Uax (10 10 25)
Differential scattering section
V.F
V.F
N/A
21 Atomic number
M(1
M10g 10
Zg Atomic
2(12))
M1o0(109; 10!)
B1
A
stn (979)
2, Zyche
Dirac's constant
Fine structure constant
V.
V.F
C.E.P (11.2)
C.E.P ((1.5)
1.055 x 10-34
7.297 x 10
Speed of light
V.F
C.F.P 4(1.2)
2.998 x 10"
E Non-relativistic kinetic energy
V.F
V.P.P. P 4(1. 3)
Maa (10 18, 10 16)
0 Scattering ungle
V,F
V.E. NN 4(1.)) 3)
24(0, 2m)
Variable
V.F
V.F
N/A ((13)
N/A Vog Mog (10 1022)
mkg
Mass (The Earth)
V.F
W.P.P.
1,2
ka Variable
4(1.3)
Uing (10 1)
B2
2F12
Distance
W.P 4(1
Uing (10% 10
1 + 11+
cos
Energy
V.F. W.P M(1,3)
3)
Mox (1026 1027)
m
61 Angle
W.
V. NN ((0 6)
U( 2
0g Angle
W. V.E. NN 24(0, 6)
24(0, 27)
Distance
V.
V.E.P N/A
N/A 2oo (10 10"o)
d Semimajor axis of elliptical arhit
V.F
V.E.P. 4013)
B3
1+0 co 993)
d01-a2)
Orbital eccentricity
V.F
V.E.P. 44(2, 4)
u(001)
0. Angle
V.F
V.E.NN (4(4, 5)
u(U,, 2m)
Angle
V.F. V. ((4.5)
24(0, 2tr)
Velocity
V.P.P.
m Mass (The Eanth)
V.F V.P.P. 2(1,))
Mog Mag 1028
B4
1.2
E Einergy
Potential energy
V.F W.P 4013)
V.F W.P 24(8, 12)
Ming Mio
Mog (10 10
1027
Angular momentum
V.F
V.F u(1, 3)
Ung(10* 10
Distance
V.F
V.P.P u(1, 3)
Uine (10* Mmn 10"0)
Orbital period
V.F
V.E.P.
N/A
NA Mon(110 10
Semimajor axis of elliptical orbit
V.F
V.P.P.
((1,))
B5
3/2 VC(mg m2)
Gravitational constant
V.F
C.F.P ((1,3)
6.674
10
my (The Earth)
V.F
W.P.P
4(1
Ming (10 10
ma Mass (The Earth)
W.P.P.
1013)
Ming (1029 1025
Orbital eccentricity
V.F
W.P.P.
N/A
N/A
Energy
V.F
V.F u(1, 3)
Ming (10 18
10 16
E Energy
V.F
V.P.P 4(1, Ling M10
10 16
B6
212g12 22 2
L Distance
Mass
V.F
V.F
V.P.P.
V.P.P.
W(1, 3)
4(1, 3) Mix (10 (10-
Min (10
10 26)
21 Atomic number
V.F
V.PP
4013)) Unog (109, 10
Zg Atomic number
V.F V.P.P
4(133)
Uhoz (10", 10
9 Electtic charge
V.F
U(1,3)
Uug (10 10
Hubble's a constant
W.F
W.P.P. N/A
N/A 6.674 x 10-11
Gravitational constant
W.
C.P.
4(1.3)
B7
H
2
Density of the Universe
V.F
W.P.P.
M(1, 3)
Max (10 27 10
47
ky Spacetime curvature
V.F
V.I
((1.2)
u(-1, 1)
Speed of light
V.F
C.P.P.
u(1
2,998 x 10
Radius Gravitationa energy
V.F
V.E.P.
u(1.3)
Uog Mog(10 10
V.F
V.F.
N/A
NAA 6.674 10
Gravitational
C.E.P
((1
Speed light
u(1
2.998 10"
B9
G*
(my 1agmmgaa [(m mg)
ma Mass
V.F
V.P.P. M(1
Mox Mag(1029 1028
c
my Mass
V.F
V.P.P M(L
Mog 1025
Distance
V.F
W.P.P
411 2)
Mog(100, 1019)
23

---

Under review as a conference paper at ICLR 2023
Table $11: Hard set of our proposed datasets (part 5).
Properties Original
Distributions
Eq. ID
Formula
Symbols
Original
cos Value
V.F
V.F N/A
N/A
B10
cos 01
ooe diym ma off (T-uye
Angle
Velocity
V.F
V.F
V.F
(11,3
((1,))
Moo (10", 10?)
M(0, 2x)
Speed light
(4(4))
2.998 10
Wave inteasity
V.P N/A
NIA Mug (10 10 10"1)
To
Amplitude wave
V. V.P.P (4(13)
B1I
== Zo
sinia/ (2) 0/2
sina N66 (2)
Wavelength of X-ray
V.F V.E.P 4(1,3)
14ng 11 10
P)
#Im(5/2) 23
N
Number of phase difference
V.F. V.LP 4(1,2)
Ling (10", 102)
8
Wavelength of X-ray
V. V.F.P 211,3)
Moel (10-1) 10"
Force
V.F V.F N/A
NIA Mor 1010-
Electric charge
V.F 4(1,6)
4rey²
Vacuum permitivvty
24(15)
8.854
B12
4v
(v² d2
gdy"
y
Ve
Distance
Voltage
V.F V.F. 4(1,5)
240,3)
Uug (10
Mu, 10%)
10
Distance
WEP ()4,
Mog (10 10
V.
Potential
V.F. V.F N/A
NAA (10 10 10)
permittivity
V.F. V.E.P ((1,5)
V/
Electric charge
V.F V.F 4(1,5)
Moz
10-1)
Distance
V.F. V.E.P 201,3)
M1og110
1 10")
Distance hetween dipoles
V. V.E.P (1(4,)
(10 10")
Angle
V.F 4(0,6)
u(00, 2x)
Patential (out)
Magnitude electric field
W. V.F 4(1.5)
M1g 10²)
Angle
V.F. V.F 24(0,))
1/(0, 2x)
B14
V,, Bj cos
Distance
V. V.P.P ((1,5)
10%)
Radius of dielectric sphere
V. V.E.P (((15)
2hog 1 2hox (10 10")
Polarizability
V. V.F 401,5)
600
Frequency of electromugnetic
V. V.F N/A
N/A 1hu M00 107)
Velacity
V. V.P.P. (11,3)
B15
Speed light
CL 20)
2.998 10
Frequency electromagnetic waves
V.P.P (4(155
Moa 10
Angle
V.F (100
M(0, 2x)
Energy
V. V.F N/A
N/A 10
Momeutum
V.
V.F 4(1,5)
M1g
qVV
Electric charge
V.F. V.F 27(15)
(10
10 10-9)
B16
Vip 92) c2 + m²
Vector potential
V.F V.F 4(1,5)
Uiux (10x 10a)
Speed of light
V.F C.E.P (11,5)
2.998 10
Mass
V.F.P (11,5)
Mox (10030 10-2%)
Voltage
V.F V.F U(1
2ho 10
Energy
V.F
NIA
1
m
Mass
V.P.P 4(1,5)
(10 (10 30
10 28 10-7)
B17
2m
P
Momentum
Frequency of electromagnetic waves
V.F.
W. V.E.P ((1,5)
V.F 24(15)
Mon (10
U0n 1)
p 2w?
(1+ a-
1
Position
W.
V.F 4(1,5)
(1n
10-2)
Devintion from the harmonic ascillator
V.F.
V.F 2(1,5)
2ho (100 10')
Distance
V. V.P.P 411
Moe(10 (10 11 10
Pressure
V. V.F N/A
N/A 6.674
Gravitational
((1,5)
Dr
Speed light
2(1,5)
2.998 10a
é kg
Variable
V.F. 4(1.5)
Mog (10 10a)
c HH2 2
2a)
de
Distance
W. WEP 4(1.5)
(100 10
H
Variable
V.F. V.P.P 21(55)
Mon Mox 103)
Variable
V.F V.F 4(1.5)
24(100 10)
A
Differential cross section
V.F V.F. N/A
N/A
Fine structure constant
V.F. C.P.5
7.297 10
a² 2
Planck constant
V.F. C.E.P ((1,))
6.626 10
A
Electron
C.E.P 4(1.5)
9.109
B20
Lio
Speed light
(11.5)
10"
w0
sin?
w00 Frequency
V.P.P. 211
Moz(10' 1021)
Frequency
W.P.P (1(,5)
Mox (00( 10-1)
Scattering angle
V.F V.F (4(06))
4(00, 2m)
24

---

Under review as a conference paper at ICLR 2023
B HYPERPARAMETERS FOR FIVE EXISTING SR BASELINES
Table ST2 shows the hyperparameter space for the five existing symbolic regression baselines. The
hyperparameters of gplearn (Koza & Poli 2005 AFP (Schmidt & Lipson 2011). and AFP-
FE (Schmidt & Lipson 2009) 7 are optimized by Optuna (Akiba et al. 2019). hyperparameter optimization framework.
Table S12: Hyperparameter sets for the five existing symbolic regression baselines.
Method
Hyperparameter sets
gplearn
100 trials with random combinations of the following hyperparameter spaces:
population.size: ,(102,, 100), generations: U(10, 10²).
stopping-criteriar u(10-10, 10 2), warm_start: {True, False),
const_range: (None, (-1.0, 1.0), (-10, (-100, 10a), (-10%, 10t), (-100,
max-samples: (((0,,, 1.0), parsimony u(10 10
AFP
100 trials with random combinations of the following hyperparameter spaces: popsize: (1000, 1000), g: (2550, (520), stop stresshsshood: 4(10-10, ,10-2). op_list: {['', 'V", '+', "-" **,, 'f, "exp'. 'log'. "."2," '2'''', "s'rrti,' I'n', 'v', "4, "#, ",,, eexp', 'log', '2', '3', 'sqrt', "sin', 'cos') "]]
AFP-FE
100 trials with random combinations of the following hyperparameter spaces: popsize: ((100, 1000), g: ((250, 2500), stop thressollhh: 1(10-19, 1002), op_list: [['n', 'V", '+', "-, "*, 'f, eexp', 'log', '2',"", "s'rr'''
I'n', 'v', "2'+"" 1-, 7,, 'exp', 'log'. '2', '3', 'sqrt', "sin', 'cos'))
AI Feynman
(bfit: 60, epoch: 300, op: '7ops.txt, deg: 3}.
(bftt: 60, epoch: 300, op: 100ops.ixt',pol poly-ollygeeee deg: 3},
(bftt: 60, epoch: 300, op: '14ops.xt',
3}.
(bftr: 60, epoch: 300, op: 'I9ops. poly deg: 3).
(bftr: 120, epoch: 300, op: "4opps.xxt, plgy: 4),
(bftt: 120, epoch: 300, op: '19ops.txt, poll-ooyleeee deg: 4}.
(bftt: 60, epoch: 500, op: '7ops.txt', poly
poops.tt'.ppolyleee slg:: 3).
(bfir: 60, epoch: 500, op: '10ops.xxt', poly deg: 3).
(bfit: 60, epoch: 500, op: '14ops.xxt' poly 3).
(bfit: 60, epoch: 500, op: '19ops.tt',, deg: 3}
DSR
(seed: 1, function.set: ['add', 'sub', 'mul', 'div', 'sin', 'cos', 'log') ].
(seed: 2. function_set: ['add', 'sub", 'mul", 'div', 'sin", "cos',"exp' 'log') ).
(seed: 3. function_seet ['add', 'sub', "mul", 'div', "co.'''o', 'exp', 'log') }. "mul", 'div',
{seed: 4, functionn_set: ['add",
'log');
(seed: 5, functionnnsee set: ['add', 'sub', 'mul', 'div', 'sin', "cos", 'log'), ).
(seed: 1, f.fuuctionsset: ['add', 'sub', 'mul', 'div', "cos", 'exp'. 'log'. 'const") }}.
(seed: 2, function set: ['add', 'sub', 'mul', 'div', "cos", exp'', 'log'. 'const" }]
(seed: 3, function_set: ['add', 'sub', 'mul', 'div', 'sin', 'cos", 'epp'. 'log'. 'const" }}
(seed: 4, function_set: ['add', 'sub', 'mul', 'div', 'sin', 'cos', exp', 'log', 'const") }}.
(seed: 5, functtioon ['add', 'sub", 'mul', 'div', 'sin", 'cos", exp', 'log", 'const']]
http:///ggeeerrrrhhhoosssseeeeee http:://githubb.com/caaaaaaa
25

---

Under review as a conference paper at ICLR 2023
C SYMBOLIC TRANSFORMER
In this section, we describe our Symbolic Transformer baseline and pretraining strategy, and discuss the degree of the test data leakage during the pretraining. We repeat that while the Symbolic Transformer itself is a new model, the main contribution of this work lies in the datasets and benchmark of symbolic regression for scientific discovery (SRSD). We also note that our Symbolic Transformer is not necessarily designed to show improvements over the existing Transformer-basee symbolic regression models VValipour et al.] [022]] Biggio et al., 202I) including contemporary work such as Kamienny al. (2022). Our Transformer baseer--bbeedeeeee bascline simply inspired by recent advances in deep learning, specifically Transformer- based high-performance. modern, and flexible models such as Vaswani al. 2017); Devlin al. (2019): Dosovitskiy et al. (2020)
C.I ARCHITECTURE DESIGN
A Symbolic Transformer is encoder-decoder network to predict binary equation tree correspond- ing to imput tabular data points, The encoder inputs sampled tabular data from an equation and outputs variable-wise features. The decoder inputs them and outputs a preorder sequence of tokens in an autoregressive manner to build binary equation tree.
First, we describe an encoder that inputs tabular data and outputs variable-wise features. The tabular data here have the following properties:
Each row is a data point sampled from the expected equation
Each column corresponds to the input variable of the equation (I1, 22, .. or the output of the equation (y).
For such data, the relation between the input tabular data and the output equation is desired to have the following properties:
coomuutationinvvaiinn each row; since there is no meaningful order between each sampled point, the output equation is expected not to change matter how the rows are rearranged.
Permutation-equiit euqivarraitti for each column and variable; when the columns of the input tabular data are rearranged, the order of variables in the output equation is expected to change corresponding to the input column. (e.g., assume the formula 21 T2, if the æn column and the 22 column of the input tabular data are swapped, the expected output is y = 22 x1-)
To handle such crrmuuatiioninvariann and cetpuuttattoonqqivviiiiee data structures, the ideas of PointNet (Charles al.) 2017). which handles 3D point clouds, and Deep Sets (Zaheer et al. (017). which handles set data, can be referred to. The approach proposed in these papers is to utilize point-wise transformation using Shared Multi Layer Perceptron (MLP) (permutation-equiiarria operation) and aggregation using Pooling (permutation-invariaan operation).
When applied to tabular data, sample point-wise feature be obtained by combining Shared MLP and column-wise (variable-wise) pooling. Similarly, a variable-wise feature can be obtained by combining Shared MLP and row-wise (sample point-wise) Pooling. In both operations, by aggregating once and then combining the features with the features of each cell (this idea is used in semantic segmentation network in PointNet (Charles et al., 2017)). can expected that the whole information of aggregated direction will propagate to each cell. We call this operation feature splatting. Note that it permutation -uquiaaiiiinnnuuuiiaaannaaa operation.
The proposed network architecture is shown in Fig. ST The proposed encoder consists of two encoding blocks, each block consisting of 1.) Shared MLP. 2.) row-wise (sample point-wise) pooling, 3.) ris- feature splatting, 4.) Shared MLP, 5.) column-wist
wariibbeewiie pooling, and 6.) column-wise feature splatting. The encoder is permutation catiiiiiiiuvvaiiinn, and information of each cell propagates to both the row-axis and column-axis Finally, row-wise (sample point-wise) pooling can obtain the variable-win wise feature.
The decoder of our Symbolic Transformer is Transformer decoder (Vaswani et al. 2017) ) that takes variable-wise feature and a preorder sequence of tokens (to express the equation as a binary equation tree) input and outputs the next token. The next token is estimated with Multi-head Attention
26

---

Under review as a conference paper at ICLR 2023
2g
Tobaker Unia
Surplc pansonte
Varahic-eie
Fomur
Varafonnie
T2a
Seeple Fanter
Encoder Block
X
A
Muds Jcnd
Mali-eet
Noraa
Nomn
Outer
Sepance (Baiha))
Transformer Decoder Block
X
Figure S1: Architecture of Symbolic Transformer.
Binary Equation Tree
Preorder Sequence [=808>]]
NS006
NN006 "MF.
Tebular Data
Encoder
Decoder
Decoder
Decoder
Mui
Mui
Mui
Mui
Mui
>-00 Mul, 10",
N-00s> "Mef. 'C",
"Mu", XK]
N-00> "M.. CC, MM' 'X, X
Decoder
Decoder
Complete Binary Equation Tree f(X, X2) CX,X2
Figure $2: Illustration of Symbolic Transformer's inference process for given tabular data which consists of 1,000 samples. Each sample is a 3-dimensional vector whose first two and last values correspond two variables and one target value, respectively. Yellow nodes indicate next tokens (nodes) to be predicted.
over the wiriable-wise feature from the input tokens. During training, use the teacher forcing method (Williams Zipser 1989) and feed the first (j - 1) tokens the ground-truth preorder sequence to the decoder for predicting the next (j-th) token and computing cross-entropy loss with scheduled sampling strategy (Liu et al. 2021a). During inference, the decoder predicts the next token autoregressively. using a sequence of previously prediicted tokens input tokens illustrated in Fig. [S2 From the resulting preorder sequence tokens, the binary equation tree is generated as the output of the Symbolic Transformer method.
27

---

Under review as a conference paper at ICLR 2023
For each training batch, we also mask next token candidates the Symbolic Transformer model will know the potential maximum number of variables in the target equation from the shape of the training batch. Suppose that a training batch consists of tabular data whose shape is (1,000, 3) [ the decoder should not predict three or more unique variable tokens as part of the preorder sequence.
It should be worth noting that when predicting the next token of preorder sequence, we take advantage of properties of binary equation tree for efficient inference. Since we predict a preorder sequence build binary equation tree, we can terminate the inference of the next token when all the child nodes in the binary equation tree built from the resulting sequence expresses are tokens for either constant or variable symbols i.e., the binary equation tree is complete and cannot be extend anymore, thus we can terminate the inference without predicting "<EOS>", ", the end of sequence token used in standard seq2seq tasks (Sutskever et al. 2014).
C.2 PRETRAINING AND HYPERPARAMETERS SER.PARAMEEEES
Here, we describe our pretraining strategy for the Symbolic Transformer baseline. As described in Section C.I.I J. the decoder of the Symbolic Transformer is trained to predict a next token of a preorder sequence to express the true binary equation tree, and thus requires both 1) true sequence of the tokens and 2) tabular data (samples).
The 120 SRSD datasets proposed in this study, however, should not be directly used for pretraining the Symbolic Transformer baseline. Since the datasets are designed to assess the potential of symbolic regression methods towards scientific discovery, we cannot use the SRSD datasets for both pretraining and testing the Symbolic Transformer baseline. Otherwise, it will result in having the baseline method learn from the pairs of tabular data and true sequences during training session and test the method using tabular data from similar distribution, which may be considered test data leakage.
Moreover, pretraining datasets are usually significantly larger than the target datasets (which are the SRSD datasets in this study) (Devlin et al. [019 Liu et al. 2019 Dosovitskiy et a 2020). To address the issues, we generated a large-scale set of synthesized datasets for pretraining the Symbolic Transformer baseline. Specifically, we built bi-gram Naeve Bayes language model to generate preorder sequences of tokens that express binary equation trees like those in the SRSD datasets. Having randomly generated target equation for pretraining, we randomly chose & for each of the variables in the equation independently to define its sampling range (10 (10k=1,00++)) Given a generated equation, we created up to 10 sets of tabular datasets with different sampling ranges we randomly chose as described above.
Using the approach, we generated 24,232 synthesized datasets. In the datasets, there are 4,501 gener- ated equations, and 1,841 of them are unique sequences. We pretrained the Symbolic Transformer baseline on the synthesized datasets for three epochs, using the SGD optimizer and a linear learning rate scheduler with warmup (1,000 steps) where the initial learning rate and weight decay are 0. and 10 respectively. A training batch at each step consists of 1,000 random samples from one of the synthesized datasets, and one epoch in this pretraining consists of 24,232 training steps. The Symbolic Transformer is implemented with PyTorch (Paszke et al. 2019). 9) and we implemented the
pretraining and evaluation sessions, using torchdistill (Matsubara 2021 for securing reproducibility of the experiments.
C.3 ARE THE TEST DATASETS LEAKED WHILE PRETRAINING SYMBOLIC TRANSFORMER?
In terms of normalized edit distance we proposed, the Symbolic Transformer baseline performed the best among our baseline methods for the SRSD datasets as shown in Section [3.3] To achieve the results, the Symbolic Transformer baseline model took advantage of being pretrained on a large-scale set of synthesized datasets we generated (See Section C.2). From the performance gap between the Symbolic Transformer and the other baselines, it is reasonable to suspect that the test set of our SRSD) datasets may be leaked during the pretraining step. Given a pair of synthesized training dataset XR and an SRSD test dataset Xs. we assess the degree of the leakage, following the two steps:
'The last (rightmost) column in input tabular data indicates the target values, thus should not he input. variable for the equation to be predicted.
'https: / /twgg//ggggnn ng ace. co /tt.d/s///rannfoomrrr//mainnn schedul Test
asses/ opt imp.mmmimm inear i wwwie__wtttt_hee
28

---

Under review as a conference paper at ICLR 2023
Step Compute normalized edit distance (Eq. (3)) between the true equations in the two datasets.
Step 2 If the resulting normalized edit distance (Step 1) is zero, then compute intersection of union
(IoU) between sampling domains of the datasets by Eq. (ST) for each input variable:
max (min (Xx) min (Xg)) min (max (Xx) max (Xs))
(S1)
min (min (Xt) min (Xg)) max (max (Xg) max (Xg))²
where Xj and X are of sampled values in the i-th variable for the given synthesized training dataset and the SRSD test dataset, respectively.
Following the procedure above, we assess the degree of the leakage for all the combinations 120 SRSD) datasets versus 24,232 synthesized datasets for pretraining. We take equation-wise average of the loU values and then compute the average IoU over the 120 equation-wise average loU values. The resulting mean IoU is 0.00215, and the IoU defined Eq. (SI) designed be normalized in range of 0 to 1. From the result, we conclude that there is no significant leakage from our SRSD datasets while pretraining the Symbolic Transformer.
D QUALITATIVE ANALYSIS
This section discusses qualitative analysis for the experimental results above.
Here we highlight some examples that the Symbolic Transformer baseline method perform better with respect the normalized edit distance than other baselines and vice versa. Taking 1.12.4 in Table as an example, E = - is simplified by sympy TO and converted to a skeleton equation as f(x) .x.2" where c's are constant tokens IT and x1 and are the first and second variables, respectively.
Table SS3 shows the predicted skeleton equations and normalized edit distance from the target skeleton. For the specific true symbolic expression, gplean, AFP, and AFP-FE could not finish the training to produce symbolic expressions for the corresponding tabular data within the session timeout, which part of our runtime constraints. While the skeleton equation produced by the Symbolic Transformer (ST) is not perfect solution, is structurally closer to the target skeleton than those produced by Al Feynman and DSR. seems that both AI Feynman and DSR attempted to fit the training data and minimize their regression errors, but they resulted in overcomplex symbolic expressions.
Table S14 shows different example (1.14.3 from Table where some of the other baseline methods performed better than the Symbolic Transformer. While DSR and Symbolic Transformer produced overcomplex simpler solutions, AFP, AFP-FE, and AI Feynman produced perfect skeleton with respect to normalized edit distance from the true equation. Overall, our Symbolic Transformer baseline seems to prefer simpler symbolic expressions, that may avoid overcomplex symbolic expressions result.
E INJECTING NOISE TO TARGET VARIABLES
Following SRBench (La Cava et al. 2021) we introduce Gaussian noise with a parameter of noise level 7 the target variables in SRSD datasets. We inject Gaussian noise each of the datasets separately, following Eq. S22
(
N
yruise = S (X,) +e, e e~
0,77
1
Σ
Stne (Xx)
(S2)
N
where 1 < N and N indicates the number of samples in the dataset.
//2 by substituting the constant values in Table Whee computing normalized edit distance, we treat all the constants (c's) in a skeleton equation as duplicate nodes. iee, As tree nodes, constant indices are not important since those will be estimated separately at the end, and in general the estimated constant values should affect the edit distance.
F(),r) 21 + f()) 9987742437.98888
29

---

Under review as a conference paper at ICLR 2023
Table S13: An example (1.12.4) that Symbolic Transformer performed better than other baselines.
Target Skeleton
61 11-20 01/0/
NED
AI Feynman
tan 122 C2)
1.00
DSR
21 (2] + C1 exp(((2 cos(22 + + cc)/a2))) exp((-22)
1.00
ST
Ci --22
0.167
Table S14: An example (1.14.3) that AFP, AFP-FE, and Al Feynman outperformed ST.
Target Skeleton
C1 21
Z2
NED
Al Feynman'
Ci 21 42
0.00
DSR
21 22 (c1 (c2 22 + C3 log (cos (I2))) (-z1 + x2 ca) /x2)
1.00
ST
21 :2
0.250
AFF and AFP-FE produced exactly the same skeleton with Al Feynman, which resulted in NED =
Table S15: Normalized edit distances of baselines for noise-injected SRSD (Easy) datasets with different noise levels.
Noise Level (7)
Method
gplearn
AFP
AFP-FE
Al Feynman
DSR
ST
0.876
0.703
0.712
0.646
0.551
0.435
10
0.928
0.799
0.814
0.797
0.820
0.435
10
0.940
0.824
0.880
0.870
0.793
0.435
10-1
0.948
0.823
0.960
0.882
0.841
0.461
Table S16: Solution rates of common baselines for FSRD and SRSD (Easy, Medium, Hard) datasets.
Dataset Method
gplearn
AFP
AFP-FE
AI Feynman
DSR
FSRD (Udrescu Tegmark
2020)
15.5%
20.48%
26.23%
52.65% 9.17%
19.71%
SRSD (Ours)
1.67%
5.83%
5.83%
15.0
Table S15 shows normalized edit distances of our baselines for noise-injected SRSD (Easy), reusing
the set noise levels in SRBench (La Cava et al. 2022 i.e., E {0, 10 10 10-2, 10001-1 }). Interestingly.
Symbolic Transformer (ST) baseline seems less sensitive noise injected to target variables than other baseline methods. Overall, the more the injected noise is, the more difficult would be for the baseline models (re-)discover the physics law in the data.
F SOLUTION RATE COMPARISON FSRD vs. SRSD
Table S16 compares the solution rates of the five common baselines for the FSRD and our SRSD datasets. We can confirm that the overall solution rates for our SRSD are significantly degraded compared those for the FSRD reported in SRBench (La Cava et al. 202I). The results indicate that our SRSD datasets are more challenging than the FSRD in terms of solution rate.
G USER STUDY R²2 SCORE VS. NORMALIZED EDIT DISTANCE
To investigate how aligned with human judges the existing SR and new SRSD evaluation metrics are, we recruited 23 people from industry and academia who either have doctoral degrees (scientists, professors, engineers) are doctoral students, and performed user study with approval from an ethics review board. The recruited people are diverse research fields such as computer science, mathematics, physics, chemistry, material science, aerospace engineering, engineering, medical nutrition, and computational biology. Given pair of true and estimate equations for an SRSD problem, the subjects asked assess estimated equation discretized 1-to-5 scale, where
30

---

Under review as a conference paper at ICLR 2023
Table S17: Pearson correlation coefficients (PCCs) between the human judges and SR/SRSD metrics.
Metries
PCC
P-value
R score
4.66 x 10-3
0.913
NED
-0.416 1.85 x 10-21
1 and indicate "1: Completely different from the true equation" and "5: Equivalent to the true equation" respectively. We chose SRSD problems among the 120 SRSD datasets such that obtain from the experimental results Section least two different equations estimated by different methods that are best terms of R² score and normalized edit distance. There were 24 resulting SRSD problems for the user study.
Table ST7 shows Pearson correlation coefficients (PCCs) between the human judges and SR/SRSD evaluation metrics, For normalized edit distance (NED), the Pearson correlation coefficient and p-value were -0.416 and 1.85 x 10 respectively, which show a much stronger and statistically more significant correlation between NED and human judges than one for R scores. In other words, results of the user study indicate that normalized edit distance more aligned with human judges than R score, and thus can be better estimate about how close the true equations the estimated equations are, a more human- unddrsstanndbrbleee way.
31


Reviews:
[{"review_id": "lHkZ-6R5AY", "reviewer": "Reviewer_2CZT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new benchmark set for symbolic regression. Starting from the Feynman symbolic regression dataset (FSRD, Udrescu et al), they propose a new dataset of 120 problems, together with rules for sampling variables and parameters. They introduce a new metric for comparing symbolic functions, based on the edit distance between the trees representing the simplified skeletons of the functions. Finally, they provide baseline comparisons between different popular models (and one introduced by the authors).", "summary_of_the_review": "The dataset proposed is interesting and welcome addition to current benchmarks and test sets. However, it is very small, and mostly amounts to a new curation of an existing dataset. The edit distance is an interesting idea, but would need significant work before it can be used as a measure of accuracy of symbolic regression. \n\nOverall, my feeling is that the contribution is very marginal, and not novel enough to feature in a venue like ICLR. ", "strength_and_weaknesses": "**Strengths**\n\nThe discussion of the limitations of current benchmarks is interesting, and the improvements proposed by the authors (e.g. keeping constants constant, introducing physics-informed ranges for the parameters and variables) makes a lot of sense. Baseline evaluations are provided.\n\n**Weaknesses**\n\nThe dataset remains very small, and most if it is a new curation/annotation of an existing benchmark (FRSD). This limits the novelty and impact of this paper. To me, the main problem of symbolic regression benchmarks is the small size of the datasets, and their lack of diversity. The paper does not address these questions.\n\nThe edit distance is an interesting idea, but needs a lot of improvements before it can be used as a reliable criterion. Two limitations spring to mind.\n\n* Some weighting of the operators is probably needed. Take f(x) = cos(ax+b), g(x)=sin(ax+b), h(x)=cos(ax), k(x)=exp(ax+b). The edit distances are edit(f, g)=1 edit(f, k)=1 and edit(f, h)=2, yet g is equivalent to f, h is a phase shift away from f (i.e. very close), but k has very different mathematical properties from the three others.\n* The magnitude of the constants should be taken into account. For small a, sin(ax) is essentially the same as ax, for large a, the functions are very different. Also, suppose you compare functions of the form f(x)+a.g(x), and u(x)+b.v(x). If a<<1 and b<<1, the edit distance between f and u is probably a much better metric than the edit distance between f+ag and u+bv. This problem will appear every time a function is represented as a sum of terms of decreasing magnitude (a very common situation in science).\n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BA8Fp-rQ6Z", "reviewer": "Reviewer_V7wv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents the design of\ndatasets and benchmark of symbolic regression for scientific\ndiscovery. Authors point our some limitations for existing datasets and\ndesign their own dataset categorizing subsets of small, medium and\nlarge complexity and develop a new metric based on tree distance\nbetween the true and the predicted equations.", "summary_of_the_review": "The main contribution of this work lies in the design of\ndatasets and benchmark of symbolic regression for scientific\ndiscovery. Authors point our some limitations fo existing datasets and\ndesign their owm dataset categorizing subsets of small, medium and\nlarge complexity and develop a new metric based on tree distance\nbetween the true and the predicted equations.\n\nWhere is I in the examples of Table 1? (It is mentioned in the caption\nof this table)\n\n\"we preprocess equations by 1) substituting constant values e.g., π and\nPlanck constant to the expression...\" --> isn't the idea to learn these\nexpressions from data? Do you mean that you *postprocess* the learned\nformulas?\n\n\"coefficient values themselves (e.g., value of C1 in Fig. 2) should\nnot be important\" --> I don't get the idea here. I'd think that\nconstants can differentiate one expression from another when modeling\nreal systems (e.g., physics, math etc).\n\n\"such true equations will not be available in practice\" --> but don't\nyou generate data from these equations in order to create the\ndatasets? How do you expect to validate learning models if you don't\nhave the ground truth? Besides, don't Feynman lectures present the\nequations?\n\nDo you have any explanation of why ST results in 0% R2 for all\nproblems? It seems strange.\n\nOther minor comments:\nboth the metrics --> both metrics\nTable 4 show --> Table 4 shows\n", "strength_and_weaknesses": "The main contribution of this work lies in the design of\ndatasets and benchmark of symbolic regression for scientific\ndiscovery. Authors point our some limitations fo existing datasets and\ndesign their owm dataset categorizing subsets of small, medium and\nlarge complexity and develop a new metric based on tree distance\nbetween the true and the predicted equations.\n\nWhere is I in the examples of Table 1? (It is mentioned in the caption\nof this table)\n\n\"we preprocess equations by 1) substituting constant values e.g., π and\nPlanck constant to the expression...\" --> isn't the idea to learn these\nexpressions from data? Do you mean that you *postprocess* the learned\nformulas?\n\n\"coefficient values themselves (e.g., value of C1 in Fig. 2) should\nnot be important\" --> I don't get the idea here. I'd think that\nconstants can differentiate one expression from another when modeling\nreal systems (e.g., physics, math etc).\n\n\"such true equations will not be available in practice\" --> but don't\nyou generate data from these equations in order to create the\ndatasets? How do you expect to validate learning models if you don't\nhave the ground truth? Besides, don't Feynman lectures present the\nequations?\n\nDo you have any explanation of why ST results in 0% R2 for all\nproblems? It seems strange.\n\nOther minor comments:\nboth the metrics --> both metrics\nTable 4 show --> Table 4 shows\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8CQY6yPcDT", "reviewer": "Reviewer_CRUL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose a new symbolic regression (SR) dataset for scientific discovery (SRSD) that is a modification of the Feynman symbolic regression database (FSRD) that fixes the following issues present in FSRD: 1) no physical meaning for a number of the datasets, 2) an oversimplified sampling process, 3) distinct equations that are marked as duplicates, and 4) inappropriate or incorrect formulas. The proposed SRSD datasets treats constants (e.g., lightspeed) as constants, samples values for variables with appropriate ranges using experimental observations, and partitions the 120 datasets into easy, medium, and hard sets based on formula operation length and variable domain range. The authors also propose using normalized edit distance to assess the similarity of the structures between the predicted equation models and their respective ground-truth models.\n\nThe authors evaluate 5 established and well-known SR methods and one transformer-based SR model on the proposed SRSD datasets and measure performance using the proposed normalized edit distance, accuracy (which captures instances where the predicted and true model differ by a constant or scalar), and solution rate. Overall, the authors find the transformer-based approach outperforms all other methods on normalized edit distance, but performs worse than all other methods in terms of accuracy and solution rate; this suggests the transformer-based approach provides more structurally similar equation models than existing methods.", "summary_of_the_review": "Although the paper proposes a potentially valuable collection of SRSD datasets and a new evaluation metric measuring normalized edit distance, the dataset is not well-documented and it is unclear whether the new evaluation metric provides significantly meaningful insights in terms of scientific discovery compared to existing evaluation metrics.", "strength_and_weaknesses": "Strengths\n---\nThe proposed SRSD datasets address issues present in the FSRD database in order to provide datasets that better align with the goals of symbolic regression for scientific discovery.\n\nThe proposed SRSD contains 120 datasets, partitioned into three different sets of varying difficulty level.\n\nThe proposed metric of normalized edit distance allows for a more fine-grained analysis into the structural differences between the predicted and true equation models.\n\nThe paper is relatively well-written and easy to read and follow.\n\nWeaknesses\n---\nIt is not clear to me how much insight is gained using the normalized edit distance (NED) over existing metrics measuring the success of symbolic regression models. The results suggest that the transformer-based model achieves slightly better NED over existing methods, but it is difficult to judge if this improvement is meaningful from a scientific discovery point of view. Even if two equation models are mostly structurally similar (as measured by NED), they may differ by one or two elements that could perceivably result in a very different equation model not representative of the actual ground truth equation.\n\nThe experimental results are not very thorough, and only provide results for a small number of metrics for a small number of baselines. Due to the difficulty evaluating the merit of equation-based scientific discovery, perhaps a metric based on manual inspection with domain experts could give some insight into whether or not the proposed metric (and existing metrics) are well-aligned with how humans judge how close a generated equation is to representing an actual scientific equation.\n\nWhen proposing a new dataset, the authors should consider documenting their dataset thoroughly using, for instance, \"Datasheets for Datasets\" (Gebru et al. 2021) or a dataset nutrition label (Holland et al. 2018).\n\nMinor Weaknesses\n---\nConsider using \"\\citet\" when using citations as a noun.\n\nThe solution rate metric is not clearly described.\n\nWhy is accuracy and NED not shown in the comparison between the FSRD and SRSD dataset (Table S16)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject