Transformer-based model for symbolic regression via joint supervised learning
Authors: Wenqiang Li
Weijun Li
Linjun Sun
Min Wu
Lina Yu
Jingyi Liu
Yanjie Li
Songsong Tian
Venue: ICLR 2023 poster (2023)

Abstract: Symbolic regression (SR) is an important technique for discovering hidden mathematical expressions from observed data. Transformer-based approaches have been widely used for machine translation due to their high performance, and are recently highly expected to be used for SR. They input the data points, then output the expression skeleton, and finally optimize the coefficients. However, recent transformer-based methods for SR focus more attention on large scale training data and ignore the ill-posed problem: the lack of sufficient supervision, i.e., expressions that may be completely different have the same supervision because of their same skeleton, which makes it challenging to deal with data that may be from the same expression skeleton but with different coefficients. Therefore, we present a transformer-based model for SR with the ability to alleviate this problem. Specifically, we leverage a feature extractor based on pure residual MLP networks to obtain more information about data points. Furthermore, the core idea is that we propose a joint learning mechanism combining supervised contrastive learning, which makes features of data points from expressions with the same skeleton more similar so as to effectively alleviates the ill-posed problem. The benchmark results show that the proposed method is up to 25% higher with respect to the recovery rate of skeletons than typical transformer-based methods. Moreover, our method outperforms state-of-the-art SR methods based on reinforcement learning and genetic programming in terms of the coefficient of determination ($R^2$).

Published as a conference paper at ICLR 2023
TRANSFORMER-BASED MODEL FOR SYMBOLIC RE- GRESSION VIA JOINT SUPERVISED LEARNING
Wenqiang Li
Weijun Li Linjun Sun 1,3,4 Min Wu Lina Yul.44
Jingyi Liu! Yanjie Li Songsong Tian
'Inttitute Semiconductors, Chinese Academy of Sciences, Beijing, China School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing, China
School of Integrated Circuits, University of Chinese Academy of Sciences, Beijing, China Beijing Key Laboratory of Semiconductor Neural Network Intelligent Sensing and Computing Technology, Beijing, China
ABSTRACT
Symbolic regression (SR) is an important technique for discovering hidden math- ematical expressions from observed data. Transformer-Be -ased approaches have been widely used for machine translation due their high performance, and are recently highly expected to be used for SR. They input the data points, then out- put the expression skeleton, and finally optimize the coefficients. However, recent transformer-basee methods for SR focus more attention on large scale training data and ignore the ill- ill-posed problem: the lack sufficient supervision, i.c.. ex- pressions that may be completely different have the same supervision because of their same skeleton, which makes it challenging from the same expression skeleton but with different coefficients. Therefore, we present transformer-based model for SR with the ability to alleviate this problem. Specifically, we leverage feature extractor based on pure residual MLP networks to obtain more information about data points. Furthermore, the core idea is that propose joint learning mechanism combining supervised contrastive learn- ing, which makes features of data points from expressions with the same skeleton more similar results show that the proposed method is up to 25% higher with respect to the recovery rate of skeletons than typical transformer-based methods. Moreover, our method outperforms state-of-the-art SR methods based on reinforcement learning and genetic programming in terms of the coefficient of determination (R²).
deal with data that may be
effectively alleviates the ill-posed problem. The benchmark
INTRODUCTION
Exploring mathematical expressions that can be fitted to real-world observed data is the core of expressing scientific discoveries. The correct expression would not only provide us with useful scientific insights simply by inspection but would also allow us to forecast how the process will change in the future. The task of finding such an interpretable mathematical expression from ob- served data is called symbolic regression. More specifically, given a dataset (X,y), y), where each feature X; E R" and target y/ E R, the goal of symbolic regression is to identify a function of (i.e., y f(X): R" RR that best fits the dataset.
Symbolic regression with the length of the expression, and the presence of numeric constants further exacerbates its difficulty (Lu et al. 2016). Considering this issue, genetic programming (GP) as the most com- mon approach is leveraged tackle the symbolic regression problems (Forrest 1993 Koza 1994 Schmidt & Lipson| 2009; Staelens et al. 2013 Arnllo et al. 2015 Bladek & Krawiec 2019).
NP-hard because the search space of an expression grows exponentially
GP-based methods iteratively "evolves" each generation of mathematical expressions through se- lection, crossover, and mutation. Although this approach can be effective, the expression it yields
*Corresponding author.

---

Published as a conference paper at ICLR 2023
is complex, and it is also known to be computationally expensive and to exhibit high sensitivity to hyperparameters. A more recent line of research has made use of the neural network to tackle the aforementioned shortcomings. Martius & Lampert (0016) propose simple fully-connected neural network called "EQL.", where elementary functions (sin, cos, +, +.... are used as activation func- tions. The limitation of EQL is the existence of vanishing gradient and exploding gradient, and the depth of the network limits the complexity of the predicted equation. More recently, deep symbolic optimization (DSO) (Petersen al., 2021) trains the RNN using the reinforcement learning algo- rithm based on a risk-seeking policy gradient to generate expressions. They take the output from RNN as an initial population for a genetic algorithm to find the target expression. Albeit the above two approaches show promising results, they still handle symbolic regression as an instance-based problem, training a model from scratch on each new input dataset for a regression task.
Inspired by the successes of large scale pre-training, recent efforts in symbolic regression have fo- cused on using the transformer-based model and training with large amount of data (Valipour et al.) 2021 Biggio et al. 2021). They all approach the symbolic regression problem as machine transla- tion problem, mapping the input data to latent representations via encoders, and then outputting the skeleton of expressions without constants by decoders. These transformer- -assed methods (Valipour iet al. 2021; Biggio et al., (022) for symbolic regression exists two main drawbacks: (i) A natural question is what architecture of the encoder is optimally suited for symbolic regression. It is clear that the decoder's ability to sample expressions efficiently is severely constrained by the encoder's ability to extract the features of the imput data. The idea is that the encoder should not just en- code the points, but also represent the expression on a high level such that the decoder only prints the representation as a sequence of symbols. (ii) They use the single character of the expression's string Vaalipour et al. [021] and the pre-order traversal of the expression tree (Biggio et al. 2021 as supervision information, respectively, which is an ill pol-p problem that does not provide sufficient supervision: different instances of the same skeleton can have very different shapes, and instances of very different skeletons can be very close. To alleviate these issues, we proposed a transformer- based method for symbolic regression using a new feature extractor and a joint supervised learning mechanism.
In summary, we introduce the main contributions in this study follows:
We leverage pure residual MLP feature extractor for extracting the local and global fea- tures of observed data targeting symbolic regression tasks, which aids the expression gen- erator in producing more correct expression skeletons.
We propose a joint learning mechanism combining supervised contrastive learning that combines the supervision of the whole expression skeleton with the supervision the pre- order traversal of its expression tree, which alleviates the ill-posed problem effectively.
Empirically, the proposed method is up to 25% better than recent transformer- meth- ods with respect to the recovery rate of expression skeleton. Moreover, our method outper- forms several strong baseline methods in terms of R².
RELATED WORK
Genetic programming (GP) for symbolic regression. Traditionally, the approaches symbolic regression are based on genetic algorithms (Forrest 1993). Later, the symbolic regression task is seen as an optimization problem for the search space (Koza 1994). By far the most popular commer- cial software Eureqa (Dubčáková 2011) is the most successful application based on GP methods. A limitation of the genetic algorithms- based methods is that they need to train for each equation from scratch, which is slow, computationally expensive and highly randomized. The models tend to generate more complex equations and they are sensitive to the choice of hyperparameters (Petersen) al. 2021).
Neural network for symbolic regression. Symbolic regression based on neural network ap- proaches can be broadly classified into three categories. First, the methods based equation learner (EQL) (Martius& & Lampert| 2016:; Sahoo et al.] 2018 Werner et al. 202T) are trained by replacing the activation function of the neural network with arithmetic operators, which inherits the ability of neural networks to deal with high- high-dimensional data and scales well with the number of onputt pairs (Biggio et al.) 202T). Nevertheless, the existence of exponential and logarithmic activation
2

---

Published as a conference paper at ICLR 2023
contrastive + (1 c)ss
Gr + Cayet
Residual Point Block
Softmax
Staletton
XX,
Add Norm
Residual Point Block
Feed Forward
X
Max Pooling
Residual Point Block
Norm
X.N.
Maiked Mult--ead Attention
Block
Geometric Affine Module
Points Foaurre Yostor
Tokon Emrbediing rg
Pos Uonal Emibedling
[(.y)) = 2x² 3y²
order expression
Feature Extractor
Expression Generator
Figure 1: Schematic diagram of training. The data inputs and expression skeletons' labels are passed through the feature extractor. Then, given the feature vectors, token embedding, and positional embedding, the expression generator produces expression skeletons in parallel. Finally, the model jointly computes the supervised contrastive learning loss and cross-entropy loss.
functions leads to gradient instability. Also, the complexity of predicted expression depends on the depth of the EQL network.
Reinforcement learning for symbolic regression. The second approach is the autoregressive model based on reinforcement learning. Petersen et al. (2021) uses reinforcement learning based on risk-seeking policy gradient to train RNN to generate probability distribution over the space of mathematical expressions. For such symbolic regression tasks, they proposed a new ob- jective function based on risk-seeking policy gradients, that focuses on learning only on maximizing best-case performance rather than the average performance of a policy, Genetic programming and neural- neural-guided search are mechanistically dissimilar, yet both have proven be effective solutions to symbolic regression. Mundhenk et al. (2021) proposed more novel approach, combining the two approaches to leverage each their strengths. They take the output from the RNN as an initial population for genetic algorithm. The method represents a significant step forward in the appli- cation of deep learning to symbolic regression (Biggio et al., 2021). The promising results make it the currently recognized state-of-the-an approach to symbolic regression tasks. Nevertheless, the limitations of this method are obvious, namely, the network has to be retrained from scratch for each new equation and the RNN is never directly conditioned the data it is trained to model (Biggio et al. (02I).
Large scale transformer-baset models for symbolic regression. The third approach is to train a large scale transformer-basee model by using a large amount of data. More recently, Symbol- icGPT (Valipour al., (021) trained GPT (Radford al.) 2019) model construct mapping of pairs of points and symbolic output. They first input the data points into T-net (Qi et al., (201)) get a potential representation of the data points and then input it to the GPT (Radford et al. 2019) for generating expression strings. They generate the expression at the character level and fi- nally concatenate into an expression. In general, they explore an alternative approach symbolic regression by considering it as a task in language modeling. Symbolic mathematics behaves as language in its own right, with well-formed mathematical expressions treated as valid "sentences" in this language (Valipour et al., (02T). Furthermore, NeSymReS (Biggio et al. (021) proposed similar method, where they use the encoder from the Set transformer (Lee et al., 2019) and the de- coder from the original transformer architecture (Vaswani et al. 2017). Their greatest contribution to show that their approach is able to improve performance as the size of the dataset increases.
3 METHODS
The proposed joint learning mechanism is shown in Figure 1 invariant feature extractor based on residual MLP networks to obtain feature vectors of data points. Then, given the feature vectors, the expression generator autoregressively generates individual math-
First, leverage permutation-
3

---

Published as a conference paper at ICLR 2023
ematical symbols until obtain the entire skeleton of the expression. In the forward propagation, we compute the contrastive loss with respect to the expression skeleton categories and the cross- entropy loss with respect to the mathematical symbol categories separately. The parameters of the network are jointly updated in backpropagation. Training can efficiently process each sequence in single forward pass of the network thanks to the masked attention and teacher forcing (Vaswani iet al. 2017). During inference, multiple predictions are sampled from the model using the beam search strategy and we select the prediction with the lowest error.
3.1 EXTRACTING EFFECTIVE FEATURE OF DATA POINTS
As mentioned in section 1 the feature extracted from data points affects generating expressions through the decoder. Valipour et al. (202I) obtain the latent representation of data using the T- net <Qi et al.) 2017). Albeit efficient, local feature loss caused by non-locality and non-hierarchy degrades the representational quality of details for point cloud (Ma et al. 2022). As a similar work, N (Biggio et al.) 202T) use the Set Transformer (Lee et al., 2019) based on self-attention to extract data points features. However, it focuses too much on local feature extraction and lacks global information, which is not suitable for expression data points. We explore and visualize the similarity of the features extracted from data points by these methods and compare the expression skeletons generated by the decoder, which confirms the above problems. The result is shown in section 4 The performance of the feature extractor has to be considered, therefore, we opt for the 2022) based on pure residual MLP as our feature extractor in order to obtain the local and overall information of the data points.
framework of PointMLP [Ma et al.
Given a set of data points D = {<.,,) y:) E R=x((+11) where n indicates the number of points and d denotes the dimension of the variable. PointMIPP learns hierarchical features of data points by stacking multiple learning stages. In each stage, NN, points are re-sampled by the farthest point sam- pling (FPS) algorithm, where indexes the stage and K neighbors are employed for each sampled point Ma et al. 2022). Conceptually, the kernel operation of PointMLP can be formulated as:
O = POS (MaxPool (PRE (fa,)) (j 1, .K)
where Júg is the j-th neighbor point feature of i-th sampled point. POS (-) and PRE (.) are residual point MLP blocks: the shared PRE (-) is designed to learn shared weights from a local region while the POS (-) is leveraged to extract deep aggregated features. POS (-) and PRE consist of several residual MLP blocks: MLP (x) + æ. We use the max pooling layer aggregate global features.
After MLP blocks, we add dropout layer (Srivastava et al. 2014).
Following (M et al. 2022). we also leverage a lightweight geometric affine module tackle the problem that is less robust and caused by the sparse and irregular geometric structures in local regions. Let [fe.i] & E RRd be the grouped local neighbors of Si E IRd containing & points, and each neighbor point fú,j is a d-dimensional vector. We transform the local neighbor points by the following formulation:
(11)) S +B, tcc
k (fis fa)²
(fi)) a
kxnx d ΣΣ
where a E I2d and B E Rdd are learnable parameters, @ indicates Hadamard production, and c = 1e-5 5 is small number for numerical stability. Note that is a scalar that describes the feature deviation across all local groups and channels. By doing so, we transform the local points to a normal distribution while maintaining original geometric properties.
3.2 TRAINING WITH JOINT SUPERVISION INFORMATION
Recent transformer-bast based face an ill-posed problem because use insufficient supervision informa- tion: they use the single character of the expression's string (Valipour et al. 2021) and the pre-order traversal of the expression (Biggio et al. (021) supervision information, respectively, and train the model by minimizing the cross-entropy loss. In this work, we propose a joint objective function that combines enoospy (CE) loss and supervised contrastive learning (CL) loss. On the basis of using symbol labels as supervision, we treat the skeleton of expressions as category labels to enrich the supervisory information. The CL loss with respect the expression skeleton categories and the
4

---

Published as a conference paper at ICLR 2023
CE loss with respect to the mathematical symbol categories separately calculated in the forward propagation. In backpropagation, the network's ' parameters are concurrently updated. As the auxil- jary loss, the CL loss is meant to capture the similarities of feature vectors between expressions with
the same skeletons and contrast them with others. The promising results are described in section 4 The overall loss is a weighted average of CE loss and proposed CL loss, as given in equation (I). The canonical definition of the CE loss that we use is given in equation (2). The novel CL loss is given in equation (3). The overall loss is then given in the following:
L = (1 C)ECE + ACCL
(1)
1 N c
LCE
Vi.e In Dt.c
(2)
N 1=
exp(si5/T)
LOL E
Σ 11zylee= In No. +E
(3)
Lkml Ligh Eisht exp (sim/T)
Here, À is scalar weighting hyperparameter that we tune for the training stage; N represents the mini-batch size; C denotes the size of the pre-specified tokens library; fi.e denotes the symbol label and Úi.c denotes the expression generator output for the probability of the ith token belonging the class c; Ni, is the total number of examples in the batch that have the same skeleton label as &; e is very small scalar preventing devision by zero; Lizj. 1,,-,, and 14,t are similar indicator functions; 81,5 = v2 / || ||æy denotes the cosine similarity between the sample i and the sample j. where o; and v, represent the high-level feature vectors of the sample and the sample j respectively from the feature extractor; >> 0 is an adjustable scalar temperature parameter that controls the separation of classes.
3.3 GENERATING EXPRESSIONS WITH A TRANSFORMER- MODEL.
We leverage framework of the GPT language model (Radford et al. 2019) as the expression gener- ator, which an autoregressive language model based on the decoder of transformer (Vaswani al.) 2017). During inference, we generate an expression one token at a time along the pre-order traver- sal. For example, the expression (x1 222) = sin (I1) + log (222 is encoded as [+,sin, 21, log, x2). We denote the 7t token of the traversal as т2 and each token has value within given library L of possible tokens, e.g., {+, f,,,,,,, cos, log, Fi, 22). Specifically, the 1th output of the gen- erator with parameters 9 and feature vector extracted from data D pass through a softmax layer to produce vector 46, which defines the probability distribution for selecting the 7hh token The likelihood of the entire generated expression simply the product of the likelihoods of its tokens:
p(T v) I p(Ti 8, v) = -I 2b(0) ((T)) Note that the output sequence from the gen- erator does not contain any numerical constants. Instead, we use special placeholder (C) denoting the presence of constant that will be optimized at a later stage.
3.4 LEARNING CONSTANTS USING BFGS BY RESTARTING MULTIPLE TIMES
At inference time, Mundhenk al. 2021 ; Valipour al. 202T) : Biggio et al. (02T) all BFGS
optimization algorithm (Fletcher 1984) on the mean squared error to fit the constants. BFGS is quasi-newton method for volging unconstrained nonlinear convex optimization problems. However, for the symbolic regression task, the loss function minimized by BFGS may be highly non-convex, which is likely to be falling into several local minima: even when the skeleton is perfectly predicted, the correct constants are not guaranteed to be found. To ameliorate this issue, we use simple and hardly any space cost method: restart the BFGS algorithm by initializing the different starting points multiple times, and the global optimal point is achieved as much possible. The result is shown in section 4
EXPERIMENTS AND RESULTS
4.1 GENERATING DATASETS
We generate mathematical expressions following the framework as described by (Lample & Char-
ton 2019).
The framework starts by generating random unary-binary tree and filling the nodes
5

---

Published as a conference paper at ICLR 2023
with operators and the leaves with independent variables constants. The unnormalized probabili- ties of each operation and operator are given in Appendix A We generate the training set containing approximately 100, 000 unique expression skeletons. For each expression, stant values for 10, 20, 30, 40, and 50 times. Once an expression tree has been generated, we can represent the expression using a pre-order traversal of the tree. We opt for scalar functions of up to two independent variables (i.e., d = 2 and = f( and three numerical constants each. Specifically, each generated expression's is constant values are replaced with the constant placeholder (C). Then, the additive and multiplicative constant values are sampled uniformly from the interval [- fill in the placeholder. After this, the entire equation will be simplified using the rules built in the symbolic manipulation library SymPy (Meurer et al. 2017). Finally, we sample uniformly at random 100 input points X = = from the interval [-10, 10) and evaluate the expressions on the X to obtain the corresponding outputs Y. X will be re-sampled if produce non-finite values (NaN or to) and we discard the expression that cannot be sampled completely in 30 seconds.
re-sample its con-
Creating the SSDNC benchmark. We generate more challenging test set to discover the perfor- mance of transformer-ban based methods, which includes approximately 100 unique expression skele- tons and 10 re- re-sampled numerical constants for each skeleton. We sample random support of 100 points from the uniform distribution (d( (-10, 10), for each independent variable. We call SSDNC, for the same skeletons with different numerical coefficients.
4.2 TRAINING AND INFERENCE
We train the feature extractor and expression generator jointly to minimize the objective loss com- bined with cross-entropy loss and supervised contrastive loss. More specifically, we train the model using the Adam optimizer (Kingma& & Ba 2014) on 4 NVIDIA V100 GPUs. More detailed hyperpa- rameters are reported in Appendix E which were found empirically and not fine-tuned for maximum performance. Note that we use the same dataset for training to facilitate quantitative benchmarking when evaluating the feature extraction and other capabilities with SymbolieGPT (Valipour et al.) 2021) and NeSymReS (Biggio al. al.]|200. [202I). At inference, most of the expressions of benchmark test sets are not seen during training, and we resampled all data points to avoid the possible overfitting problem.
4.3 METRICS
We have selected the coefficient of determination (R²) to assess the quality of our method, The R² Gllntt & Slinker 2001 is defined as follows:
Σ 1= (3) 01)2
R² 1) =
E (3)2 7)
where yi and vi are the ground-truth and predicted values for point i, respectively. 7 is the average of yi over all the points. i the number of test points. The advantage of using R² its nice interpretation. if R > 0, then means the prediction is better than predicting just the average value, and if 8² = 1, then we get a perfect prediction. However, due to the presence of the max operator, R² is sensitive to outliers, and hence to the number of points considered test time (more points entail a higher risk of an outlier). To circumvent this, we discard the 5% worst predictions for all methods used, following (Biggio et al. 2021).
4.4 BASELINES
We compare the performance of our method with four strong symbolic regression baselines:
SymbolicGPT (Valipour et al., 2021) recent novel transformer baasdd language model for symbolic regression. We use the open-source implementation provided by the authors
Neuaral Symbolic Regression that Scales (NeSymReS) (Biggio et al., 2021) Recently proposed transformer-based symbolic regression model on the large training data. We use the open-source implementation provided by the authors.
hateess//attaal.cc////aaaa//a//p Naaao////oeraaaaaaaallll Symbol w icRegress on hhathhataa aattaall les

---

Published as a conference paper at ICLR 2023
(a) SymbolicGPT
(b) NcSymReS
(c) Ours w/o CL
(d) Ours w/ CL
Figure 2: For the expression skeleton eı sin (æ11 +2 cos (222) + four heat maps of cosine similar- ity between the fifty different feature vectors from different methods, where the redder color means the cosine similarity closer to 0, and the greener color means the cosine similarity closer to
Deep Symbolic Optimization (DSO). A symbolic regression method based RNN and reinforcement learning search strategy (Mundhenk et al. 202I). We use the open-source implementation provided by the authors
Genetic Programming. Standard GP-based symbolic regression (Koza 1994) based on the open-source Python library gplearn
All details for baselines reported in Appendix D
4.5 FEATURE EXTRACTION PERFORMANCE
For transformer-based methods in symbolic regression, we empirically demonstrate that the per- formance of the feature extractor plays a critical role in the overall training and evaluation stages. We evaluate the feature extraction performance of recent icGPT /Valipour et al., 2021) and NeSymReS (Biggio al.) 2021), and our method with/without CL on SSDNC benchmark. After training, we input the data points corresponding to the specific expression skeleton with different constants into the feature extractor, then we compute the cosine similarity between the different feature vectors. As shown in Figure 2 SymbolicGPT (Valipour) jet al. 202I) and NeSymReS (Biggio et al. (021) produce dissimilar feature vectors for the data the same skeleton. The reason is that the feature
meatfoommr-bbaee methods, i.e., Symbol-
points of different expressions belonging tractor they used focuses more on local features and loses key information, which is not appropriate for the data points of symbolic regression. By manual inspection, find that this problem versely affect the expression generator to produce an expression skeleton. The results in section 4.6 illustrate the high correlation between the feature extractor and expression generating. Benefiting from using the architecture of pure residual MLP, our feature extractor is able to obtain more similar feature vectors when facing the same skeleton, even without CL. After training jointly, we make the feature vectors of data points from the same skeleton more similar, which effectively alleviates the ill-posed problem and improves the performance of generating expressions with the same skeletons.
ad-
4.6 EVALUATION ON BENCHMARKS
Recovery rate of skeletons. In inference, since transformer-based methods first generate expression skeletons based on features of data points, we first com- pare the performance of these methods in recovering ex- pression skeletons.
Table 1: Revovery rate of expression
skeletons the SSDNC benchmark.
Method
Recovery rate (%)
evaluate the recovery rate of ex- pression skeletons on the SSDNC benchmark. As shown in T our method with the more effective feature tractor and the joint leaning mechanism can better guide the generator in generating expressions, thus outperform- ing other transformer-based methods, Generating the right expression skeleton is crucial to the final
SymbolicGPT NeSymReS
50.3 = 1.7
63.4 = 1.1
Ours w/o CL
69.7 0.9
Ours w/ CL
75.2 + 1.3
http:://pleammretttheeeess

---

Published as a conference paper at ICLR 2023
Table 2: Results comparing our method with CL with state-of-the-art methods several bench- marks. Our method, SymbolicGPT. and NeSymReS all use the beam search strategy with the beam size equaling 128. We report the average value of R for each benchmark.
Ours
SymbolicGPT
NeSymReS
DSO
GP
Benchmark R²
R² f
R² +
R 1
R² 1
Nguyen
0.99999
0.64394
0.97538
0.99489
0,89019
Constant
0.99998
0.69433
0.84935
0,99927
0,90842
Keijzer
0.98320
0.59457
0.97500
0.96928
0,90082
0.99999
0.71093
0.99993
0.97298
0.83198
AI- Al-Feynman
0.99999
0.64682
0.99999
0.99999
0,92242
SSDNC 0.94782
0.74585
0.85792
0.93198
0.88913
Overall avg.
0.98850
0.67274
0.94292
0.97806
0.89049
10
05-
86-
Syce -PP
5.00
8100
Staben webealeepoeeee mellih
Figure 3: Training on differ- ent datasets that contain various numbers of expressions with different constants. Inference on SSDNC benchmark.
Figure 4: R gaussian noisy data. Error bar represent stan- dard error. Inference on SS- DNC benchmark.
Figure 5: R for different restart times of BFGS in the constant optimization stage. In- ference on SSDNC benchmark.
result because optimizing constants is relatively simple. This is reflected in the comparison for R² in the following results.
Statisties of fitting accuracy We evaluate our method and current state-of-the-art approaches on the widely used public benchmarks, i.e., the Nguyen benchmark (Uy et al. (01T), Constant, Kei- jzer (Keijzer) (003), R rationals (Krawiec & Pawlak (013), Al-Feynman database (Udrescu & [egmark|[022 and our SSDNC test set. Nguyen was the main benchmark used in Petersen et al. (2021). There are terms that appear in three ground truth equations that are not included in the set of equations that our model can fit, specifically z6 and x", but we can find approximate expressions. Al-Feynman database is extracted from the popular Feyuman Lectures
Physics series and con- tains expressions with up to nine variables. In our study, we consider all the expressions with up variables. The complete benchmark functions are given in Appendix 4 From the results in Table our method outperforms all bascline methods in terms of average R on six benchmarks.
Performance under difffrent training sets sizes. Following subsection 4.1 generate the differ- ent size of data set that contains the various number of expressions with different constants and the same skeletons. Since DSO Mundhenk al. 202T) and GP-based methods are trained from scratch for specific problems, they are not included in this comparison. We train our model, SymbolicGPT and NeSymReS on our training sets separately. As shown in Figure 3 our method with/without CL all outperforms these two baseline methods in terms of R² on the SSDNC benchmark. Symbol- icGPT and NeSymReS can also improve performance simply by increasing the size of the dataset, which shows that the data-driven approach for SR can continuously improve performance the size of the dataset increases.
Performance under noisy data. We evaluated the robustness of our and baseline methods to noisy data by adding independent Gaussian noise to the dependent variable, with mean zero and standard deviation proportional to the root-mean-squan of the dependent variable in the training data (Mund- henk et al. 2021). In Figure we varied the proportionality constant from 0 (noiseless) to 10 following Mundhenn al. 2021). and evaluated each algorithm across the SSDNC benchmark.
8

---

Published as a conference paper at ICLR 2023
46--
360
16-
200
200
0-
100 Pro. -10
Pool
-20
-10
(3)
(b)
Figure 6: Examples of model predictions using beam search with beam size equaling to 128. The shaded area represents the sampling range. For all functions, r and were sampled from [-10, 10). "GT" denotes ground-truth and 'Pred', the model prediction. Specifically,
(a): GT: +x* f Pred: x (xr sin (t));
(a +a")
(b): GT: x cos (tan (r)), Pred: x cos (tan (I));
(c): GT: 0.2x. 0.5y
Pred: 0.1853r 0.4974y² 0.8608y.
Our method with CL is competitive compared with DSO based on reinforcement learning, not over- fitting the noise when adding small amount of noise.
Optimizing constants via restarting BEGS multiple times. We try to reach the global optimum rather than the local optimum by restarting BFGS several times, each time using different initializa- tion. We set the restart times 5, 10, 15 and 20. 0 means that the optimization only once. Figure shows that R improves as the number of BFGS restarts increases. We can conclude that multiple initializations can effectively jump out local optimum in the constant optimization process so as achieve better fitting accuracy.
Finding mathematically equivalent expressions. By manual observation, we find that our method generate more expressions with the same symbol the target expression, which benefits from the high skeleton recovery rate. Interestingly. we also find that the model sometimes predicts a more simple expression compared the ground truth with fairly high fitting accuracy. For example, for the Keijzer-4 expression x3 exp(-z) cos(x) x)- sin (x) (sin x² cos(x) 1). our method (9.7933333399477) which is simpler in the terms of skeleton and achieving high R² through the constant optimization. Additionally, the model can learn some transformation relations of trigonometric functions, e.g., for Constant-7 expression 2 sinn11334 cxos our method predicts 2 cos (1.3z) 3) cos(x22) which implicates the transformation relation, i.e., sin (r) = cos (x ))
predicts 11841993499900111r...ooooo (xpp(=)) 1.8822699311523772
Out-of domain performance. To test the out-of domain performance of our method, we first run the inference on the points sampled from training range and then evaluate these predicted functions on points outside the sampling range. In Figure we visualized some of the model predictions about unary and binary expressions. The experimental results show that our model can have good gen- eralization performance outside the sampling domain when predicting complex expressions, without overfitting the sampled data.
5 CONCLUSION
We propose a transformer-based method for symbolic regression using a new feature extractor and a joint supervised learning mechanism. Specifically, we leverage a pure residual MLP feature ex- tractor to obtain more valuable features on input data points. In order to fundamentally alleviate the ill-posed problem, we propose a joint supervised learning mechanism combining supervised con- trastive learning, which strengthens the similarity of feature vectors from the same skeleton. The expression skeleton recovery rate of the proposed method is up to 25% higher than that of recent coinsformerbbsee methods. Evaluated forms current state-of oftee--f-hhe--ee the-art methods based on reinforcement learning and genetic programming in terms of R². It is worth noting that method is competitive with the reinforcement learning- based method terms of robustness. Finally, by evaluating the performance of our method outside the sampling range, showed that has good extrapolation capabilities.
six benchmarks, the results show that our method outper-

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by the Key-Area Research and Development Program of Guangdong Province (No. NN.201990101070011, and the Key Research Program of the Chinese Academy of Sci- ences (No.XDPB22).
REFERENCES
Ignacio Arnaldo, Una-May O'Reilly, and Kalyan Veeramachaneni. Building predictive models feature synthesis. In Proceedings of the 2015 annual conference genetic and evolutionary computation, pp. 983-990, 2015.
Rohit Batra, Le Song, and Rampi Ramprasad. Emerging materials intelligence ecosystems propelled by machine learning. Nature Reviews Materials, 6(8):655 678. 2021.
Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascan- dolo. Neural symbolic regression that scales. In International Conference Machine Learning, pp. 936-945. PMLR, 2021.
Iwo Btadek and Krzysztof Krawiec. Solving symbolic regression problems with formal constraints. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 977-984, 2019.
Renáta Dubčáková. Eureqa: software review, 2011.
R. Fletcher. Practical methods of optimization. SIAM Review, 26(1):143-144) 1984.
Stephanie Forrest. Genetic algorithms: principles of natural selection applied to computation. Sci- ence, (51()33)):72--878 1993.
Stanton A Glantz and Bryan K Slinker. Primer of applied regression & analysis of variance, ed, volume 654. McGraw-Hie Inc., New York, 2001.
Maarten Keijzer. Improving symbolic regression with interval arithmetic and linear scaling, In European Conference on Genetic Programming, pp. 70-82. Springer, 2003.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
John Koza. Genetic programming means for programming computers by natural selection. Statistics and computing, 4(2):87-112, 1994.
Krzysztof Krawiec and Tomasz Pawlak. Approximating geometric crossover by semantic backprop-
agation. In Proceedings of the 15th annual conference on Genetic and evolutionary computation, pp. 941-948, 2013.
Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXiv preprint 19Xiv:19/2.014/2 2019.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invarian neural networks. In Interna- PMLR, 2019.
fional conference machine learning, pp.
Christian Loftis, Kunpeng Yuan, Yong Zhao, Ming Hu, and Jianjun Hu. Lattice thermal conductivity prediction using symbolic regression and machine learning. The Journal of Physical Chemistry A, 125(1):435-455)) 2020.
Qiang Lu, Jun Ren, and Zhiguang Wang. Using genetic programming with prior formula knowledge to solve symbolic regression problem. Computational intelligence and neuroscience, 2016, 2016.
Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geom- etry in point cloud: A simple residual MLP framework. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL httttps:://ppaeeeiiiitie net www.:/oommtddd __76
10

---

Published as a conference paper at ICLR 2023
Georg Martius and Christoph H Lampert. Extrapolation and learning equations. arXiv:1610.02995. 2016.
arXiv preprint
A. Meurer, C. P. Smith, M. Paprocki, Čertík, and Anthony Scopatz. Sympy: Symbolic computing in python. PeerJ Computer Science, 3(3):e 103, 2017.
N. Mundhenk, M. Landajuela, R. Glatt, C. P. Santiago, D. M. Faissol, and B. K. Petersen. Sym- bolic regression via neural-guu geurdl-guided genetic programming population seeding. 2021.
Brenden Petersen, Mikel Landajuela, T Nathan Mundhenk, Claudio P Santiago, Soo Kim, and Joanne T Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. Proc. of the International Conference on Learning Representa- fions, 2021.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapolation and control. In International Conference on Machine Learning, pp. 4442 4450. PMLR, 2018.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. science, 324(5923):81-85,, 2009.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting The journal of machine learning research, 15(1):1929-199) 1958, 2014.
Nicolas Staclens, Dirk Deschrijver, Ekaterina Vladislavleva, Brecht Vermeulen, Tom Dhaene, and Piet Demeester. Constructing no-reference h. 264/ave ditstream-based video quality metric ing genetic programming based symbolic regression. IEEE Transactions on Circuits and Systems for Video Technology, 23(8):1322-1333 2013.
Sheng Sun, Runhai Ouyang, Bochao Zhang, and Tong- Yi Zhang. Data-driven discovery of formulas by symbolic regression. MRS Bulletin, 44(7):559- 564, 2019.
Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A regression. Science Advances, 6(16):eaay2631, 2020.
method for symbolic
Nguyen Quang Uy, Nguyen Xuan Hoai, Michael O' O'Neill, Robert I McKay, and Edgar Galván- López. Semantically- crossover in genetic programming: application to real-valued sym- bolic regression. Genetic Programming and Evolvable Machines, 12(2):91-119, 2011.
Mojtaba Valipour, Bowen You, Maysum Panju, and Ali Ghodsi. Symbolicgpt: A generative trans- former model for symbolic regression. arXiv preprint arXiv:2106. 14131, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Yiqun Wang, Nicholas Wagner, and James M Rondinelli. Symbolic regression in materials science. MRS Communications, 73)5793-8 2019.
Baicheng Weng, Zhilong Song. Rilong Zhu, Qingyu Yan, Qingde Sun, Corey G Grice, Yanfa Yan, and Wan-Jian Yin. Simple descriptor derived from symbolic regression accelerating the discovery of new perovskite catalysts. Nature communications, 11(1))1-8, 2020.
Matthias Werner, Andrej Junginger, Philipp Hennig, and Georg Martius. Informed equation learn- ing. arXiv preprint 1rXiv:2/0S.063:: 2021.
Tailin Wu and Max Tegmark. Toward an artificial intelligence physicist for unsupervised learning. Physical Review E, 100(3)003331, 2019.

---

Published as a conference paper at ICLR 2023
A DETAILS FOR DATASET GENERATION
In this section, we describe the configurations used to generate the training set, validation set, and SSDNC benchmark. We sample each non-leaf node following the unnormalized weighted distri- bution shown in Table 3 Our base data set approximately contains 100,000 unique expressions' skeletons. Then the different training sets are generated according to the number of expressions with different coefficients. We set this number to 10, 20, 30, 40 and 50. The validation set contains 1K skeletons that are randomly sampled from the base data set and assigned constants that differ from the training set. We generate the more challenging test benchmark SSDNC, which includes approximately 100 unique expression skeletons and 10 re-sampled numerical constants for each skeleton. Source code is available at /tttps://githute www.ua..oomIIIia Attp://IW///J////t/// Supervised Learning_fes for_oo_rrsoo SR
Table Unnormalised probabilities of unary and binary operators used by the dataset generator.
Operation Mathematical meaning
Unnormalized probability
add
mul
10
sub div sqrt V- exp
5
exp-
In
In
sin sin
cos
cos
tan
tan
pow2
(1)²
4
pow3
())3
pow4
(1)
pow5 (.)5
B BECHMARK FUNCTIONS
This section describes exact functions used to compare our method with the current state-of-the- art methods. In Table show the name of the benchmark and corresponding expressions.
Table 4: Benchmark functions that have used in experiments. Input variables are denoted by x and/or y. We have restricted ourselves only to the univariate and bivariate functions.
Name
Expression
Nguyen-1
x + 42 +x
2
x² + + +2 +
Nguyen-3
x +x* +I +
Nguyen-4
26 +r* x²
+
5
sin(x?) cos -
6
sin(x) + sin iin(+tx'
N
In(r + + 1n(22 +1)
N
N
sin(x) sin(?²
Ng
ssin(z) (1)
Nguyen-11
x!!l
Nguyen-12 44 x3 + 112 -
Constant-1
3.39 + 2.12. + 1.78%
Constant-2
sin cos x 0.75
Constant-3
sin (1.5x). (0.5g) 2.7x%
Constant-4
12

---

Published as a conference paper at ICLR 2023
Constant-5
V1.23.r
Constant-6
g..126
Constant-7
2sin (1.3z) cos y
Constant-8
In(r + 1.4) + In(x + 1.3)
Keijzer-3
0.3 x sin (2 7.2) .)
Keijzer-4
s² exp (-x) cos (n) (si cos x 1)
Keijzer-6
w-(x+1)
Keijzer-7
In
Keijzer-8
V
Keijzer-9
In (x + V (x² + 1))
Keijzer-10
x/ll
Keijzer-11
Ty + sin ((r 1)(y 1))
Keijzer-12
x² v + 1
2 y
Keijzer-13
6sin (α) cos (y)
Keijzer-14 14
24
Keijzer-15 15
5
1/ I
(x+1)
R-1
R-2
3a
x1+1
R-3
#*+= 4 42+1
Feynman-1
V2-T
Feynman-2
exp(
(2-m)y
Feynman-3
x7 y
Feynman-4
X y
Feynman-5
2 T y²
Feynman-6
Feynman-7
sin(11)
sin(O2)
Feynman-8
V.u
Feynman-9
Feynman-10
1.5 x
Feynman-11
Feynman-12 12
a.y"
Feynman-13 13
x yy²
Feynman-14 14
2-(1+
Feynman-15 2.x
C MODEL HYPERPARAMETERS
In this section, we give more details about the hyperparameters of our models with/without CL., The full hyperparameters can be seen in Table
D BASELINES DETAILS
Transformer-based methods. For SymbolieGPT (Valipour et al.) 2021 and NeSymReS (Biggio al., 202T), we use the standard hyperparameters provided the open-source implementation of these methods [h Some parameters such the dimension of input, vocabulary size, and manually adjusted as the dataset changes. Note that we do not change model hyperparameters that may affect performance.
3ymbol
i cRegressi "hatthcc l
13

---

Published as a conference paper at ICLR 2023
Table 5: Hyperparameters for models.
Encoder
Number of stages
Number of pre-blocks for each stage
Number of pos-blocks for each stage
Original dimension embedding
64
Dimension expansion for each stage
Number of neighbors for FPS
5
Dropout rate
0.1
Decoder
Dimension model
512
Number of heads Dropout rate
0.1
Number of layers
Dimension of output
60
Training
Batch size
128
Learning rate
0.0001
Scale weight (w/ CL) X Temperature parameter
0.2
0.5
Max norm of gradients
1.0
Inference
Beam size
128
Re-start times of BFGS
20
Deep symbolic Optimization (DSO). For DSO (Mundhenk et al. (021), use the standard pa- rameter settings in the open-source implementation! DSO depends on two main hyper-parameters namely the entropy coefficient λи and the risk factor and hyperparameters related to genetic pro- gramming hybrid methods. The λи is used to weight a bonus proportional to the entropy of the sampled expression which is added the main objective. The intervention in the definition of the final objective depends on the (1 (1-c) quantile of the distribution rewards under the current policy. According to the open-source implementation, the chosen hyperparameters are listed in Table
Genetic Programming (GP). For GP-based methods, we opt for the function SymbolicRes ceebbessceerrssorr of open-source Python library gplearn Our choices for the hy- perparameters are mostly the default values indicated in the library documentation. The detailed settings are reported Table 7
Table 6: Hyperparameters for DSO.
Table 7: Hyperparameters for GP.
Parameter name
Value
Parameter name
Value
Entropy coefficient λи
0.05
Population size
1000
Risk factor
0.005
Generations
20
Generations
20
Tournament size
20
Population size
1000
Crossover probability
0.9
Crossover probability
0.5
Mutation probability
0.01
Mutation probability
0.5
Const range
(-5,5)
14

---

Published as a conference paper at ICLR 2023
E EXPLORING PHYSIC LAWS
Symbolic regression used by many research communities to advance the study of numerous scien- tific fields, e.g., Physics (Wu & Tegmark] 2019 Udrescu Tegmark 2020),
Chemistry (Batra al.,
2021), and Materials (Sun et al. 2019 Wang et al. 2019) Weng et al. 2020 Loftis et al. 2020). Our method shows great potential recovering some of the laws of physics. As reported in Table 8
successfully recover all the expressions of two variables in Feynman benchmark test sets. Since is not included in the dictionary during training, the corresponding value in the recovered expression is predicted to be a decimal.
Symbolic regression algorithms are getting better. Our work will be useful for future data-driven symbolic regression methods. We look forward to the day when computer helps physicists discover basic laws physics, even just like Kepler, discovers a useful and hitherto unknown physics expression through symbolic regression.
Table 8: Recovery expressions on each of the Feynman benchmark of our methods. U(a b, c) denotes random points uniformly sampled between and b for each input variable. R² values are rounded to 5 decimals.
ID
Expression
Constant
Dataset
Ours
Accuracy(12))
166.a
exp
4(1,3, 100)
0.39894 exp( -0.xxp5((000 0.502)
0.99999
1,2
0.39888 rxpp -0.4559802
1.6.2
exp
U11,3, 100)
0.99999
V (2-TTa
1.12.1
F n.N
None
41,,5, 100)
N.N,
1.0
L.2.55
F 92
None
5,,,5, 100)
U2 Es
1.0
I.14.4
U = kspring 2
None
(111, 5, 100)
Rapring 2
1.0
1.25.13
V = t
None
(111, 100)
g
1.0
I.26.2
sin(A n sin(Og)
None
4(00, 3, 100)
sin sin(O1) sin (in
1.0
1.29.4
l: =
U(1, 10, 100)
1.0
1.34.27
h W 27
h
U(I,5, 100)
0.15924hw
0.99999
1.39.1
U - p. V
None
U(1, 100)
P.
1.0
11.3.24
h Puze 4mr
U(1,5, 100)
0.079617P
1.0
E
11.8.31
U(1,5, 100)
1.0
2
11.27.18
2
U11, 5, 100)
e
1.0
11.38. 14
# 2(1+))
None
(1,1,,, 100)
0.5Y
1.0
1+0
1II.12.43
J 2x
n.
(1,15, 100)
0.15923mhh
0.99999
15


Reviews:
[{"review_id": "cP_Zgmfld_", "reviewer": "Reviewer_jvHV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a symbolic regression algorithm that uses a transformer backbone to learn a new feature extractor via joint supervised learning mechanism. The ill-posed problem mentioned in the paper was addressed by using the joint supervised learning mechanism combining supervised contrastive learning, which strengthens the similarity of feature vectors from the same skeleton. Experiments were done to prove the efficacy of the method.\n", "summary_of_the_review": "This paper proposes a good approach to tackle the symbolic regression problem. Although the presented algorithm seems to be a combination of contrastive learning and Transformers, its performance on benchmark data sets looks good. ", "strength_and_weaknesses": "This paper is motivated by a practical issue. I like the idea of using the supervision of the whole expression skeleton and the supervision of the preorder traversal of its expression tree in contrastive learning (maybe with data augmentation) to deal with the case that different instances of the same skeleton can have very different shapes, and instances of very different skeletons can be very close.\nThe authors did extensive experiments, the presented algorithm performs very well on the benchmark dataset.\nThe paper has covered a wide range of related work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C7HVQNs2Oa", "reviewer": "Reviewer_eWWS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel transformer-based model for symbolic regression (SR), which produces mathematical expression skeletons from data points. In the proposed method, a feature extractor using pointMLP is added and jointly trained using contrastive loss to realize efficient training. The experimental evaluation using benchmark functions demonstrates that the proposed method exhibits higher recovery rates and R2 scores compared to existing SR methods.", "summary_of_the_review": "Basically, this paper is well-written, and the motivation and contribution are clear. The effectiveness of the proposed method is supported by the numerical evaluation.", "strength_and_weaknesses": "**Strengths**\n* Introducing the idea of feature extractor and contrastive learning into the transformer-based symbolic regression model is convincing. Although each component of the proposed method was previously proposed in the context of other tasks, incorporating such methods into the SR model seems to be novel.\n* The numerical experiment supports the effectiveness of the proposed method.\n\n**Weaknesses**\n* In the experiment, the same dataset is used for training and testing. I am not confident whether such a setting is reasonable. From the machine learning perspective, the obtained model should be evaluated on test datasets.\n\n**Comments**\n* It would be nice if the proposed method was evaluated using recent benchmark datasets such as SRBench (https://github.com/cavalab/srbench).\n* In Figure 2, the redder color might indicate the cosine similarity is closer to **0**.\n* How is the computational cost of the proposed method compared to other SR methods?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6uu49vb_G-", "reviewer": "Reviewer_dp8W", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a transformer-based method for symbolic regression using a new feature extractor and a joint supervised learning mechanism.  They evaluated the framework on synthetic datasets and compared it with several baseline methods. Experimental results show good performance of the proposed framework. ", "summary_of_the_review": "This is a good paper to tackle an interesting problem. While the potential application of the proposed technique and method design should be further clarified. The technical novelty of this paper seems limited.", "strength_and_weaknesses": "Strength: \n\n1. The problem is interesting. \n2. The proposed framework achieves good results. \n\nWeakness:\n\n1. It would be better if the authors could further clarify the potential application of symbolic regression. \n2. It is unclear to me why the authors adopt a point cloud-like feature extractor (like PointMLP). It seems that the tackled problem is a special regression problem, so why not use 1D convolution or directly use a transformer to handle it? Also, there are some works tackling tabular data, like TabNet. The authors should further clarify the motivation of network design. \n3. Since the proposed framework was trained and evaluated on the same synthetic dataset. There may be overfitting problems. The authors should discuss this point. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster