RAG-SR: Retrieval-Augmented Generation for Neural Symbolic Regression
Authors: Hengzhe Zhang
Qi Chen
Bing XUE
Wolfgang Banzhaf
Mengjie Zhang
Venue: ICLR 2025 Spotlight (2025)
Keywords: Symbolic Regression
Genetic Programming
Transformers
Deep Learning
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Abstract: Symbolic regression is a key task in machine learning, aiming to discover mathematical expressions that best describe a dataset. While deep learning has increased interest in using neural networks for symbolic regression, many existing approaches rely on pre-trained models. These models require significant computational resources and struggle with regression tasks involving unseen functions and variables. A pre-training-free paradigm is needed to better integrate with search-based symbolic regression algorithms. To address these limitations, we propose a novel framework for symbolic regression that integrates evolutionary feature construction with a neural network, without the need for pre-training. Our approach adaptively generates symbolic trees that align with the desired semantics in real-time using a language model trained via online supervised learning, providing effective building blocks for feature construction. To mitigate hallucinations from the language model, we design a retrieval-augmented generation mechanism that explicitly leverages searched symbolic expressions. Additionally, we introduce a scale-invariant data augmentation technique that further improves the robustness and generalization of the model. Experimental results demonstrate that our framework achieves state-of-the-art accuracy across 25 regression algorithms and 120 regression tasks.

Published as a conference paper at ICLR 2025
RAG-SR: RETRIEVAL-AUGMENTE GENERATION FOR NEURAL S SYMBOLIC REGRESSION
Hengzhe Zhang!.. Qi Chen Bing Xue Wolfgang Banzhaf Mengjie Zhang 1.3+
Schooo of Engineering and Computer Science, Victoria University of Wellington. New Zealand DDpaatment of Computer Science and Engineering, Michigan State University, USA Centtee for Data Science and Artificial Intelligence, Victoria University of Wellington, New Zealand
ABSTRACT
Symbolic regression is key task in machine learning, aiming discover math- ematical expressions that best describe a dataset. While deep learning has in- creased interest in using neural networks for symbolic regression, many existing approaches rely on pre-trained models. These models require significant computa- tional resources and struggle with regression tasks involving unseen functions and variables. A pre-training-fie free paradigm is needed to better integrate with search- based symbolic regression algorithms. To address these limitations, we propose a novel framework for symbolic regression that integrates evolutionary feature con- struction with neural network, without the need for pre-training. Our approach adaptively generates symbolic trees that align with the desired semantics in real- time using a language model trained via online supervised learning, providing effective building blocks for feature construction. To mitigate hallucinations from the language model, we design a retrieval-augmentte augmeeted generation mechanism that explicitly leverages searched symbolic expressions. Additionally, we introduce a scale-invariant data augmentation technique that further improves the robustness and generalization of the model. Experimental results demonstrate that our frame- work achieves state-of-the-an accuracy across 25 regression algorithms and 120 regression tasks
INTRODUCTION
Symbolic regression (SR) is a machine learning technique that searches the space of symbolic ex- pressions to identify models that best fit a dataset (Sun al., 2023) Fong al., (023). Unlike traditional regression methods, which assume fixed model structure, SR automatically determines both the structure and parameters of the model. This flexibility allows SR achieve both high ac- curacy and interpretability, making it especially valuable in fields such as physics (Udrescu et al.) 2020), biology (Brunton al. 2016), and finance (Liu & Guo 2023), where uncovering transparent, understandable models is crucial for scientific discovery and informed decision-making
In this paper, we focus on an automated feature construction approach to SR (Cava et al,, 2019). The key idea is to generate set of symbolic trees/features, & {ØI...., 2m. J. from dataset [X Y) to enhance the performance of an interpretable model M, such as linear regression (Cava et al. 2019 Zhang et al. 2023a). The objective minimize the loss function L(Ø: X, Y), defined
1 N
L(0;
2m (x.))) Yi)
where N represents the number of instances, and X; and Y; represent the features and label for the i-th instance in the training data. By decomposing the SR task into the discovery of feature sets, this approach reduces the complexity of the problem. Even if each feature é is weakly correlated with the target Y, the model can still perform well as long as the features collectively complement each other in predicting the target. This symbolic regression paradigm is particularly effective for
"Emails: (hengzhe.zhang, qi.chen, bing.xue, mwngjic./hngg ccs www.hhaannn..cs............ banzhafw @mu..duu.
Source Code: http::h/t////. thub. http://ggthh....coco//gggggg//// zhang RRG-SR

---

Published as a conference paper at ICLR 2025
complex real-world problems, where the complexity of the underlying system cannot be captured by a simple equation.
Traditional SR methods, predominantly based on genetic programming (GP) (Banzhaf et al. 1998), perform gradient- free searches within the symbolic space (Jiang & Xue 2024). While effective at exploration, these methods often lack search effectiveness due to limited guidance from accumulated knowledge during the evolutionary process. Recent advances in deep learning for SR (Biggio et al.) 2021 Kamienny al. 2022) aim to address these inefficiencies by leveraging knowledge more effectively.
Deep learningbbased SR typically follows three primary paradigms: pre-trained language mod- els (Biggio et al. 2021 Kamienny et al. 2022), reinforcement learning (Landajuela et al.) [021), and sparse supervised learning (Sahoo al, 2018). Sparse supervised learning does not gener- ate symbolic models directly; instead, it relies on heuristic pruning and neural architecture search to sparsify the network so that can be converted into symbolic expressions (Li et al.) 2024). In contrast, pre-trained language models and reinforcement learning generate symbolic models di- rectly. However, pre-trained language models require prior assumptions about the problem space, limiting their generalizability identifying an optimal set of features for modeling complex real-world systems is tine-consuming, making it impractical to generate many pairs of symbolic models and their outputs for pre- pre-training. While reinforcement learning with a language model offers task adaptability (Landajuela et al.) 2022). its low sample efficiency remains significant drawback. Therefore, it is desirable to explore supervised learning methods that do not rely on pre-training for SR to overcome these challenges.
novel tasks involving unseen functions and features. Additionally,
To develop effective and efficient neural network for SR, we propose novel neural network- based symbolic regression framework inspired by geometric semantic genetic programming (GSGP) (Moraglio al., 2012). As illustrated in Figure [ the core idea is to use neural net- work to dynamically predict the best feature & to replace an existing feature in the current set {o..... 3m with the goal of filling the gap in the residual R, referred to the desired semantics in this paper. Throughout the evolutionary process, the relationship between the seman- tics/outputs of each symbolic tree ((X) and its symbolic representation ø is captured and stored in neural semantic library, which is continuously updated in online fashion.
One challenge with neural semantic libraries that language models may generate features & that are grammatically correct but irrelevant to the desired semantics R. In the language model domain, this is known as hallucination (Sun et al. 2024). To mitigate this, we propose a retrieval-ane -arreeee- generation technique to reduce hallucination and generate symbolic trees that better align with the desired semantics. In summary, the key contributions of this paper are as follows:
We propose a semantic descent algorithm optimize symbolic models using a neural net- work with online supervised learning. The neural network continuously learns to generate symbolic trees that precisely capture the desired semantics, pushing the boundaries of deep symbolic regression to handle complex problems.
To reduce hallucination in language models, we develop a retrieval-augmented generation mechanism. This technique makes the generated symbolic models are only grammati- cally correct but also better aligned with the desired semantics, resulting in more accurate predictions.
To better capture the relationship between desired semantics and retrieved symbolic pressions, we propose a masked contrastive loss, which more accurately generates sym- bolic trees by aligning the embeddings of desired semanties with those of the retrieved expressions.
We propose a data augmentation and double query strategy to fully exploit the scale- invariant properties of feature construction bassed symbolic regression, further improving the effectiveness of generated symbolic expressions.
2
RELATED WORK
In the domain of neural symbolic regression, key advantage of pre-trained models is that, once pre-trained (Biggio et al. 2021 Kamienny al. 2022). models be reused for similar tasks
2

---

Published as a conference paper at ICLR 2025
Solution BLB
New Solution B
Old Solution
Tree
Tree
Current Semnantics
Trcc
Target
Target
Tree
Tree
Tree
Delete
Tree
Replace Subtrees with Randomiy Generated
Generated Neural Netwerkk
Mutation
Prompt
Tree B
May
Residual
Semantics
Tree A'
Tree
Tree
Tasi You
Tree (Pending Replacement)
Target
Query
Neural Nettwork with Retrioual
Computing
Augmertztion to Gonerate New Tree
Foclutionaay Algorithi
Retrieval Augmented Neural Generation [Semantic/Outpu Spoce)
(yytacttc/Symbblk Space)
Figure I:
Comparison of the evolutionary algorithm and retrioval-augmmened neural semantic library for feature construction-bart based symbolic regression.
without further optimization. These models are designed to solve a distribution tasks through mechanisms such invariance encoding (Holt et al. 2023). contrastive learning (Li et al.) 2022). or conditional constraints (Bendinelli et al., 2023) to capture relationships among different SR tasks within problem space. However, these methods may struggle with tasks beyond the scope of the pre-training data, particularly when encountering different function sets or more variables than those seen during training (Shojaee et al. 2024a Meidani al., 2024). Fine-tuning could alleviate the misalignment between training and target tasks, through approaches like reinforcement learn- ing (Holt 2023) or using imitation learning learn successful mutations (Kamienny et al.) 2023). However, fine-tuning large pre-trained language models can be challenging, Thus, exploring how online learning techniques can be applied exclusively to enhance SR remains promising and underexplored direction.
Reinforcement learning (RL). on the other hand, learns the probability distribution of promising symbolic models (Landajuela et al. [2211] Xu et al. (2024) by interacting with the environment, al- lowing it to adapt to different function sets for various tasks. However, deep symbolic optimization via RL often suffers from low sample efficiency, requiring integration with GP (Mundhenk et al.) 2021 or Monte Carlo Tree Search (MCTS) (Xu et al. 2024) techniques to improve performance. Furthermore. RL typically simplifies feedback to a scalar reward, such mean squared error (Lan- dajuela et al. 2021). which limits the richness information provided during the search process. A more effective approach would involve using loss vector rather than scalar loss to provide richer feedback and enhance overall search effectiveness.
Sparse supervised learning methods, such deep equation learners (Sahoo et al. 2018) and efficient symbolic policy learning (Guo et al., 2024). aim to derive interpretable symbolic models by regular- izing neural networks (Zhang et al. 2023c). However, since the Lo norm is con-differentiable these techniques often rely on heuristic pruning approaches to convert neural networks into interpretable expressions. Additionally, they typically require neural architecture search methods identify suit- able architectures before gradient-basee training (Li et al. 2024).
Evolutionary symbolic regression is primarily based on the GP framework, which automatically discovers symbolic models without predefined structures to fit the training data (Fong et al. 2023) Recently, semantic GP has gained substantial attention (Moraglio af. 2012 ZZhng et al. 2023b)
Unlike traditional GP, which operates in the syntactic/symbolic space, semantic GP works in the semantic/output space. By focusing on semantic space, solution generation operators can ensure that the newly generated solutions have more predictable behavior, such as guaranteed loss reduc- tion--ionnn tone-somttiing conventional GP operators often lack. A key challenge in semantic GP is gener- ating GP trees that satisfy the desired semantics (Moraglio et al. 2012). A common strategy is to build semantic library that stores evaluated GP trees (Pawlak et al. 2014). In semantic mutation, this library searched for trees that closely match the target semantics, and the best-matching tree
3

---

Published as a conference paper at ICLR 2025
Algorithm 1 Semantic Descent
1: Input: Features
1d1,-..,0m semantics library L. neural network model N, current semantics ((X), target Y, neural generation probability Peerra
2: Output: Updated features +
3: O < Random permutation of (1,2.....m) m.]
Shuffle tree indices
4: for each i E O do
(X) x- -18)-
Normalized feature
9t ((.X) um + ((X) BiO:(X)
6:
7:
R + Y -((X)um
Compute residual R
8:
if rrand() Preural then
9:
@i (R, L)
D Generate new tree using neural model
10:
((X) 6 @(X) temp
11:
continue
B Proceed to next tree
12:
end if
13: Onew keacct 0i, L)
14: 0 ((X) +
(Drew; Puen (X), (X)), ((X) R. Y)
15:
end for
is selected. However, this approach relies solely existing building blocks without leveraging his- torical knowledge to create new ones. To address this issue, is crucial to incorporate deep learning techniques to learn from the evolutionary learning process and generate better symbolic models that align with the desired semantics.
ALGORITHM
The proposed method is based evolutionary algorithm framework, encompassing solution ini- tialization, generation, evaluation, selection, and archive maintenance. This work primarily focuses on the solution generation phase, introducing neural semantic library for solution generation, de- signed to explicitly retain and apply knowledge throughout the evolutionary process. Solution gen- eration consists of two primary components: semantic descent and retrieval-augmeenee generation.
3.1 SEMANTIC DESCENT
In this work, we propose Semantic Descent (SD), an iterative optimization procedure designed to improve model performance by selectively replacing suboptimal features. Unlike methods such as geometric semantic GP (Moraglio et al. 2012) or gradient boosting (Feng et al. 2018), which incrementally add new features minimize error, SD focuses replacing existing trees in the model with more informative ones. This approach helps maintain a compact model structure while continuously improving accuracy.
At each iteration, a tree de is randomly selected from the set of trees {di,..., 2m } that define the semanticssoutputs of the model ((X) di (X) + ..... 3m + a, where , represents the coefficients and a is the intercept. The contribution of o; is temporarily removed, resulting in temporary semantics (ilmp((X)
((X) (X). The residual R Y deemp (X) of the model is then computed, where Y is the target output. The residual A represents the difference between the prediction and the target.
As shown in Algorithm D the core idea of SD is to fill the gap in the residual R by replacing the current tree de with better alternative, either generated by neural model A (line 9 or retrieved from semantic library L. (line The semantic library L. stores all previously evaluated symbolic trees and subtrees along with their semantics/outputs vxX). The neural model / learns the mapping between the semanties (xX) and the corresponding symbolic tree v. This enables the neural network. A construct a new feature dnew using as input, thereby generating a new feature to reduce the model's error.
The probability of generating new trees using the neural network is Poeurat- detailed in Section 3.2 The probability of retrieving tree from the semantic library is 1 Pacural- The key idea of exact retrieval is to search the library for the tree that most closely matches the desired semantics, i.c., the residual R, as detailed in Appendix C Since the linear regression model automatically adjusts
4

---

Published as a conference paper at ICLR 2025
feature magnitudes and intercepts, the residual R is normalized using the Lg norm before being used as input for retrieval neural generation, i.e., R R-R peated iteratively until all trees & within the solution & have been traversed. By focusing on feature replacement instead of addition, SD enables efficient model refinement while maintaining fixed feature set, allowing for both interpretability and performance improvements.
Ta The replacement process is
3.2
RETRIEVAL-AUGMENTT GENERATION
To learn the mapping between symbolic trees, and their corresponding semantics, ((X), the pro- cess involves three steps: First, the trees and semantics are collected from the evolutionary process (Section 3.2.2 Next, they converted into training data using specially designed encoding rules (Section 3.2.1p Finally, neural network is trained on the collected data (Section 3.2.3) using cross-entropy loss and masked contrastive loss (Section3.2.. 3.2.4
3.2. DATA COLLECTION AND NETWORK TRAINING
The semantic library L is dynamically constructed during the evolutionary process. During solution evaluation, each subtree t and its corresponding semantics (xX) are stored in a first-in-first-ow queue Q with an upper limit of 10,000 entries for training the neural network and future retrieval. To facilitate efficient retrieval, a k-dimensional tree (k-d tree) is constructed using the semantics stored Q the end of each generation in the evolutionary process, reducing query complexity to (log()))) where N is the number of stored trees. The neural network is also trained at the end of each generation. To prevent unnecessary training, an internal validation set monitors per- formance degradation. If the validation loss does not increase, network training is skipped for that generation to save computational resources. Nevertheless, the retrieval library is updated even when network training is bypassed, ensuring that knowledge base is continuously updated throughout the evolutionary process.
3.2.2 ENCODING AND DECODING RULES FOR SYMBOLIC TREES
To generate valid expressions without requiring an explicit end token, we designed specialized encoding and decoding scheme. Symbolic trees are encoded using level-order traversal method, specifically breadth-first search, to convert the tree into linear sequence. To maintain interpretabil- ity, the number of functions in the symbolic tree is capped at np. Given this limit and the maximum number children any function can have, max the maximum number of terminal nodes is:
(nF X (0mmx
Given np and ny, the neural network outputs a fixed-length sequence with the first nF tokens de- coded functions terminals, while while the remaining ny tokens are restricted to terminals by setting function probabilities to zero during decoding. This formulation reframes the symbolic tree genera- tion task multi-class classification problem. Detailed pseudocode for the encoding and decoding processes is provided in Appendix
3.2.3 OVERALL ARCHITECTURE FOR RETRIEVAL-
GENERATION
As shown Figure 2 the neural architecture consists of two main components: a Multilayer Per- ceptron (MLP) and Transformer model. Their relationship is defined as:
O E IR BxLxs Transformer Decoder Transformmr Encoder(d) @ MLP(R)) w
(3)
Each of the two components plays distinct and complementary role in generating symbolic tree ó based on the desired semantics R. The MLP transforms raw semantics into meaningful feature representation that can guide the generation of the symbolic tree. On the other hand, the Transformer encoder processes the nearest symbolic tree o, retrieved from the semantics library L. which serves as prompt reduce hallucination. The outputs of the MLP and Transformer are concatenated and then passed through a Transformer decoder to generate a sequence of tokens. W E Sx is a I.x linear layer that projects the output of the Transformer decoder H R xxxx into the symbol space O E IR L.XS, where the number of unique symbols. These tokens are subsequently decoded into valid symbolic tree.
5

---

Published as a conference paper at ICLR 2025
Semantic Library
Transformer Encoder
Transformer Decoder
Sortmox
Linsar
Add Norm
AGG Norm
Feed Forward
FoCO
Query
%
Prompt
Add Norm
Mulli-tead Altention
Mult-llead
Attontion
X
Oxa
Oas
Vutrlleed
Tree with semantics
Positional Encoding
Positional
X
Encoding
Emberoding
Qutrue
Predictions
Ru
Ryz
Rus
Inputs
Concatenate
Outputs (shifted right)
Desired Semantics
Level Order Traversal
X
MLP
Tree
Figure 2: Neural network architecture for symbolic tree generation.
Intention Encoding: The desired semantics R E BBNN is processed through an MLP to produce a feature matrix F MLP Bx where K is the dimensionality of the hidden layer. The MLP consists of N2 layers, and at each layer i, the transformation is defined as:
X = Dropout, (SiLU, (BN; (W; x; + b,))) +X;
(4)
where x, E BBx is the input to the i-th layer, W; E K'k the weight matrix, b, E K the bias vector, BN, denotes the batch normalization layer, SiLU, is the Sigmoid Linear Unit activation function (Elfwing et al., 2018), and Dropout, is the dropout layer with a specified dropout rate. This MLP layer results a feature matrix F MLP E R BxX K which is then passed through a linear layer to match the dimensionality from K to D, yielding Emaaped E BxD, where D the dimensionality of the Transformer-encoded representation.
Retrieval-Augmented Encoding: For the desired semantics R, a KD-Tree is used to retrieve the nearest symbolic tree I from the semantic library L, based on Euclidean distance and subject to the constraint that the tree contains more than ny nodes. The retrieved tree Ø is then processed the sequence length of the tree
through an embedding layer generate V E TR BxLxE, where encoding and E is the dimensionality of the embedding space. The embedding layer consists of an embedding matrix E S The embedded representation Vg is then encoded using the Trans-
former model to produce symbolic model embedding Hrransfomme Bx The Transformer encoder applies self-attention and feedforward layers with residual connections follows:
Hsalf-Atm = LayerNorm VV + SelfAttention
V
E RBXLX
(5)
Hraasformer LayerNorm (Hsalt-Au + FeedForward (Hselr-Am)) ))
E R BxLxD
Decoding: The combined feature representation HCombined
F mapped MLP + Hranstommer E BX
fed into a Transformer decoder to generate the contextual embeddings H E BxLxD The decoding process is performed auto-regressively. utilizing greedy decoding strategy.
3.2.4 LOss FUNCTION
Masked Contrastive Loss: The intention encoding should ideally learn useful knowledge not only from target expressions ø but also from the retrieved symbolic expressions J. In parallel, the retrieval-augee -aggeeeee encoding should be aware of the semantics of the nearest symbolic expres- sions. To fulfill these objectives, propose contrastive loss that aligns the embeddings from both the intention encoding and the retrieval-augmmened encoding components.
Given the nearest semantics d(X) E R BxN is processed through a MLP to generate feature ma- trix of nearest semantics F nearest BBX Simultaneously, the embedding of the symbolic model

---

Published as a conference paper at ICLR 2025
is averaged along the sequence length dimension produce the averaged embedding Havg E BxD Then, the InfoNCE loss (Oord et al. 2018). a popular objective in contrastive learning, is employed maximize the similarity between the nearest semantics feature matrix F nearest and the averaged symbolic embeddings Havg while minimizing similarity with neg- ative samples from the same batch. To alleviate false negatives, i.e., when two samples in batch are semantically similar, the InfoNCE loss is masked by a mask matrix mask. The mask matrix is designed such that non-diagonal elements with an absolute cosine similarity greater than 0.99 are marked as false (indicating false negatives), while all other entries are marked as true. The masked InfoNCE loss is formally defined as:
Hrunfommrr E
B Σ 1=1 Dj=1 exp (sim(F (F nearest [i], Hagg()) mask/r)
exp (sim( neurest [i], H Hug (i)/T)
LINONCE =
B log
(6)
where sim(...) denotes cosine similarity, and is a temperature parameter controlling the sharpness of the softmax function. This contrastive loss ensures that the nearest semantics are closely aligned with their corresponding symbolic representations in the embedding space, while differentiating them from unrelated samples.
The model is also trained using cross-entropy loss over the sequence of L symbols. Let due E I denote the one-hot encoded ground truth for the i-th position, and ofred E RR denote the predicted probability distribution at that position. Formally, the cross-entropy loss for each sequence is defined Leross-entrppy Σ i=1 o'me log( prea)- The final loss C is a weighted
Cross-Entropy Loss:
sum of the cross-entropy loss and the contrastive loss:
L
Eeross-entrogy + À LInfONCE
where À is hyperparameter that balances the contributions of the two losses.
3.3 DATA AUGMENTATION AND DOUBLE QUERY
In linear regression, the sign of coefficients is automatically adjusted, so the sign of the semantics is not crucial. However, the training data may only include side of training pair (v,, (xX)), with- out considering its opposite, (v), -x(X)) Consequently, when the desired semantics is -x((X. ), the model may fail to generate the correct symbolic tree v. To address this issue, we augment the train- ing data by including both (v,, (X)) and (19, (X)) pairs:
T - TU (w) (x (W, (xX)) ET).
During decoding, both and - are used to query the neural network, generating candidate trees ø and ø'. The with the highest probability is selected as the final symbolic model. This technique, referred to as double query (DQ), allows the model to generate symbolic trees with sign-insensitive semantics, thereby improving the effectiveness of neural generation.
EXPERIMENTS
This section is divided into two parts. The first part evaluates the effectiveness of the proposed components in improving the prediction accuracy of the neural semantic library. The second part investigates the performance of integrating the SR method with the retrieval-aus auggeeeed neural se- mantic library. It compares this integrated approach to stat-offthe-art SR methods.
4.1 EXPERIMENTAL RESULTS OF NEURAL SEMANTIC LIBRARY
Experimental Settings: To evaluate the effectiveness of the proposed techniques in enhancing the learning capabilities the neural semantic library, we conduct the first experiment on synthetic data. The objective is to evaluate how various components contribute to the learning effectiveness of the neural semantic library. In this experiment, 10 variables and 50 training instances are randomly drawn from Gaussian distribution (0, 100). Then, total of 10000 symbolic expressions with random heights h e [0, 5) are generated using the grow method (Banzhaf et al. 1998) from GP and evaluated on the randomly generated data. The maximum number of functions nF is set to and
7

---

Published as a conference paper at ICLR 2025
expressions exceeding this limit are filtered out. To avoid redundancy, only one semantically equiv- alent GP tree is retained, ensuring no symbolic expressions overlap between training and test sets. This setup ensures that the final metric reflects the ability of the neural network to learn patterns and generalize to unseen data, rather than simply fitring to previously seen examples. A total of 80% of the symbolic models evaluation metric is the edit distance (Matsubara et al.) 2022] Bertschinger et al,, (2023) between the generated symbolic tree and the ground truth, where smaller distance indicates that the neural semantic library generates more effective building blocks, significantly aiding the evolutionary al- gorithm in finding optimal solutions. Each experiment is run 5 times to ensure stable and reliable results.
used for training, while the remaining 20% are reserved for testing. The
For the neural network, the dropout rate is set to 0.1. The MI. consists of layers, while both the encoder and decoder Transformers have 1 layer each. The hidden layer size set to 64 neurons. A learning rate of 0.01 and a batch size of 64 are used. Early stopping with patience of 5 epochs is employed to prevent overfitting. The weight of contrastive loss À is set to 0.05.
Parameter Settings:
Experimental Results (Edit Distance); The experimental results for edit distance on the test set are presented in Figure 3 First, comparing neural generation with simple retrieval from the library (W/O NN), neural generation performs better by a large margin, indicating the effectiveness of us- ing a neural network for symbolic tree generation. As for the ablation results of components, the results show that including all components achieves the lowest median edit distance, indicating that the combination of all proposed techniques provides the best overall performance. Among the com- ponents, the RAG technique has the most significant impact, highlighting that external knowledge from the semantic library significantly improves the neural network's 's ability to generate relevant symbolic trees. Data augmentation (DA) also plays crucial role, ranking as the second most im- portant component. Without DA, the model struggles to handle the scale-invarian nature of feature construction, leading to worse performance. The compact boxplots reflect the consistency and relia- bility of these components. Dropout has a moderate positive effect, indicating that overfitting control techniques are helpful for training the neural semantic library. Similarly, contrastive learning (CL) shows moderate impact, confirming the effectiveness of using contrastive loss to align the intention encoding with retrieval augmentation encoding components. Finally, DQ also improves effective- ness, showing that even simply generating multiple solutions during inference
lead to better
solutions, which aligns with findings from large language models (Wang et al.) 2023). The impact of DQ becomes more pronounced in the absence of DA, suggesting that DA partially compensates for the lack of DQ
Experimental Results (Running Time): The running time comparisons in Figure 4 demonstrate that RAG moderately increases the overall running time. However, one advantage of incorporating RAG into the component is that new trees can be seamlessly added the retrieval library to improve accuracy without requiring model fine-tuning, making the algorithm efficient for application in an online learning setting. For DA and DQ, removing these components reduces the running time from 44 seconds to 35 and 29 seconds, respectively, indicating that they do introduce some computational overhead. However, given the accuracy improvements they provide, the increase in computational time is acceptable. Although removing both DA and DQ significantly reduces computational cost, the substantial loss of edit distance from 3.82 to 4.35 outweighs the benefit of faster execution.
Examples of Generated Trees: Table provides examples of symbolic trees generated by the neu- ral network with and without retrieval augmentation, along with the retrieved trees. The results demonstrate that the retrieved trees share certain similarities with the ground truth, such as variable usage. These results validate that providing the retrieval tree as a prompt helps the neural network generate more relevant trees, reducing hallucination compared to relying solely on the desired mantics.
4.2 EXPERIMENTS OF RAG-SR
Datasets: In this study, we primarily focus on 120 black-box datasets from the PMLB bench- mark (Olson et al., 2017), which particularly challenging for pre-training methods (Kamienny jet al. (022) due to the potential absence of simple symbolic expressions to model these datasets. The results on the 1 119 Feynman and 14 Strogatz datasets are presented in Appendix L.2
8

---

Published as a conference paper at ICLR 2025
Table 1: Examples of symbolic trees generated by the retrieval-augmentted neural network, simple neural network, retrieval library, and ground truth.
RAG-NN Generated Tree (Distance)
Simple Generated Tree (Distance)
Sin(Sin(ARGB))
Cas(CosCoss (4)
AQIARG7, ARG8) (0)
Log(Max(ARG7, ARG7)) (3)
Max(ARGI, ARG8) (0)
SubtracKARGi, ARGI) (2)
Squ(Squ
Log(Log(ARG2)) (2)
Subracu ARG7) (0)
Max(ARG7, ARG7) (2)
Retrieval Tree (Distance)
Ground Truth Tree
Ain(A (1)
Sin/Sin(ARG33)
Max ARGO), AQIARG7, ARG8))) (6)
AQIARG7. ARG8)
(aaiaaddiADsSSiCooosAAAA ARG8), ARGI) (6)
Max(ARGI. ARG8)
SquareLLo((RRG2)
SqniSqut
Subrrac(ARG7. ARG6) (2)
Suburac(ARG6, ARG7)
Distribucion Edit Distance by Configuratigg
Median Running Time by Configuration
Modian Running Timre [s]
4.10
4.29
40-
34(20
4.2
4.18
20:00
23/48
4.0
3.82
AB
Wro Di
Wio cl
wro Dropouit
w/o D4
wi RMG
wo DA-DQ
woo Da
wo d
wr proprut
w/o D4
wi D4
Wio
Wi
Figure 3: Ablation study of components based on Figure 4: Ablation study of components with re- spect running time (training and inference).
edit distance on the test set.
The evaluation follows the established procedures of state- of- the-art symbolic regression benchmarks (La Cava al.] 2021). Specifically, each dataset is split into training and testing sets with a 75:25 ratio, and experiments are repeated 10 times for robustness. The R² score on the test set used as the evaluation metric. To better handle categorical variables, we use target encoder Micri-Barreca] (001). Furthermore, to prevent any single feature from disproportionately influencing the semantics, all input features are normalized using min-max scaling (Raymond al., 2020).
Evaluation Protocol:
Parameter Settings: For GP, we follow conventional parameter settings: a population size of 200 and a maximum of 100 generations. Each solution consists of 10 trees, representing 10 features. The probability using neural generation, Pneural, is set to 0.1.
The experimental results on SRBench are presented in Figure The proposed method, RAG-SR, outperforms all state-of-the-ar symbolic regression and machine learning techniques in terms of R² scores. Notably, it surpasses the TPSR method (Shojace al. 2024a). which combines MCTS with pre-trained end-to-end Transformer (Kamienny et al.) 2022) confirmed by the Wilcoxon signed-rank test with The improvement is statistically significant, Benjamini- Hochberggghhhhgeoe correction, shown in Figure This indicates the effectiveness of using a purely online training language model for learning symbolic expressions. Compared to SBP-GP (Pawlak et al. 2014), which is a purely retrieval-basee geometric semantic GP that does not use neural network, the significant advantage of RAG-SR demonstrates the effectiveness of using neural network to dynamically generate symbolic models.
Experimental Results (Accuracy):
Experimental Results (Complexity):: The model complexity of RAG-SR follows the definition of SRBench, where the final model is converted into a SymPy-compatible expression, and the number of nodes in the symbolic tree is counted as a measure of complexity. As shown Figure RAG-SR order of magnitude smaller in size compared to Pr-Tree (Zhang al.
produces models that 2022), which is a piecewise SR method that ranks second in H² scores in Figure 5 The Pareto front of test R² scores and model size rank is shown in Figure 7 where RAG-SR appears the first Pareto front, indicating that RAG-SR achieves a good balance between accuracy and model complexity.
Experimental Results (Training Time): The training time of RAG-SR is comparable to that of FEAT, standard feature-consttuaciio---asss SR method (Cava et al. 2019), suggesting that the

---

Published as a conference paper at ICLR 2025
R² Test
Model Size
Training Time (s)
RRAG-SR
TPSTre
'Operon
'SBP-GP
*FEAT
'EPLEX
XGB
LGBM
"GP.GOMEA
AdaBoost
RandomForest
'ITEA
"AP FE
AAPP
*FFX
KemeeRiidee
"gpleam
"DSR
"MRGP
Linear
'BSR
AlFeymman
-0.55 0.00
0.25
0.50
0.75
1.00
10'
10"
10"
10"
10"
10"
10"
Figure 5: R scores, model sizes, and training time of 25 algorithms 120 regression problems.
Wicooon signod tenk R2 Test, a 6.0e 02
All Wytan
XG2
368
ps fedla
Acalcory
BA.EX
71X
Model Size Rank
All Mikymrrr
TEA
(PStraa
KumoR
L0BM
MRGP
Ozenn
Openit
Randen Fowel
SMp
Xte
APP
ER
on GO-CCONT
R Test Rank
Figure 6: Pairwise statistical comparisons of Figure 7: Pareto front of the rank of test R² scores and model size for different algorithms.
test R scores on regression problems.
computational cost learning a neural semantic library is within an acceptable range. However, compared TPSR, which directly leverages pre-tre trainrdnd model guide SR without requiring fine-tuning, RAG-SR is an order of magnitude slower. This discrepancy is partly due to the fact that, the current implementation, all neural networks in RAG-SR are trained limited computational resources. Training the neural networks on a GPU could potentially reduce the computational time of RAG-SR.
CPU due
CONCLUSIONS
In this paper, we propose novel feature construction-basse SR method with a eetrieval-augmentt neural semantic library. Ablation studies confirm that the retrieval augmentation mechanism effec- tively mitigates the issue of hallucination, enabling the generation of more accurate symbolic trees that align with the desired symbolic trees. Furthermore, data augmentation and double query tech- niques effectively improve the neural network's 's ability to generate symbolic trees that account for scale-invariant characteristics of feature construction-barn symbolic regression benchmarks demonstrate that RAG-SR significantly outperforms state-of-the art SR techniques, including those guided by pre-trained language models. For future directions, introducing constraints on model complexity may help reduce the risk of overfitting, particularly with datasets that contain noise limited samples, presenting promising direction for future search.
SR. Experimental results on large-scale
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ignacio Arnaldo, Krzysztof Krawiec, and Una-May O'Reilly. Multiple regression genetic program- ming. In Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, pp. 879-886, 2014.
Wolfgang Banzhaf, Peter Nordin, Robert E. Keller, and Frank D Francone. Genefic programming: an introduction: on the automatic evolution of computer programs and its applications. Morgan Kaufmann Publishers Inc., 1998.
Tommaso Bendinelli, Luca Biggio, and Fierre-Alexande Kamienny. Controllable neural symbolic regression. In International Conference on Machine Learning, pp. 2063-2077. PMLR, 2023.
Amanda Bertschinger, Q Tyrell Davis, James Bagrow, and Joshua Bongard. The metric is the mes- sage: Benchmarking challenges for neural symbolic regression. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 161-177. Springer, 2023.
Manish Bhattarai, Javier E Santos, Shawn Jones, Ayan Biswas, Boian Alexandrov, and Daniel O'Malley. Enhancing code translation in language models with few-shot learning via retrieval- augmented generation. arXiv preprint arXiv:2407.1900 2024.
Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascan- dolo. Neural symbolic regression that scales. In International Conference on Machine Learning, pp. 936-945. Pmir, 2021.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932-3937, 2016.
William La Cava, Tilak Raj Singh, James Taggart, Srinivas Suri, and Jason Moore. Learning concise representations for regression by evolving networks of trees. In International Conference on Learning Representations, 2019.
Stefan Flfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Feng, Yang Yu, and Zhi Hua Zhou. Multi-layered gradient boosting decision trees. Advances in Neural Information Processing Systems, 31, 2018.
Kei Sen Fong, Shelvia Wongso, and Mehul Motani. Rethinking symbolic regression: Morphol- ogy and adaptability in the context of evolutionary algorithms. The Eleventh International Conference on Learning Representations, 2023.
Arya Grayeli, Atharva Sehgal, Omar Costilla-Reyes, Miles Cranmer, and Swarat Chaudhuri. Sym- bolic regression with a learned concept library. arXiv preprint arXiv:2409.0933: 2024.
Jiaming Guo, Rui Zhang, Shaohui Peng, Qi Yi, Xing Hu, Ruizhi Chen, Zidong Du, Ling Li, Qi Guo, Yunji Chen, et al. Efficient symbolic policy learning with differentiable symbolic expression. Advances in Neural Information Processing Systems, 36, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference Computer Vision, pp. 1026 1034, 2015.
Thomas Helmuth, Lee Spector, and James Matheson. Solving uncompromising problems with lex- icase selection. IEEE Transactions on Evolutionary Computation, 19(5):630-643, 2014.
Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Deep generative symbolic regression. In The Eleventh International Conference on Learning Representations, 2023. URL https:
7/ /ooenrevi ew, net./ fart//oousa forum? d=o KOEEMA bR.
Joey Hong, David Dohan, Rishabh Singh, Charles Sutton, and Manzil Zaheer. Latent programmer: Discrete latent codes for program synthesis. In International Conference on Machine Learning, pp. 4308 4318. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2025
Nan Jiang and Yexiang Xue. Racing control variable genetie programming for symbolic regression. In Proceedings of the AAAI Conference on Artificial Intellience, volume 38, pp. 12901-12900 2024.
Pierre- Deexe-Ale Kamienny, Stéphane d' Ascoli, Guillaume Lample, and François Charton. End-to- end symbolic regression with transformers. Advances in Neural Information Processing Systems, 35:10269-10288 10288, 2022.
Pierre-Alexandre Kamienny, Guillaume Lample. Sylvain Lamprier, and Marco Virgolin. Deep gen- erative symbolic regression with monte-carlo-tree-sean In International Conference on Ma- chine Learning, pp. 15655-15668. PMLR, 2023.
William La Cava, Thomas Helmuth, Lee Spector, and Jason H Moore. probabilistic and multi- objective analysis of lexicase selection and g-lexicase selection. Evolutionary Compulation, 27 (3):377-402, 2019.
William La Cava, Bogdan Burlacu, Marco Virgolin, Michael Kommenda, Patryk Orzechowski, Fabrício Olivetti de França, Ying Jin, and Jason H Moore. Contemporary symbolic regression methods and their relative performance. Advances in neural information processing systems, 2021(DBI):1, 2021.
Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep rein- forcement learning. In International Conference Machine Learning, pp. 5979-5989. PMLR, 2021.
Mikel Landajuela, Chak Shing Lee, Jiachen Yang, Ruben Glatt, Claudio P Santiago, Ignacio Ar- avena, Terrell Mundhenk, Garrett Mulcahy, and Brenden Petersen. unified framework for deep symbolic regression. Advances in Neural Information Processing 35:1emms33:33985333999, 2022.
Wenqiang Li, Weijun Li, Linjun Sun, Min Wu, Lina Yu, Jingyi Liu, Yanjie Li, and Songsong Tian. Transformer-basee model for symbolic regression via joint supervised learning. In The Eleventh International Conference Learning Representations, 2022.
Wenqiang Li, Weijun Li, Lina Yu, Min Wu, Linjun Sun, Jingyi Liu, Yanjie Li, Shu Wei, Deng Yusong, and Meilan Hao. A neural-guided dynamic symbolic network for exploring mathematical expressions from data. In Forty-first International Conference on Machine Learning, 2024. URL https:://ppnreete
net www.:e/oooormmd-eeeeeeeea...ee
Jiacheng Liu and Sigi Guo. Symbolic regressions in nou- nou-physical systems, 2023. URL https: 7/ //openzeview.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In In- ternational Conference on Learning Representations, 2017. URL https://openrrriew
het./Eozun21ddskk$BScca.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL
Bkq6RiCqY7
Yoshitomo Matsubara, Naoya Chiba, Ryo Igarashi, and Yoshitaka Ushiku. Rethinking symbolic regression datasets and benchmarks for scientific discovery. Journal of Data-centric Machine Learning Research, 2022.
Kazem Meidani, Parshin Shojaee, Chandan K. Reddy, and Amir Barati Farimani. SNIP: Bridging
mathematical symbolic and numeric realms with unified pre-tinin In The Twelfth International Conference on Learning Representations, 2024, URL http/htttss//peeeeiiii ew net fforum?
id KGS
Daniele
preprocessing scheme for high-cardinality categorical attributes in clas- sification and prediction problems. ACM SIGKDD explorations newsletter, 3(1):27-32, 2001.
12

---

Published as a conference paper at ICLR 2025
Alberto Moraglio, Krzysztof Krawiec, and Colin G Johnson. Geometric semantic genetic pro- gramming. In Parallel Problem Solving from Nature- XII: 12th International Conference, Taormina, Italy, September 1-5, 2012, Proceedings, Part I 12, pp. 21-31. Springer, 2012.
Terrell Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio Santiago, Brenden Petersen, et al. Symbolic regression via deep reinforcement learning enhanced genetic programming seeding. Advances in Neural Information Processing Systems, 34:24912-24999 2021.
Ni, Russ H Drieberg, and Peter I Rockett. The use of an analytic quotient operator in genetic programming. IEEE Transactions on Evolutionary Computation, 17(1):146-152, 2012.
Randal Olson, William La Cava, Patryk Orzechowski, Ryan J Urbanowicz, and Jason H Moore. Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData mining, 10:1-13, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 107.0:1807.03348, 2018.
Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Retrieval augmented code generation and summarization. arXiv preprint arXiv:2108. 16001, 2021.
Tomasz P Pawlak, Bartosz Wieloch, and Krzysztof Krawiec. Semantic backpropagation for design- ing search operators in genetie programming. IEEE Transactions Evolutionary Computation, (9(3):326344) 2014.
Christian Raymond, Qi Chen, Bing Xue, and Mengjie Zhang. Adaptive weighted splines: A new representation to genetic programming for symbolic regression. In Proceedings of the 2020 Ge- netic and Evolutionary Computation Conference, pp. 1003-1011, 2020.
Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapolation and control. In International Conference on Machine Learning, pp. 4442-4450. Pmlr, 2018.
Parshin Shojaee, Kazem Meidani, Amir Barati Farimani, and Chandan Reddy. Transformer-baat based planning for symbolic regression. Advances in Neural Information Processing Systems, 36, 2024a.
Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. Lim-sr: Scientific equation discovery via programming with large language models. arXiv preprint arXiv:2404. 18400, 2024b.
Fangzheng Sun, Yang Liu, Jian-Xun -un Wang, and Hao Sun. Symbolic physics learner: Discov- ering governing equations via monte carlo tree search. The Eleventh International Confer- ence on Learning Representations, 2023. URL htttttts://oppeeeee /wwp///veeee.....////. net fwr.m? id ZTK35efE8_2.
YuHong Sun, Zhangyue Yin. Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Hui Zhao. Benchmarking hallucination in large language models based unanswerable math word problem. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLIN) 2024), pp. 2178-2188, Torino, Italia, May 2024. ELRA and ICCL. URL
1tt.:22222.....1e000 1rec- main. in. 196
Udrescu, Andrew Tan, Jiahai Feng, Orisvaldo Neto, Tailin Wu, and Max Tegmark. Ai feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. Advances in Neural Information Processing Systems, 33:4860 2020.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. The Eleventh International Conference on Learning Representations, 2023. URL https://openrest net 1t
13

---

Published as a conference paper at ICLR 2025
Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. Coderag-bench. Can retrieval augment code generation? arXiv preprint arXiv:2406. 14497, 2024.
Yilong Xu, Yang Liu, and Hao Sun. Reinforcement symbolic regression machine. In The Twelfth International Conference on Learning Representations, 2024.
Hengzhe Zhang, Aimin Zhou, Hong Qian, and Hu Zhang. PS-Tree: piecewise symbolic regression tree. Swarm and Evolutionary Computation, 71:101061, 2022.
Hengzhe Zhang, Qi Chen, Bing Xue, Wolfgang Banzhaf, and Mengjie Zhang. Modular multi-tree genetic programming for evolutionary feature construction for regression. IEEE Transactions on Evolutionary Computation, 2023a.
Hengzhe Zhang, Qi Chen, Bing Xue, Wolfgang Banzhaf, and Mengjie Zhang. A semantic-based hoist mutation operator for evolutionary feature construction in regression. IEEE Transactions on Evolutionary Computation, 2023b.
Michael Zhang, Samuel Kim, Peter Lu, and Marin Soljačić. Deep learning and symbolic regres- sion for discovering parametric equations. IEEE Transactions on Neural Networks and Learning Systems, 2023c.
14

---

Published as a conference paper at ICLR 2025
A ALGORITHM WORKFLOW
The algorithm workflow, shown in Figure includes the following steps:
The population is initialized using random symbolic trees gener- ated through ramped half-and-half methods (Banzhaf al., 1998). In this approach, half of the GP trees are initialized with random depths, while the other half are initialized at the maximum initial depth. Each solution contains m symbolic trees, with each tree represent- ing unique feature.
Solution Initialization:
Solution Evaluation: Features (X)(X).... ..... 2m (X)} are constructed from the symbolic trees 1- = {Øi...., )m and the training data X. These features are evaluated via ridge regression using efficient leave-one-out voos-valioation to compute the loss (((()))
Semantic Library Construction: To reduce the dimensionality of inputs to the neural network and optimize memory usage, a subset of hard instances is selected every 10 gen- erations. Specifically, all training instances X; X are ranked by the median loss across all solutions, i.e., Median (()(),) T P), and the top k = 50 instances, ranked in descending order, are chosen. The instances are further sorted by their corresponding training labels Y, Consequently, the positions of some hard instances remain unchanged after instance selection, which may facilitate the continuous training of the neural network. The semantics of these selected instances are stored in retrieval library, which is used for training the neural semantic library and for exact retrieval. is important to note that in- stance selection is solely for semantic library construction and semantic descent. Solution evaluation is conducted on the entire training dataset.
Retrieval Library: Symbolic trees and their corresponding semantics are stored in a first-in-first-out queue Q, which can hold up 10,000 symbolic trees. A KD-Tree is used for efficient retrieval, reducing the search complexity from O(N) to O(log(N)) For neural network training, a subset of symbolic trees from Q, where function nodes are fewer than or equal to five, is used to construct a separate KD-Tree for rapid retrieval of eligible trees.
The neural network is trained using pairs of symbolic trees and their corresponding semanties from the retrieval library. The architecture, described in Section 3.2 combines a multi-layer perceptron with a Transformer to process desired semantics with retrieved symbolic trees.
Neural Semantic Library:
Lexicase selection (Heimuth et al., 2014) is used to identify promising solutions based on their objective values. Solutions are iteratively eliminated by applying randomly constructed criteria. In each round, a random instance E [O, .N] is selected, and solutions with losses greater than the median absolute deviation are eliminated (La Cava et al., 2019). This elimination process continues until only one solution remains, which is then selected as the promising solution. For population of P solutions, P solutions selected by iteratively applying these steps.
Solution Selection:
New solutions are generated based on the selected promising so- lutions using a memetic algorithm framework. First, semantic descent performs local search to improve a solution, followed by evolutionary search to explore the search space and mitigate the risk of local optima:
Solution Generation:
Semantic Descent:
As detailed in Section 3.1 the residual R is used to query the neural semantic library the retrieval library to either generate retrieve GP trees. The neural generation process is described in Section 3.2 For retrieving GP trees, Appendix Ceepllains the strategy to retrieve relevant GP trees from the library based on desired semantics. Since the retrieved features have already been evaluated, the loss reduction can be easily calculated after replacement, explained in Appendix D If replacing the tree does not reduce the loss, the current round of replacement is skipped.
Evolutionary Search:
New symbolic trees are generated using genetic programming
operators such random subtree crossover and mutation (Banzhaf al. 1998). For m symbolic trees, the crossover and mutation operators are applied m times with their respective probabilities ensure sufficient exploration of the search space.
15

---

Published as a conference paper at ICLR 2025
End
Termination?
Population Initialization
Archive Maintenance
Solution Evaluation
Neural Network Training
Retrieval Library Construction
Retrieval Library Construction
Neural Network Training
Solution Evaluation
Parent Selection
Exact
Retrieval
Evolutionary Generation
Neural Generation
Neural Generation
Figure 8: Workflow of neural semantic library-based symbolic regression.
Archive Maintenance: The historically best-performing solution is stored in archive for making predictions on unseen data.
The processes solution evaluation, library construction, solution selection, solution generation and archive maintenance are repeated until the stopping criteria are met.
B LIBRARY UPDATE
The library is updated the end of each generation during the evolutionary process. As detailed in Algorithm 2, the library is managed using two ftst-in-first-ou queues, T and S, which store the most recent 10,000 trees and their corresponding semantics, respectively. The semantics (X) are first normalized to t'nom( ()) to ensure that retrieval is insensitive to scale. This normalization is necessary because the linear regression algorithm can automatically determine the scaling coefficient for each constructed feature. Subsequently, all GP trees appended to the queue T, while their corresponding semantics are appended to the queue S and stored in a hash set Hseen-
To control memory usage, the hash set prevents duplicate GP trees from being added to the library. When the queues T and reach their maximum limit Tmax, the oldest entries are removed, as illustrated in Line 21
For computational efficiency, the semantics each subtree (X) are precomputed during the solu- tion evaluation phase. Consequently, during the library update phase, it is unnecessary to recompute the semantics of each tree & based on the data X. This precomputation ensures that the library update process is computationally efficient.
Additionally, for efficient querying in exact retrieval, KD-Tree is constructed at the end of the library update. To efficiently retrieve symbolic expressions that meet the function node limits of neural generation, another KDD-Tree is constructed based on symbolic expressions with a maximum of n function nodes. More details on the construction of this KD-Tree for neural generation are provided in Appendix FF Together, the two queues, T and S, along with the two KD-Trees, form the semantic library L.
C EXACT RETRIEVAL OF THE SYMBOLIC TREE
Exact retrieval is directly using symbolic trees from the retrieval library that align with the desired semantics R. complementing neural generation. This occurs with probability of 1 Pacurat- as described in Algorithm I this process, the desired semantics R are used to retrieve the top 10 symbolic trees with the smallest distances to R, as shown in Algorithm These trees
16

---

Published as a conference paper at ICLR 2025
Algorithm 2 Library Update
Input: Population individuals P, tree queue semantics queue S, hash set seen semantics
Hseen maximum tree limit max
2: Output: Updated tree queue T and semantics queue S
3: for each individual Φ E P do
4: for each pair (vb(X), where v is GP tree/subtree & and (x() is corresponding
semantics do
5:
(X)( = then Continue
6:
Skip semantics with zero norm
7: end
8:
Viccm (X)
(X]]
((X)
Normalize semantics
9:
Dere (X) contains NaN or oc then
10:
Continue
Skip invalid trees
11:
end
12:
if morm (X) Howen then
13:
Continue end if
Skip already seen semantics
15:
T + TU
D Add tree to queue Add semantics queue > Update hash set of seen semantics
S UU {V'mom (X)}
17:
Haven &- Hoven U { [Nnom ( (X))
end for
19:
end for
20: if
[ > Tmax then
Retain only the most recent Tax entries
21:
T + T Tmax
22:
S S
Tmax
23: end if
Algorithm& Exact Retrieval
1: Input: Desired semantics R, current tree Φ. semantic library C, maximum retrieval count
K 10
2: Output: Proposed tree Onew
3: Lx + Top-k (C, distance C, R))
Retrieve top k trees with smallest distances to R Sort by increasing distance to > Initialize proposed tree
4: fu + Sort(fxxx distance Lrx, R))
5: Onew +- None
6: for each tree & E Lx do
7: if size(() S size(di) then
8:
Onew 6- Φ Break
Select the first valid tree Stop further iteration
9:
10: end if
end for
12: if &now = None then
Return Skip
Skip generation if no valid tree is found
14: else
Return Onew
D Return the proposed tree
16: end if
then ranked based their distance the desired semantics R, and the first tree with the same or fewer nodes than the current tree Φ; is selected as the proposed tree Φnew. This size constraint encourages the generation of simpler, more interpretable symbolic expressions. If no such tree is found, the current round of retrieval-augmenter generation is skipped.
D EXACT REPLACEMENT
Exact replacement is mechanism that replaces features in solution with more effective features stored the retrieval library. Since both the features & and their associated semantics o(X) are stored in the retrieval library L, it is straightforward to determine the optimal scaling factor during
17

---

Published as a conference paper at ICLR 2025
Algorithm 4 Exact Replacement
1: Input: New proposed tree (rew, current semantics ((X), proposed updated semantics @rew (X), temporary semantics $ (X) ((m)e residual R, target values y
2: Output: Updated tree ds, updated semantics 4
3: if (new @ then
4: continue
No suitable replacement found
6: if [|x(X)]2
Aeew then To (X)Il2
continue
Identical semantics; skip replacement
8: end
9: 3 + Lobernl X) - Bree (R.-
> Compute scaling factor
(Shen Srew (x))²
10: R 3 Onew (X)
D Compute bias term Propose updated semantics
+ 8. onow (X) + 12: MSEaall + [X] )5Ex (((,) y,
11: ((X)onil x- ((X) jamp
13: MSEeureae + |X| jee (Ø(),) - -2,)
MSEnaal S MSEcumeau then
15:
Φ + Onew
> Accept replacement
16: Φ(X) + (X)
17: end if
the replacement process and calculate the new residual after the replacement. The detailed logic is presented in Algorithm coefficient 3 and bias term a, respectively, based on the semantics of the newly proposed feature, (X)), and the desired semantics, R. The algorithm then calculates and compares the error before and after the replacement, denoted by MSEcurrem and MSE respectively. As shown in Line [5] the error decreases after the replacement, the new feature is accepted, and the semantics are updated. Otherwise, the original feature retained.
Specifically,, Lines g and I0 describe how to compute the scaling
E TRAVERSAL ALGORITHM
Prediction to Symbolic Tree The output format of the neural network consists of a sequence of n nr elements, where the first n elements represent functions or terminals, and the remaining nr elements represent terminals. For example, the output could be [mall abs, add, 21, 22, vgl. which corresponds to the mathematical expression muul(bbs(xi)) add( xa))), pre-order traversal of a symbolic tree. Thus, to decode the output, conversion from level-order traversal to pre-order traversal is required, as described Algorithm Since the neural network does not use an explicit end token, some terminal elements may remain unused after decoding and are discarded. As shown in 5i a tracking variable is used to monitor whether a full symbolic tree has been formed. Once complete tree generated, the decoding process stops, and any remaining tokens discarded. Additionally, during decoding, if a constant placeholder is predicted (line (line[8), random constant is sampled from the uniform distribution 11|1, 1]. This delegates the task of constant tuning to the evolutionary algorithm rather than relying on the neural network to predict the constant accurately.
Symbolic Tree to Training Target For training data, symbolic expressions such as mul(abs((1)). add(x2, xz)) are represented as [mul, abs, I1, add, x2, zz]. However, to serve as proper training labels, this representation must be reordered to reflect level-order traversal. The algorithm for converting a list of pre-order traversals into level-order representations is detailed in collect the required number of child nodes at each level (line 9 and then recursively process these child nodes to obtain subtrees in pre-order traversal (line 13). Finally, the subtrees are assembled into an integral tree pre-order traversal (line 17
Algorithm 6 The key idea
F NEURAL NETWORK TRAINING DETAILS
F.1 PARAMETER SETTINGS
To encourage the neural network to escape local optima, we employ cosine annealing with warm restarts (Loshchiloy & Hutter 2017). Additionally, the Adam optimizer with decoupled weight
18

---

Published as a conference paper at ICLR 2025
Algorithm 5 Level Order Traversal to Preorder Traversal
1:
procedure LEVEILORDRRRRAARRSSS..)
N is the list of node names > Initialize tracking variable
2:
7 0
3:
S+
0
Initialize an empty list S Iterate over each node in N Exit loop if complete tree has been formed
4:
for each n, EN do
5:
if then break
6:
7:
end
8:
if F then
Check if n, is function in F
9:
+ Fln,]
10:
- arity
P Get the arity of function f
11:
7 + r --1 +a - S S
12:
13:
D. Append f to list S Check if is terminal in T
else n4 E T then
15:
+ 7-1
t T[me]
17:
if' is constant then
1 f 11-111111]
19:
end if
20:
Su
Append to
21:
end if
22:
end for
23: return S
Return the final list S representing the tree
24:
end procedure
Algorithm Preorder Traversal to Level Order Traversal
1: procedure PREORDEETRRAVERSALON
2: if 0 then
Return an empty list if nodes are left
3: return 0
4: end
5: Thro < pop(N, 0)
Get the first node Check if the root a function Initialize argument list > Initialize root argument list Collect the function's 's arguments
6:
if Tmm is Primitive then
7:
A +-0 0
8:
R < 0
9:
for i + to (Troo) do
10:
n. +- pop(W.0)
11:
R. R (n.)
12:
end for
for each n; E do
Reorder arguments recursively
14:
M₁ + PreorderTraversall A AUN
UN)
16:
end for return (Nrone) } UAA
Return the function followed by its subtrees
18:
else
19:
return }Rooot
> Return the terminal as is
20:
end if
21:
end procedure
decay (Loshchilov & Hutter 2019) is used to mitigate overfitting, with weight decay parameter of 10 For the MLP initialization, we apply Kaiming initialization (He et al., 2015) to ensure appropriate scaling of the weights. The number of training epochs is set to 1000.
19

---

Published as a conference paper at ICLR 2025
Table 2: Functions and Corresponding Expressions
Primitive
Expression
Add
+ a -
Subtract
Mul
b a v
AQ
Sqrt
Va
Log
log( a + 1)
Abs
Square
Max
max(a, b)
Min
min(a, b)
Neg
-a sin(T x a)
Sin
Cos
cos( iT x a)
F.2 TRAINING PROCEDURE
The pseudocode for training the neural network is provided in Algorithm 7 First, the data is split into training and validation sets, as shown in Line The data consists of pairs of semantics and GP trees from the library L. excluding GP trees that exceed the limit on the number of function nodes. Xunin represents the semantics of GP trees, and Virain represents the nodes of GP trees correspond- ing to the semantics in Xuain- Similarly, X.al and Yval are the corresponding validation data with analogous meanings. Next, the data is augmented using the data augmentation strategy introduced Section 3.3 Subsequently, retrieval trees are prepared for each instance in both the training and validation data.
To avoid trivializing the training task as a simple retrieval task, the KD-Tree is initially constructed using only the training set, shown in Line 4 When preparing the training data in Line the second-neares. tree is selected as the nearest GP tree to form the sets of retrieved trees, N Irains since the nearest tree corresponds to the GP tree itself. For the validation set, the KD-Tree includes only the training set Line (5] the nearest GP tree is used as the retrieved tree to form the set N After retrieving the required samples, the KD-Tree is reconstructed using combination of the training and validation sets, as shown in Line 7
To computational resources during training of the neural semantics library, employ three strategies. First, the neural network is continuously trained during the evolution process. Second, the beginning of training, current validation set loss, Leurrent is smaller than the previous validation set loss, L If the validation loss does not degrade, training is skipped to save computational resources. Notably, the construction of the KD-Tree precedes this step, ensuring that the retrieval mechanism can always access the latest information without requiring updates to the neural network. This a key advantage of RAG-based neural generation. Lastly, the neural network tracks the validation loss throughout the training process. If the validation loss does not improve over iterations, the algorithm performs early stopping to conserve computational resources. These three mechanisms collectively ensure that RAG-SR an efficient SR algorithm.
shown in Line 9 the validation loss checked determine the
G DETAILS OF GENETIC PROGRAMMING PARAMETERS
The crossover and mutation rates for GP are set to 0.9 and 0.1, respectively. The depth limit of GP trees is set to 10. The function set used in GP is listed in Table [ To prevent division by errors, the division operator is replaced with the analytical quotient operator (Ni et al. 2012). Random constants are drawn from uniform distribution, 2 [11,,]]
20

---

Published as a conference paper at ICLR 2025
Algorithm 7 Training NN
Input: Training data D train batch size epochs T, learning rate 7,, validation split a, patience
T. last validation loss prov
2: (Xirain Xval, Vurann Y Yval SplitDataa Dirain: a)
3: (Kuriin Xval, Nuain Yval)
AugmentData Xtrain Xval, Vtrain Vval)
4: (Nirain: Xmain: Strii)
RetrieveTrees/ (rann Yrain)
5: K. < BuildKDTree X
main; Ytrain)
KC is the KD-Tree
6: (N val: Xval- Yval) <
Yval)
7: IC <- BuildKDTree Xurain U X Vtrain U Yval)
D Reconstruct KD-Tree using full data
curvent f Calculate ValiiittiillliiiinnLsssos (Xval, Yvals Nval) 8: L val
9: if Leurentt L prev then
10:
return
Skip training if validation loss improves
11: end if
12: L yal
Initialize best validation loss > Initialize patience counter Train using augmented data and retrieved data
13: patience f
14: for
t= 1 to T do
15:
TrainBatch( train Ymain: N Nerin)
Lval + Calculate ValidationLoss Xaal Vval: Nval)
17:
if' Lval < best then
18:
L < Lval
D Update best validation loss Reset patience counter
Opatience t
20:
else
Opatience < Cpatience
Increment patience counter
22:
end if
23: if Patienes > T then
24:
Break
> Early stopping
25:
end
26:
end for
27:
return
H EFFECT OF CONTRASTIVE LEARNING
The contrastive learning mechanism is designed capture relationships between samples within each batch. To evaluate its effectiveness in enhancing representation learning, we present the pair- wise correlation between the encodings of MLP in Figure The first two figures show the corre- lation of the encoded desired semantics Immaapedd for randomly selected test instances, with and without contrastive learning, respectively. The third figure shows the raw correlation between de- sired semantics, while the fourth figure presents the absolute correlation, which is a more appropriate metric for feature construction-bart trastive learning successfully captures two desired semantics with opposite signs that are highly correlated and represent the same symbolic expression. In contrast, without contrastive learning, the neural network lacks this knowledge and may generate significantly different trees when queried with an opposite sign, fails recognize their equivalence. These results highlight the benefit of using contrastive learning to jointly train the intention encoding and retrieval augmentation encoding components.
symbolic regression techniques. The results indicate that con-
I SENSITIVITY OF THE WEIGHT OF CONTRASTIVE LOss
In this section, we analyze the sensitivity the contrastive loss weight in RAG-NN, Specifically, we evaluate five different À values: 0.01, 0.025, 0.05, 0.1, and 0.2. The impact of varying À on the edit distance of the test set is shown in Figure I0 The results demonstrate that the weight of the contrastive loss significantly influences accuracy. While the contrastive loss can improve performance, the weight À must be carefully tuned using cross-vard validation. Otherwise, the contrastive loss may fail to achieve optimal performance. Regarding training time, the effect of different values on runtime is presented in Figure [[] Overall, the runtime remains stable across different values. Therefore, when tuning X, accuracy should be the primary consideration.
21

---

Published as a conference paper at ICLR 2025
With Contrastive Learning
Without Contrastive Learning
R2: 0.7506
R2: 0.2735
0.10
0:22
0.52
0.07
0.22
1.00
0.50
0.17
0:10
0.14
0.64
0.59
0.17
0.15
0.20
1.00
0.26
0.57
0.07
0.21
0.41
0.15
0.05
0.20
0.35
0:23
0.21
0.24
0.17
0.20
1.00
0.06
0.35
0.15
0.20
0.92
0.22
0.16
0.57
0.26
1.00
0.00
0.35
0.01
0.15
0.35
0.07
Embedding Index
Embedding Index
0.68
0.14
0.05
0.35
0.16
1.00
0.04
0.26
0.29
0.07
0.31
0.21
0.38
0.35
0.69
1.00
0.08
0.59
0.72
0.19
0.10
0.64
0.35
0.15
0:10
0.04
1.00
0.27
0.23
0.20
0.52
0.41
0.21
0.01
0.14
0.08
1.00
0.23
0.02
0.43
0.59
0.79
0.20
0.44
0.26
0.27
1.00
0.12
0.29
0.35
0.15
0.11
0.15
0.68
0.59
0.23
1.00
0.22
0.35
0.20
0.17
0.20
0.92
0.10
0.29
0.23
0.12
1.00
0.19
0.07
0.05
0.29
0.35
0.40
0.72
0.34
0.22
1.00
0.12
0.20
0.08
0:27
0.02
4
Embedding Index
Embedding Index
Raw Correlation
Absolute Correlation
1.00
-0
11
0.31
0.04
-0.23
-1.00
0.11
-0.31
-0.04
0.23
1.00
0.11
0.31
0.04
0:23
1.00
0.11
0.31
0.04
0.23
0.05
1.00
0.05
1.00
0.31
.0.11
1.00
0.06
-0.52
-0.31
0.11
-1.00
0.06
0.52
D.31
0.11
1.00
0.06
0.52
0.31
0.11
1.00
0.06
0.52
Emberading Index
Embeeding Index
-0.23
-0.
05
-0.52
-0
05
1.00
0.23
0.05
0.52
0.05
-1.00
0.23
0.05
0.52
0.05
1.00
0.23
0.05
0.52
0.05
1.00
0..11
0.11
11000
0.11
.0.28
0.05
.0.11
1.00
.0.11
0.28
-0.05
0.11
1.00
0:11
0.28
0.05
0.11
1.00
0.11
0.28
0.06
-0.31
0.11
-1.00
-
0.06
0.52
0.31
-0.11
1.00
0.06
-0.52
0.31
0.11
1.00
0.06
0.52
0.31
0.11
1.00
06
0.52
.0.04
-0.28
-0.06
-1.00
0.05
0.04
0.28
0.06
1.00
-0.05
0.04
0.28
0.06
1.00
0.05
0.04
0.28
0.06
1.00
0.05
0.23
0.05
0.52
0.05
-1.00
-0
23
-0.05
-0.52
-0.05
1.00
0.23
0.05
0.52
0.05
1.00
0.23
0.05
0.52
0.05
1.00
Embedding Index
Embedding Index
Figure 9: Effect of contrastive learning pairwise correlation of MLP encodings.
J DIFFERENT NUMBER OF VARIABLES
The effect of the proposed components the neural semantic library may vary with the number of variables. In this paper, we study the impact of the number of variables on the results. The experiments in Section 4.1 are conducted with a variable count of 10. For comparison, additional experiments with variable counts of (15, 20, 25, 30, 50} are also performed. Since more variables mean more possible symbolic expressions, the number symbolic expressions sampled is scaled by the ratio between the number of variables and 10. The results show that the conclusions of the ablation studies Section 4.1 remain consistent across different numbers of variables. Specifically, both RAG and DA important contributors to the effectiveness of the neural semantic library. Interestingly, the impact of RAG becomes more significant as the number of variables increases. A possible reason for this trend is that the data might be imbalanced, with some variables being
22

---

Published as a conference paper at ICLR 2025
Distribution of Edit Distance by Configuration
Median Running Time by Configuration
60
3.97
3.92
3.91
Median Running Time
44
43006
Distance
40
39,30
36/61
Edit
20
3.85
3.80
0.01
0.025
0.09
0.1
0.2
0.01
0.025
0.05
0.1
Figure 10: Impact of varying À edit distance
Figure 1
Impact of varying À running time.
of the test set.
Distance by Number of Variables
Training Time by Number of Variagles
3.90
3,83
Running Time (s)
Edit Distance
3.85
300
3.76
3.75
3.80
3:76
200
3.72
134.89
103,50
104,60
100
67:54
44.41
10
15
20
25
50
10
20
25
50
Number of Variables
Number of Variables
Figure 12: Impact of varying the number of vari- ables edit distance the test set.
Figure 13: Impact of varying the number of vari- ables running time.
Distribution of Edit Distance by Configuratinn
Median Running Time by Configuration
100
15
4.15
Median Time
27
Est 1atooaaaoo
4.2
75
62
51
3.77
3.93
50
3,8
3.72
3.76
25
wio DQ
wio Drapait
wio NN
wio DA+DO
Wr BAS
wio DV
wo Drapout
WrO DA
wio DA+DD
wio RAG
w/
wD
wD
Figure 14: Ablation study of components based on edit distance on the test set (15 variables).
Ablation study of components with respect to running time (15 variables).
Figure 15:
underrepresented. In such cases, the retrieval process can extract the relevant variables, even if they represent only a small portion of the data. Therefore, the RAG technique becomes increasingly important as the number of variables increases. More analysis of why RAG could be more effective in high- dinddsiimnsiinal scenarios is provided in Appendix K
To better visualize how the number of variables influences the effectiveness of the neural semantic library, we present the change in edit distance with respect to the number of variables in Figure 12 and the running time with respect to the number of variables in Figure 13 The results indicate that the model can maintain stable level of prediction accuracy with different number of variables. This indicates that the proposed method demonstrates good scalability for tasks with varying numbers of variables. In comparison, the end-to-end Transformer for symbolic regression (Kamienny et al. 2022) can only be used to train tasks with up to 10 variables. Regarding training time, an increase in the number of variables does lead to longer training time; however, the increase is modest. These results demonstrate that the neural semantic library is also efficient for tasks with different numbers of variables.
23

---

Published as a conference paper at ICLR 2025
Distribution of Edit Distance by Configuration
Median Running Time by Configuration
Median Runrinng Time
104,82
EiiDiiance
100
92
4.03
4.2
52
5393
3.76
3
3.8
WiB DO
w/o Dropout
wio DA
w/o NN
wo RMG
wro DA+00
wio DO
wio Dropout
CL
wr DA
wo RAG
wio DA+00
wio
Figure 16: Ablation study of components based on edit distance on the test set (20 variables).
Figure 17: Ablation study of components with respect to running time (20 variables).
Distribution of Edit Distance by Configuration
Median Running Time by Configuration
4.48
4
Edt Di
4.50
4.07
Median Running Time
113
26
104.60
96 34
205 75
8637
4.25
4.13
59
4.00
3:74
3.76
3.80
50
3.75
wro D0
wio Drapoit
wio NN
wio DA+D0
wro RAS
wio CL
w/o Dg
wi Drapout
Wr DA
wio DA+D0
wio RAG
w/
Wro
Figure 18: Ablation study of components based
Figure 19: Ablation study of components with respect to running time (25 variables).
on edit distance on the test set (25 variables).
Distribution of Edit Distance by Configuration
Median Running Time by Configuration
4
Median Runring Time
Eait Disrance
150
134
69
128.00
129
21
123.9
124.40
3.81
3,89
90
92/72
4.00
3,75
3.78
50
wro D0
wio Dropaut
WHO DA+00
wo NN
wo RAG
W/o DQ
w/o Dropoul:
WrO DA
wio DA+D0
wi R4G
w/
wio
Wio
Figure 20: Ablation study of components based on edit distance on the test set (30 variables).
Figure 21: Ablation study of components with respect to running time (30 variables).
Distribution of Edit Distance by Configuration
Median Running Time by Configuration
4.5
5.0
Median Running Time
Idttiiaace
4.5
3,83
3
4,05
300
223.22
273
63
232.43
246.85
3,86
4,18
13
4.0
100
wio Dg
wio Dropaiit
wio AW
wio DA+DQ
wro RAG
w/o DQ
wo Drapout
WOO DA
wio DA+D0
wjo RAG
w/o
wDo
Wio
Figure 22: Ablation study of components based on edit distance on the test set (50 variables).
Figure 23: Ablation study of components with respect to running time (50 variables).
24

---

Published as a conference paper at ICLR 2025
Distribution of Terminals
Distribution of Terminals
124
124
Counts
113
105
104
103
94
Counts
105
104
38
35
27
19
ARG3
ARG3
AfGS
ARoZ
ARG4
ARG7
ARGa
ARGI
ARG6
ARGo
A313
ARGS
ARG
ARGa
ARGI
ARG9
Anoa
ARG7
ARGO
ARGZ
Figure 24: Distribution of terminals with 10 vari- ables.
Figure 25: Distribution of terminals with 10 vari-
ables and an imbalance ratio 0.25.
Edit Distance without RAG
Improvement on Edit Distance
Edit Distance by Imbalance Ratio
4.21
0.45
4.39
4,37
Distance
4.4
0.36
0,29
4.3
0.29
0.3
4.2
4.15
4.1
0.25
0.5
0.75
Imbalance
0.25
0.75
No Imbalance
Imbalance Ratio
Imbalance Ratio
Figure 26: Edit distance across different imbal- ance ratios without retrieval-augmenned genera- tion.
Performance improvement using retrieval-augmented generation across different imbalance ratios.
Figure 27:
K RETRIEVAL AUGMENTATION FOR IMBALANCED DATA
In Appendix J RAG has been shown to be particularly beneficial when handling large number of variables. One potential explanation is the increased data imbalance as the number variables grows. Specifically,, the frequency distribution of variables position 3 (starting from 0) for 10 and 50 variables is depicted in Figure 24 and Figure 28] respectively. With 10 variables, the most frequent variable appears 49 times and the least frequent appears 37 times, showing relatively balanced distribution. However, for 50 variables, the most frequent variable appears 26 times while the least frequent variable appears only times, indicating significant imbalance. This confirms that larger variable sets lead to more imbalanced data distributions.
To validate the effectiveness subsample half of the terminals with varying ratios to create imbalanced data for the case of 10 vari- ables. For training data with selected terminals at position 3, subsampling retained the data with probability equal the subsampling ratio The subsampling ratios are [0.25, 0.5, 0.75]. Figure 25 shows distribution after subsampling with a ratio of 0.25. The experimental results are shown in Figure 26 and Figure 27 As shown in Figure 26 the edit distance worsens as the subsampling ratio increases, confirming that imbalanced training data negatively impacts performance. The re- sults in Figure 27 demonstrate that the performance improvement from using RAG becomes more pronounced as the subsampling ratio increases. This indicates that RAG effectively mitigates the issues associated with imbalanced data, explaining why its impact is more significant when dealing with larger numbers of variables.
the RAG mechanism in handling imbalanced data, we randomly
L FURTHER ANALYSIS ON THE SYMBOLIC REGRESSION BENCHMARK
L.1 COMPARISON ON LOW-DIMENSIONAL BLACK-BoL Box DATASETS
In the domain of deep symbolic regression, low-dimensional subsets of black-box datasets are widely used evaluation criteria. Specifically, select all datasets from the PMLB benchmark with 10 or fewer variables and no categorical features (Kamienny et al.) 2022, Shojace et al. 2024a] Meidani et al. 2024). Figure shows the results under these criteria. Compared to F the rel-
25

---

Published as a conference paper at ICLR 2025
Distribution of Terminals
20
Counts
10
Figure 28: Distribution of functions and terminals in the case 50 variables.
R Test
Model Size
Training Time
RAG-SR
'PSTiee
'TPSR
-3C
'Operon
'FEAT
'SPPGGP
"EPLEX
X
GGPGGMEA
LGBM
Random
'ITAA
AdaBoosi
KemelRidge
AAP FE
*AFP
*FFX
1
'spleam
MLP
"DSR
X
'MRGP
Linear
"BSR
AIFeymman
-0.25 0.00
0.25
0.50
0.75
1.00
10'
10"
10
10"
10
10
10"
10"
Figure 29: R scores, model sizes, and training time of 25 algorithms on low-dimensional black-box regression problems.
ative rankings of the algorithms remain largely unchanged. Therefore, the conclusions in Section 4.2 remain valid in this case.
L.2 COMPARISON ON FEYNMAN AND STROGATZ DATASETS
Experimental Settings The Feynman and Strogatz datasets are widely used synthetic benchmarks for symbolic regression. Al-Feynman consists 119 datasets, while Strogatz contains 14 datasets. For RAG-SR, employ function relevant to the physics domain, including the operations + cos. The division operator implemented as protected division to handle edge cases, capping extremely small positive and negative denominators at 10- and -10- respec- tively. The training protocol follows the SRBench standard. All baseline algorithms use 10,000 subsampled training instances from the Al-Feynman datasets (La Cava al. 202T). However, our method subsamples only 500 instances from the Feynman datasets to reduce computational costs. No noise is added the labels. For the Strogatz datasets, we use available training instances. Following the definition in SRBench (La Cava et al. 2021), accuracy is defined as if the R²2 value exceeds 0.999, and otherwise.
Experimental Results The experimental results on the Feynman datasets are shown in Figure 30 Algorithms are ranked based on their test It2 scores on the Strogatz datasets. As shown, RAG- SR achieves the highest test I22 scores the Strogatz datasets, demonstrating the strong learning capability of the proposed method. On the Feynman datasets, RAG-SR ranks second, trailing only MRGP Armaldo et al. 2014). However, the model size of MRGP is more than order of magnitude larger than RAG-SR, while RAG-SR achieves good balance, In comparison to deep learning-bases based symbolic regression methods such TPSR (Shojaee al. 2024a) and SNIP (Meidani al. 2024
26

---

Published as a conference paper at ICLR 2025
Accuracy (%)
Training
Operon
SBP-GP
MRGP
-
GP-GOMEA
FEAT
TPSR
AFayymmn
Data Group
Faynman
Strogotz
AFP
EPLEX
BSR
ITEA
DSR
20
40
100 10"
10"
10
10'
10'
Figure 30: R2 scores, model sizes, and training time of 17 algorithms 119 Feynman and 14 Strogatz datasets.
RAG-SR outperforms them by large margin, further validating the effectiveness of leveraging neural semantic library in symbolic regression.
M PARAMETER SENSITIVITY ANALYSIS
M.1 INFLUENCE OF PNEURAL
P Pieural controls the at which the neural semantic library used generate new individuals. In this section, we analyze the influence of Paeural after evaluating 2000 solutions. Each probability value of P nneural was tested on 120 black-box datasets, with each experiment repeated 30 times to ensure stability. The results, shown in Figure 31 demonstrate that the test R² scores gradually improve as the probability of using the neural semantic library increases. Since neural generation is more cone-con than retrieval, the default value of 0.1, which corresponds to applying neural generation to one out of ten GP trees each solution during replacement, a reasonable choice that provides good balance between accuracy and efficiency. Nonetheless, in cases where GPU available to provide faster inference, a higher value Pneural can yield better results.
M.2 INFLUENCE OF NUMBER OF TREES
The number of trees determines the number of constructed features in each solution. RAG-SR fol- lows conventions in the evolutionary feature construction literature, using 10 as the default parameter. In this section, we compare three configurations for the number of trees in each solu- tion: 5, 10, and 15. To ensure stability, each configuration was tested on 120 black-box datasets, with each experiment repeated 30 times. The results after evaluating 2000 solutions are shown in Figure 32 The results indicate a substantial performance gap between using 10 trees and using trees. Several factors may contribute to this improvement. First, larger number of GP trees in each solution provides more candidates to be stored in the library, allowing neural generation and exact retrieval to identify better GP trees. Additionally, having more trees in each solution enables the base learner to more flexibly select appropriate features for making predictions, which may further enhance performance. However, increasing the number of trees also leads to increase in model size and evaluation time for each solution, as shown in Figure [33 and Figure 34 Therefore, setting the number of trees 10 strikes good balance between accuracy, model size, and training time.
27

---

Published as a conference paper at ICLR 2025
Test R² by p
Test R² by Number of Trees
0.927
0.93
0.920
0.925
0.920
0.917
Test
0.92
Test R2
0,900
0,907
0.91
0.847
0.850
0.90
0.0
0.05
0.1
0.2
0.5
15
Preurai
Number of Trees
Figure 31: Sensitivity of test R2 score to neurt
Figure 32: Sensitivity of test It score to number
of trees.
Model Size by Number of Trees
Training Time by Number of Trees
119-500
525 158
120
Training Time (s)
500
Model Size
100
81 127
400
60
300
44
627
225
10
15
10
15
Number of Trees
Number of Trees
Figure 33: Sensitivity of model size to number of Figure 34: Sensitivity of training time to number trees.
of trees.
N RELATED WORK
N.1 RAG FOR CODE GENERATION
The Aetrieval-Augmenttee Generation (RAG) technique has been widely used in the domain of code generation to enhance accuracy by retrieving code from open-source databases (Parvez et al. 2021 Wang et al. 2024). Similarly. RAG has also been applied to code transformation tasks, demonstrat- ing improved performance compared to zero-shot code translation (Bhattarai al.) 2024). These RAG-based techniques typically rely on existing repositories (Parvez et al. 2021]. In this paper, demonstrate that RAG is not restricted pre-existing code repositories but also be applied to code generated by evolutionary algorithms. Furthermore, we show that data augmentation can improve prediction accuracy when using RAG in conjunction with neural networks for symbolic expression generation. This suggests that, for code generation, employing synonym replacement or paraphrasing models to augment code summaries could further enhance the efficiency of RAG-based systems.
N.2 VQ- VAE FOR CODE GENERATION
In the program synthesis domain, Vector Quantized- Variational Autoencoder (VQ-VAE) (an Den Oord al., 2017) has been employed to learn discrete representations from input-output pairs, enabling the capture of high-level concepts (Hong et al.) (021).. Both VQ-VAE and RAG leverage continuous semantics retrieve discrete information for code generation. However, VQ-VAE uti- lizes an autoencoder to learn high-level conceptual representations from data, whereas RAG directly retrieves relevant information from a database. Compared to VQ-VAE, the RAG technique offers greater interpretability for end-users by allowing inspection of the retrieved information. Moreover, RAG provides an efficient mechanism to update knowledge without requiring retraining of the neu- ral network. Although there many differences between VQ-VAE and RAG, these techniques are not mutually exclusive. Integrating them the context of neural symbolic regression presents promising avenue for future research.
28

---

Published as a conference paper at ICLR 2025
Table 3: Examples of symbolic trees generated by the rerrieval-augmenned neural network when the ground truth exceeds the generation limit.
RAG-NN Generated Tree (Distancc)
Ground Truth Tree
Log(LoggUoo((AGG))) (3)
Abs(Log( Aog
Con(SinCCos/ARGG)) (3)
Log(Abs((C (absCoooSintSSSttCCooo/aaaa
Log(Abs(Log(ARGG)))C
Log(Abst Abst AbstSg (aggr///bbbssstpog/////71111111
SinSSqurc(Sin(ARCG8)))
LogiCos(ARGG8) (4)
Cos(Logg CogoSSiiCCCCoLLLggCCCAAAAC)))))
Sqr((qq((ARG4)) (4)
Sin(I Lgg Log (ggeSpgtt(LooogsStttt4441))))))
Log(Sin(ARGI)) (4)
Log(Neg (CCASSSppSSSSrtSSSS//Ae/)))))
Log(CosLog/ARGGS))
I
(4)
Log(Square( Log(Neg( ogt AQAARG7 ARGS))))
Syri(Sqpi(ARG2))
Syri(Neg( AbLLSSSprSSSStSSSSuruAAAAAAA
N.3 LARGE LANGUAGE MODELS FOR SYMBOLIC REGRESSION
Recently, large language models (LLMs) have been increasingly applied to symbolic regression tasks. These approaches often involve leveraging historical programs (Shojaee et al. 2024b) or extracting patterns from historical programs (Grayeli al. 2024) to guide LLM-based symbolic regression through prompt engineering. By instructing LLMs to learn from high-quality historical programs (Shojace al. 2024b) while avoiding low-quality ones (Grayeli et al. (024), these meth- ods aim to leverage LLMs to implement effective SR systems. However, using the MSE score as the criterion for crafting prompts can be misleading. Some expressions with higher MSE scores may still serve as valuable building blocks. From the perspective of the semantic space, the most useful expression for candidate solution
necessarily the one with the lowest MSE but the that effectively fills the residual gap. This insight highlights the need for fine fine-grained retrieval aug- mentation strategy that focuses on the semantics of each promising solution when crafting prompts. Thus, this paper builds a library of GP trees semantics and retrieves from this library to guide the language model in generating useful expressions for each promising solution.
O OUT-OF-LIMIT GENERATION
The neural retrieval library is trained on symbolic expressions with heights ranging between [0, 5). However, in an SR task, the ground truth expression may exceed the generation limit of the neural model. In this section, we evaluate the performance of the neural retrieval library when the ground truth surpasses this limit. Specifically, we generate GP trees with height of 6 and compare the outputs of RAG-NN with the ground truth, as shown in Table The results indicate that even when the ground truth exceeds the neural generation limit, the neural model can still generate meaningful sub-expressions that assist GP in discovering better GP trees.
29


Reviews:
[{"review_id": "a9akZQEvG8", "reviewer": "Reviewer_aRJ2", "summary": "The paper proposes a novel approach to symbolic regression based on neural-network based Retrieval Augmented Generation (RAG) and semantic descent (SD). The main idea is to dynamically construct a semantic library associating trees with their corresponding semantics (numerical evaluations of the tree). Semantic Descent starts from a set of initial trees and calculates the residual, i.e. the numerical error obtained from removing a specific tree from the original tree set. The goal of this step is to understand the contribution of the missing tree and replacing it with a new tree, possibly improving the residual. The proposal of a new tree is based on querying the semantic library with the residual and (optionally) refining such a tree with the prediction of a neural network conditioned on both the proposed tree from the semantic library and an appropriate encoding of the residual. A simple MLP is responsible for obtaining the latter encoding. A transformer encoder processes the tree extracted from the library and produces another embedding. The embeddings from the MLP and the transformer encoder are then concatenated and fed into a transformer decoder which in turn outputs the final proposed tree. The whole model is trained with online supervised learning. The library is dynamically updated with first-in-first-out queue mechanism. A data augmentation strategy is also proposed to improve the robustness of the model. Experiments are divided into two parts: first the model is evaluated on synthetic data and the various components of the model are ablated to quantify their relative importance and their contribution to the final running time; second the model is tested on the PLMB benchmark, resulting in very good results both in terms of accuracy and complexity.", "questions": "See weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- SR-RAG represents, to best of the reviewer knowledge, a novel method for symbolic regression combining techniques from the genetic programming literature and recent advances in neural networks. This combination is interesting as the two components have the potential to nicely address each other's limitations. In particular, the ablation studies seem to show that neural generation performs better than simple retrieval, suggesting that the neural network is benefitting the whole approach.\n\n- Results are encouraging: the model performs very well in terms of R^2 score and complexity.", "weaknesses": "- **Clarity of presentation could be improved**: I believe that the way the approach is presented should be made clearer. In particular, Fig. 1 is a bit hard to interpret and does not aid the reader understanding. In addition, I feel more details should be given on how the semantic library is populated and how the retrieval is performed given a residual.\n\n- **I feel the way the neural network is trained is not explained in sufficient detail**: While the way the different neural network components are described is well done, I feel important details on their training are missing. In particular, I do not understand how the cross-entropy loss is used as it is not clear to me what the ground truth labels are and where they are taken from. In addition, more details about the online supervised learning training routine should be reported. I feel this is very important to precisely understand how to model is trained and to have an intuition of the overall complexity of the approach. I would appreciate if the authors could clarify these points.\n\n- **Question on runtime**: RAG-SR performs relatively well in terms of training time, even though it incorporates neural networks and models are trained on CPU. I am wondering why possibly less complex models, exclusively based on GP techniques, perform comparably or worse in terms of runtime, despite not incorporating any neural network.\n\n- **Question on complexity**: RAG-SR seems to perform well in terms of complexity, achieving a good trade-off between accuracy and complexity. I am wondering where this property comes from. Why does the model has a preference toward simpler expression. At which point of the model is this constraint imposed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rXaS3zofeH", "reviewer": "Reviewer_e4Uw", "summary": "The paper proposes RAG-SR, a retrieval-augmented generation framework for symbolic regression (SR) that utilizes a neural semantic library, semantic descent, and scale-invariant data augmentation to generate symbolic models effectively without pre-training. Through various experiments, the paper demonstrates that RAG-SR outperforms existing SR models, especially those relying on pre-trained language models, by producing interpretable and accurate symbolic expressions.", "questions": "1. **Explain Hyperparameter Choices and Sensitivity**: Could the authors provide more insights into the choice of hyperparameters, such as the probability $\\( P_{\\text{neural}} \\)$ for neural generation versus retrieval, and the value of $\\( \\lambda \\)$ for the contrastive loss term? A sensitivity analysis on these hyperparameters would be valuable for practical deployment.\n\n2. **Computational Cost Analysis**: The results indicate increased computational costs with RAG-SR. Could the authors discuss how the computational demands scale with larger datasets? Are there specific optimizations for the semantic library or model architecture that could reduce inference times?\n\n3. **Ablation on Data Augmentation and Double Query**: Data augmentation and double querying are innovative and appear to significantly contribute to the model's effectiveness. However, would these techniques still benefit RAG-SR in less structured or non-SR tasks? Clarifying this point would highlight the broader applicability of the approach.\n\n4. **Comparison with Pre-trained Language Models**: While RAG-SR surpasses pre-trained models in SR tasks, does the method show robustness against varied functional distributions, especially in cases with substantial function overlap? A breakdown of results based on function novelty (seen vs. unseen functions) would deepen the insights into RAG-SR’s advantages.\n\n5. **Interpretable Complexity Metrics**: Interpretability is a key focus in SR, yet model complexity is only briefly addressed. How does RAG-SR ensure a balance between interpretability and accuracy, especially as model complexity scales? Including complexity metrics or visualizations would substantiate claims on interpretability.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Innovative Retrieval-Augmented Generation (RAG)**: The RAG mechanism effectively addresses the common issue of model hallucinations in SR tasks by incorporating relevant expressions from the semantic library. This approach is well-justified and experimentally validated, showing a significant positive impact on both the quality and relevance of generated models.\n\n2. **Effective Feature Construction through Semantic Descent**: The proposed semantic descent strategy is a notable contribution that improves model refinement through selective feature replacement rather than addition, which helps maintain a compact and interpretable feature set while boosting accuracy.\n\n3. **Data Augmentation and Double Query Strategy**: These techniques address the inherent challenges in SR by ensuring scale-invariance and robustness, with the double query strategy, in particular, effectively capturing symmetrical relationships in symbolic expressions.\n\n4. **Strong Empirical Validation**: The paper demonstrates that RAG-SR achieves state-of-the-art results across diverse benchmarks, reinforcing the proposed method's efficacy. The study includes a well-structured comparison with relevant baselines, supporting claims of improved performance and interpretability.", "weaknesses": "1. **Complexity and Training Time Trade-offs**: While RAG-SR is shown to improve SR accuracy, the additional computational overhead from maintaining and querying the semantic library may be a concern, particularly for larger datasets or real-time applications. It would be beneficial to discuss potential optimizations or alternative approaches to mitigate the added computational cost.\n\n2. **Limited Generalization to Non-SR Domains**: The focus on SR limits the method’s applicability to other machine learning tasks. A discussion on the adaptability of RAG-SR to different types of regression or generative tasks, particularly those that do not rely on symbolic expressions, would add value to the paper's impact on broader ML applications.\n\n3. **Dependence on Hyperparameter Tuning**: The effectiveness of RAG-SR, particularly in terms of balancing the retrieval and neural generation probabilities, appears sensitive to hyperparameter choices. The paper could benefit from including a sensitivity analysis or guidelines on setting these parameters in diverse settings.\n\n4. **Interpretability of Generated Models**: Although RAG-SR produces interpretable symbolic models, the paper could improve by further quantifying interpretability. For instance, how do the generated models compare in complexity or comprehensibility against baselines from a domain expert’s perspective?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qzh7AOc5rt", "reviewer": "Reviewer_b3Mt", "summary": "This paper introduces a novel approach to neural-guided symbolic regression. While prior work has focused on training generative models to directly produce new expression trees, such models often suffer from hallucination and struggle when faced with out-of-distribution data. In contrast, the authors propose training the generative model to produce a single feature at a time, rather than generating an entire expression tree. This approach aims to improve the model's performance on out-of-distribution problems. Additionally, the authors incorporate a prompting mechanism, feeding the model with relevant expression trees to help reduce hallucination.\n\nThe authors evaluate their method (RAG-SR) on a custom synthetic dataset and the no-noise version of SRBench. They find that RAG-SR outperforms other baselines on these task in terms of accuracy, size of the final equation, and speed.", "questions": "- L71: \"pretraining free supervised learning:\" I'm a bit confused by this phrase. Isn't pretraining, as used colloquially in NLP, a self-supervised learning strategy? Under this definition, all supervised learning algorithms should always be \"pretraining free.\" Consider replacing this phrase.\n- L103: Considering this paper has \"RAG\" in the title, it might be useful to have a section which introduces RAG, how it's relevant to symbolic regression / program synthesis problems, and how it's relevant to this algorithm. I suggest making a general section on \"Foundation Models for program induction.\" These papers (the RAG paper, RAG + code generation papers, and symbolic regression + foundation models) might be useful:\n  - https://arxiv.org/pdf/2406.14497: RAG Paper\n  - https://arxiv.org/pdf/2108.11601, https://arxiv.org/pdf/2407.19619 : RAG + Code generation\n  - https://arxiv.org/abs/2404.18400, https://arxiv.org/abs/2409.09359 : RAG + Foundation Models\n- L198: Is the depth and number of trees fixed? How does the algorithm scale when $m$ is increased/decreased? How is $m$ selected for the experiments (Synthetic benchmark and SRBench)?\n- L195-L209: \"SD focuses on replacing existing trees in the model with more informative ones...\"  Is the residual a tensor of real numbers or a single real number? Also, how is the difference measured? Regardless, it seems that calculating the residual requires evaluating the performance of the equation without the subterm to be replaced. For many domains where SR is used, evaluating the performance of a program is very expensive (eg: computational weather models).\n- L206: What is the definition of the semantics $\\psi(X)$? Is it a vector (e.g., $\\psi(X) \\in \\mathbb{R}^N$), and is it learned, or does it represent subtree performance on indices in $X$?\n- L210: How is P_neural computed? This isn't mentioned in the algorithm or in the following section. Is it a hyper-parameter?\n- L409: Please add error bars to the runtime evaluation experiments by averaging across different seeds.\n- L1177: Many SRBench ground-truth equations have depths greater than five. What do the failure modes on this experiment look like? What do the discovered equations look like? How do they compare to the ground truth equations (edit distance)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The general strategy of using a set of discrete latent codes for robustness is sound and widely acknowledged as beneficial in practical applications (e.g., VQ-VAE, discrete World Models, etc.).\n- The paper is well written with great figures.", "weaknesses": "- **Clarity**: Many related ideas exist within the broader community, and while they may not necessarily be weaknesses, these papers should be discussed in the related work section:\n\n  - VQ-VAE (https://arxiv.org/pdf/1711.00937): This paper from the representation learning community learns a discrete latent representation (a _codebook_). The authors might benefit from reviewing this work. I believe the manuscrpt is more closely tied to \"learning a discrete representation for neural guided symbolic regression\" rather than \"retrieval augmented generation.\" Regardless, I highly recommend adopting the nomenclature in this paper to increase comprehensiveness. This technique has also been applied in synthesizing general-purpose programs (https://arxiv.org/pdf/2012.00377), a comparision with this work would be useful.\n\n  -  Since this work is heavily influenced by work in retrieval augmented generation, it would be useful to include a discussion on how the authors think their work is connected/influenced by work in RAG (https://arxiv.org/pdf/2406.14497) and how it's different from current work that uses RAG for code generation (https://arxiv.org/pdf/2108.11601, https://arxiv.org/pdf/2407.19619).\n\n\nRegardless, I believe the authors have adequately demonstrated that this is novel and performant algorithm for symbolic regression. As such, I am in favor of accepting this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)