In-Context Symbolic Regression: Leveraging Large Language Models for Function Discovery (2024)

Abstract: State of the art Symbolic Regression (SR) methods currently build specialized models, while the application of Large Language Models (LLMs) remains largely unexplored. In this work, we introduce the first comprehensive framework that utilizes LLMs for the task of SR. We propose In-Context Symbolic Regression (ICSR), an SR method which iteratively refines a functional form with an LLM and determines its coefficients with an external optimizer. ICSR leverages LLMs’ strong mathematical prior both to propose an initial set of possible functions given the observations and to refine them based on their errors. Our findings reveal that LLMs are able to successfully find symbolic equations that fit the given data, matching or outperforming the overall performance of the best SR baselines on four popular benchmarks, while yielding simpler equations with better out of distribution generalization.

heading: 1 Introduction
text: Classical Machine Learning regression methods can be divided into two broad categories: statistical methods, which learn an implicit statistical (black-box) model of the relationship between the observations, and rule-based methods, which instead attempt to extract an explainable set of rules that explicitly model the transformation between the inputs and outputs (Lample and Charton, 2019). Symbolic Regression (SR) is a particular subset of the latter category, which searches the set of all possible explicit mathematical expressions to find the equation that best fits the given set of observations. This has the clear advantage of explainability, as well as a potential for better generalization, if the trend holds outside of the observed data.
The traditional approach for SR algorithms is Genetic Programming (Willis et al., 1997) (GP),
*Denotes equal contribution.
which combines fundamental blocks for mathematical expressions (e.g., basic operators, trigonometric functions, etc.) into more complex formulas using strategies borrowed from evolutionary biology, such as mutations and fitness. The recent success of Transformer models, first introduced by Vaswani et al. (2017), has revolutionized various fields of Artificial Intelligence, notably Natural Language Processing (Brown et al., 2020; Achiam et al., 2023; Touvron et al., 2023; Anil et al., 2023) and Computer Vision (Dosovitskiy et al., 2021). Transformer-based methods have also been proposed for SR (Biggio et al., 2021; Kamienny et al., 2022), typically by employing a model pre-trained on a large amount of synthetic SR datasets.
Large Language Models (LLMs), also based on the Transformer, have proven to possess unprecedented reasoning and generalization abilities, based on their capability for In-Context Learning (ICL) (Brown et al., 2020). This refers to the ability to perform tasks based on the context provided in the input text without any additional fine-tuning. With the help of ICL, these models can be leveraged for a wide range of different tasks, suggesting a potential use case for Symbolic Regression.
In this paper, we examine the integration of LLMs into the SR pipeline, with the aim of using them to search for new equations that could fit the data. Inspired by the Optimization by Prompting (OPRO) approach presented by Yang et al. (2023), we propose In-Context Symbolic Regression (ICSR)1. This approach leverages pre-trained language models by providing a number of previously tested equations and their fitness scores in the prompt, tasking them to generate a new candidate that could be a better fit. The method is repeated until convergence is reached or the computational budget is exhausted. To the best of our knowledge,
1We release the code at: https://github.com/merlerm/ In-Context-Symbolic-Regression.
589
only a contemporary work by Shojaee et al. (2024) has ever explored the use of LLMs for SR. However, they focus on working with equations from a scientific domain where natural language knowledge can be directly incorporated, while this work aims to generally explore the capabilities of LLMs for SR without any additional information, in order to lay a foundation that can be expanded later. We discuss in depth the differences between the two works in Section 2.
Our approach presents several advantages compared to models specifically trained for SR: as the LLM is not fine-tuned for this task, improvements in the underlying base model can improve ICSR without any changes to the method itself. Further, LLMs provide a natural language interface that can be leveraged to include additional information about the problem, like the domain of the equation and the interpretation of the observation values. The models could also be asked to explain the reasoning behind the proposed functions, potentially leading to a more interpretable process.
In summary, we make the following contributions: 1) We propose ICSR, the first general framework to leverage LLMs for the SR task. 2) We compare the method with a range of competitive SR baselines, matching or outperforming state of the
art results on four popular SR benchmarks: Nguyen (Nguyen et al., 2011), Constant (Li et al., 2023d), R (Krawiec and Pawlak, 2013) and Keijzer (Keijzer, 2003). 3) We show that the equations generated with our method tend to exhibit lower complexity, which correlates with stronger out of distribution performance.

heading: 2 Related Work
text: Symbolic Regression. GP has traditionally formed the backbone for SR methods (Smits and Kotanchek, 2005; Schmidt and Lipson, 2011; Virgolin et al., 2021). Typically, from an initial population, an iterative tournament is played where functions with the highest fitness are selected to ’reproduce’ with some random mutation, as in Koza and Poli (2005).
More recently, Deep Learning methods have been applied to enhance the available toolkit for SR. Udrescu and Tegmark (2020) proposed an iterative simplification of the problem relying on insights from physics and outsourcing the function approximation part to a neural network. Petersen et al. (2021) used a Recurrent Neural Network (RNN) with a risk-seeking policy to perform a hierarchical search over the space of user-defined operators and mathematical functions. The main drawback
of these methods, including GP, is the fact that the algorithms start from scratch for every new expression, with very limited abilities of knowledge preservation between tasks.
To address this limitation, numerous Transformer-based methods inspired by language modelling have been developed. SymbolicGPT by Valipour et al. (2021), NeSymReS by Biggio et al. (2021) and CL-SR by Li et al. (2023d) proposed different generative Transformer models specifically trained for SR. These models generate a functional form (’skeleton’) of the equation with a special token for coefficients which are fitted via an external numerical optimizer. Subsequently, Kamienny et al. (2022) presented E2E, a Transformer model able to produce the full expression including the coefficient values. While retaining knowledge between tasks, Transformer-based methods are quite limited in refining their solutions for the given set of points. To this end, Shojaee et al. (2023) presented a method integrating a pre-trained Transformer with Monte Carlo Tree Search to guide the equation generation merging the strength of the search and model pre-training. The proposed framework can be also viewed as a combination of a pre-trained model and an iterative refinement process. However, none of the prior methods employ a foundation model (Bommasani et al., 2021), such as an LLM, in order to leverage mathematical knowledge, but either pre-train an SR model (Biggio et al., 2021; Kamienny et al., 2022), or learn from scratch for every new function (Petersen et al., 2021).
Mathematical Reasoning with LLMs. As LLMs form the backbone of the method presented in this work, we rely entirely on their mathematical reasoning capabilities, such as ICL (Brown et al., 2020), to explore the solution space. Mirchandani et al. (2023) show that LLMs are able to recognize patterns from in-context examples and can extrapolate them to complete related tasks in the input. Similarly, Gruver et al. (2023) find that LLMs can extrapolate zero-shot the pattern from a timeseries (although they do not extract any functional representation). Furthermore, Fu et al. (2023) present a study in which they find that Transformer models can learn higher-order optimization methods (similar to Newton’s method).
Contemporary to our work, Shojaee et al. (2024) also propose to perform SR with an LLM aided by an external coefficient optimizer. However, they
focus exclusively on the case where LLMs can leverage scientific knowledge for SR, by including a description of the input and output variables in the LLM prompt. In contrast, we focus on the general case where no extra knowledge is given and test on standard benchmarks within the SR community and include a wider range of established baselines, with the aim to directly evaluate the capability of LLMs on the task of SR. Furthermore, we propose advancements in the structure of the prompt, including the coordinates of the points to be regressed, the score of previous attempts, and more in-context examples. Finally, rather than asking the LLM to optimize a Mean Squared Error (MSE) objective, we employ a more advanced loss function, presented in Section 4, which jointly optimizes for the accuracy and complexity of the function for improved generalization properties.

heading: 3 Background
text: The Optimization by Prompting (OPRO) framework was introduced by Yang et al. (2023) for prompt optimization, i.e., for increasing the performance of models (such as LLMs) that receive a textual prompt in the input and have to perform a specific task (such as mathematical reasoning). Closer to our interest, the authors also present experiments on classical optimization problems (Linear Regression and Travelling Salesman Problem), suggesting that OPRO can solve such tasks.
The key idea of the method is the use of a socalled meta-prompt, a higher level prompt which contains a description of the task to be optimized and previous attempts (examples) in solving it with their corresponding scores. An example of such task can be querying the model to find a linear function that fits a set of points. In this case, the prompt is augmented by the functions that have been tried out and the mean squared error on the data, obtained with an external evaluation procedure. The assumption behind it is that LLMs have the ability to extrapolate the pattern formed by the examples, thanks to ICL, and propose a better alternative. The meta-prompt is given as input to the LLM and the model’s output is then evaluated and added back to the meta-prompt if the score is good enough. This approach can then be iterated until a satisfying result is achieved or a certain computational budget is exhausted.

heading: 4 Method
text: We consider a regression dataset D with N observations and target variables {xi, yi}Ni=1, also denoted with (X,Y ) more compactly, to be used for producing a function f̂ that well approximates the data. To leverage the OPRO approach for SR, we need to design a meta-prompt suitable for the task and fill it with the available observations (X,Y ), an initial set of k functions F̂0 = {f̂ (1)0 , f̂ (2) 0 , . . . , f̂ (k) 0 } (either hand-written or model-generated) and a measure of their fitness (score) on D. For our purposes, we frame the refinement process as a minimization problem over an objective function, also called the error function, such that generated equations with the lowest error have the highest fitness. The goal is then to iteratively refine the set of functions F̂i, i ∈ [1, . . . , n] for each iteration i, until a sufficiently low error is obtained by one of them or a maximum number of iterations is reached; we denote this process as the optimization loop. Due to the finite size of the LLM context window, we only keep the k best performing previous attempts in the set F̂i, where k is a design choice (k = 5 in this study). The meta-prompt used in the experiment can be found in Appendix C.
Seed Functions. At the first iteration, F̂0 is empty as there are no previous guesses from the model. Thus, an initial population of seed functions is required to kickstart the optimization loop. Instead of relying on a fixed set of initial functions, which could be restrictive in general, we ask the model to generate the initial seed functions (with the prompt provided in Appendix C). This results in a complex and diverse set of functions, from which the LLM can refine its future predictions with the optimization loop. In our implementation we repeat this initial process ns times, as some of the generated functions can be undefined for certain input points (e.g., log(x) for negative numbers). We set ns = 10 for this work and explore its impact in Section 5.5 through an ablation study.
Error Function. The immediate choice for the objective function would be an MSE, or a similar error metric, over the regression dataset D. However, simply minimizing this error can result in overfitting on the training points in D. As overfitting in SR often occurs due to a growing number of terms in the generated equation, we adapt from Shojaee et al. (2023) a fitness function r(f̂ |D) with an extra penalty term for the complexity C of the
generated expression, defined as the following:
r(f̂ |D) = 1 1 + NMSE(f̂ |D) + λe
( −C(f̂)
L ) , (1)
where f̂ is the predicted function, C is the complexity defined as the number of nodes in the expression tree, L is the maximum sequence length (set to 30), and λ is a hyperparameter to trade-off between the fit to the data and the complexity. The Normalized Mean Square Error (NMSE) is calculated as
NMSE(f̂ |D) = ∑N
i=1(yi − f̂(xi))2∑N i=1 y 2 i + ϵ , (2)
where ϵ is a small regularizing constant. Finally, we use err(f̂ |D) = r(f̂ |D)−1 as our error function to frame ICSR as a minimization problem. We explore the choice of the λ parameter in Section 5.5 with a sensitivity analysis.
Parameter Fitting. We utilize the LLM only to generate functional forms (skeletons), while the unknown coefficients associated to the predicted functional form are optimized by Non-linear Least Squares (NLS) (Kelley, 1999) available from SciPy’s (Virtanen et al., 2020). This not only yields better coefficient values, due to the superior optimization performance of NLS over LLMs, but also allows for more efficient exploration of the space of functions, by grouping them in equivalence classes of unique functional forms. In our implementation, we optimize the function’s coefficients five times starting from different random initial values, to avoid local minima, similarly to Li et al. (2023d).
For other details about the OPRO implementation, we follow the original work. Specifically, we also sample multiple functions for every iteration (asking the model to generate 5 functions for every call) in an attempt to improve the stability of the loop and we experiment with a decreasing temperature parameter to balance exploration/exploitation (with a higher initial temperature encouraging the exploration of the underlying functional space, and a lower temperature at the later stages forcing smaller tweaks to the trajectory). To avoid saturating the model’s context window, we limit the amount of training points that are included in written form to a certain threshold, empirically set to 40. We discuss this in more detail in the Limitations Section 6.1.

heading: 5 Experiments
text: We empirically evaluate ICSR and compare it against a set of competitive baselines, checking both in-domain and out of distribution performance of the proposed approach.

heading: 5.1 Benchmarks
text: For our experiments, we choose four popular SR benchmarks containing functions with one or two input dimensions: Nguyen (Nguyen et al., 2011), Constant (a modified version of some of the Nguyen equations with different numerical values for the coefficients (Li et al., 2023d)), R (Krawiec and Pawlak, 2013) and Keijzer (Keijzer, 2003). The symbolic equations and ranges for both the training and testing points are reported in Appendix D. We leave for future work the evaluation of ICSR on higher dimensionality benchmarks.

heading: 5.1.1 Metrics
text: While we use the error function err(f̂ |D) during the optimization loop (see Section 4), we follow the literature in reporting the coefficient of determination R2 (Glantz et al., 2017) to evaluate the quality of our method. This staple metric in SR can be interpreted as follows: a function will get a positive score if it is more accurate than the average prediction and will get a score of 1 for a perfect prediction. The coefficient is computed as:
R2 = 1− ∑n
i=1(yi − ŷi)2∑n i=1(yi − ȳ)2 , (3)
where yi is the ground truth value, ŷi is the predicted value and ȳ is the average of all yi.
We report the R2 metric computed on a set of unseen testing points, obtained from a dense grid
within the same range of input point values as during training (for in-domain performance) or its extended version (for out of distribution performance). We follow Li et al. (2023d) and Biggio et al. (2021) in removing the 5% worst predictions in all methods to ensure robustness against outliers. We further report the complexity C of the generated equations, calculated as the number of nodes in its expression tree. For all methods, we repeat all experiments across five different random seeds and report the average values together with the standard error of the mean. For ICSR, we allow up to 50 iterations in the optimization loop and end it earlier if the R2 score on the training set exceeds 0.99999.

heading: 5.2 Baselines
text: To evaluate the performance of the proposed method we opted for the following list of competitive baselines: gplearn (Stephens, 2022), a classical GP approach; DSR (Glatt et al., 2022) and uDSR (Landajuela et al., 2022), two searchbased methods; NeSymReS (Biggio et al., 2021) and E2E (Kamienny et al., 2022), selected as representatives for Transformer-based model pre-trained over a large-scale SR dataset; and TPSR (Shojaee et al., 2023), which augments E2E with a decoding strategy guided by Monte-Carlo Tree Search, as an efficient combination of pre-training and search. The details of the baseline model and the hyperparameters can be found in Appendix B.
We compare various properties of the considered methods in Table 1. Thanks to the use of LLMs, ICSR is able to leverage a much larger model size without the need for SR-specific training examples, as opposed to the other Transformer based methods. Furthermore, our method is far more sample
efficient when compared to search-based methods like DSR and uDSR. The LLM is slower in generating a single expression, but is able to produce more meaningful equations, thanks to the large pretraining bias, as opposed to methods like gplearn and DSR which have to be trained from scratch on each problem. ICSR is also the only method with a natural language interface and a flexible vocabulary, which we discuss further in Section 6.

heading: 5.3 Comparison across Baselines
text: For comparison of ICSR with the baselines, we choose Llama 3 8B (Meta, 2024) as the underlying LLM. The results (see Table 2) show that the ICSR approach is very robust, consistently achieving very high scores across all benchmarks while producing expressions with a lower average complexity. The overall average columns show that ICSR outperforms all baselines, with only uDSR matching its R2 score at the cost of significantly higher complexity. In general, it is important to consider both metrics simultaneously, as simpler functions can lead to a sightly lower R2 value while bringing other advantages, such as better out of distribution generalization, which we explore in Section 5.4. As seen in the headers in Table 2, the complexity values of the ground truth equations align much more closely to the ones recovered by ICSR as opposed to the ones for other high-performing baselines, such as uDSR or TPSR. The improvement in complexity compared to TPSR is particularly noteworthy, as both methods are using the same objective function: this could be a sign that LLMs tend to produce more human-readable expressions thanks to their pre-training bias. It is also worth noting that ICSR can potentially improve over time by simply increasing the performance of the underlying LLM backbone without any additional training, while that is not the case for the other methods.

heading: 5.4 Out of Distribution Performance
text: We further explore the advantage of producing functions with a lower complexity value by testing the out of distribution capabilities of the expressions recovered by ICSR and the other baselines. We exclude gplearn from these experiments, as we observe its performance to be significantly lower compared to the rest of the methods. To include out-ofdomain test points, we extend the input range by 100% to all directions (in which the function is defined). We compute the R2 value on the extended range, reporting the results in Figure 2. Note that the R2 value can quickly become increasingly negative when the functions diverge significantly. In order to keep the results stable, we treat all negative values as 0 when computing the average and report the fraction of experiments with a negative R2.
In general, we observe a sharp decline in performance for all methods, with the fraction of negative R2 values quickly increasing towards the further extensions of the range. Specifically, ICSR is the highest performing method in the 175% and 200% domain increases, with the second lowest and lowest number of failures respectively. Generally, methods with lower complexity such as NeSymReS, E2E and TPSR tend to perform better than uDSR, with the exception of DSR which exhibits the poorest out of distribution performance even with a low average complexity. The comparison between ICSR and uDSR is particularly meaningful: as reported in Table 2, the two methods are tied for the best overall average performance, but ICSR outperforms uDSR when extrapolating further outside
of the training range thanks to the lower complexity of the recovered expressions. We present some qualitative examples that demonstrate the difference between the methods in Figure 3.

heading: 5.5 Sensitivity Analysis and Ablation Studies
text: In this section we first investigate the impact of the λ parameter and then test the importance of the iterative refinement of the equations with the optimization loop. Finally, we compare ICSR with a baseline where the LLM was not given any information about the observations. All results are reported in Table 3.
Lambda Parameter. In our sensitivity analysis we considered the complexity penalty parameter λ = [0, 0.05, 0.1, 0.5, 1]. We noticed that the smallest penalty λ = 0.05 was already sufficient to considerably reduce the complexity of the selected functions and increasing the penalty further had a relatively smaller impact on complexity. Therefore we used λ = 0.05 for our experiments. With λ = 0 the complexity is not considered and the equations overfit on the observations: the R2 score tends to improve slightly at the cost of a large increase in complexity, with expressions composed of many different terms attempting to fit perfectly the training set. As the value for the parameter λ increases, both the R2 score and the complexity tend to decrease, resulting in equations that underfit the data, as they do not have enough terms to properly capture all the observed dependencies. These results align with Shojaee et al. (2023), who introduced the fitness function we use. They chose 0.1 as the
final parameter value, which we find performing similarly to 0.05, but slightly underfitting on some benchmarks, particularly R.
Optimization Loop. The results suggest that the seed functions generation step plays a key role in our approach, as with ns = 10 the results already show a high fitness on the test set, although they still underperform the full method in terms of both R2 and complexity. We notice that using the best seed functions without refinement can outperform the results with ICSR for some values of λ (e.g. λ = 0, 0.1) in the most complex benchmarks (R and Keijzer). This is because the performance is reported on the set of test points and can decrease when refining, due to overfitting on the training points. It’s also worth noting that some of the experiments with only a single initial call did not result in any valid seed functions, showing the need for repeating the generation process multiple times. In the prompt used to generate the seed functions (reported in Appendix C) we specifically ask for a diverse and complex set of functions that can be optimized, which is likely why the complexity on the seed functions is much higher, as it will be lowered later in the optimization loop. Overall, both parts of the method are necessary for the best possible performance; repeating the seed function generation step multiple times allows the model to generate a large number of potential initial expressions, resulting in a solid set of initial candidates for the optimization loop to build upon.
Random Guessing. As some of the benchmarks contain common equations such as simple polynomials, the LLM could simply be randomly generating functions that fit the data points, instead of actually making use of the information provided in the prompt. To ensure that this is not the case, we compare ICSR with a ’Random Guessing’ baseline, where the LLM was prompted for 60 times (matching the budget used for ICSR, which uses 10 prompts to generate the seed functions and 50 prompts for the optimization loop) to generate five random functions, without any information about the observations or previous guesses (the prompt is reported in Appendix C). The results show that this baseline underperforms ICSR on all four benchmarks, especially on Keijzer, the hardest one. Empirically, we observe that the functions generated by the LLM in this way are all extremely simple, mostly constrained to basic polynomials. This confirms that LLMs are able to extract patterns from
the prompt and are not simply randomly generating the solutions.

heading: 6 Discussion
text: Optimizing for out of distribution. A general framework for optimizing the out of distribution performance of a predictive model (such as a symbolic equation) is to regularise its complexity, following the Occam’s Razor principle that simpler explanations are preferable to more complex ones, all other things being equal. In our work we use the working definition of complexity as the number of nodes in the expression tree of an equation. However, more optimal choices could be available: for instance, equations containing expressions not defined on all the real domain (such as logarithms and square roots) could be penalised more, as they could be undefined when extrapolating to larger domains. Knowing in advance the full domain in which an equation is supposed to hold could also greatly improve out of distribution performance by filtering out invalid candidate functions. In the case of ICSR, it could also be leveraged as extra information by the LLM. Furthermore, we observe that numerous equations that we derive with ICSR have extra terms with very small coefficients (e.g. O(10−3)) that do not contribute significantly to the shape of the equation and could be safely suppressed, resulting in expressions with a lower complexity. This could be done by modifying the optimization procedure of the coefficients, to eliminate coefficients under a certain threshold, which would be a hyperparameter of the method.
Vocabulary. In general, most SR methods are limited to a predefined vocabulary of operators and tokens, while LLMs can virtually explore any possible function and combination. An example of this is with the xx21 function in the Nguyen benchmark: in Biggio et al. (2021), the authors mention that it is not included in the set of equations that their model can fit, while our approach can recover the exact expression. We also observe a similar trend with the other baselines for this specific expression. In our prompts (see Appendix C) we include a vocabulary for the LLM, but this is meant more to guide the LLM into the correct search space and is by no means a hard restriction: for example, we observe that ICSR can produce the erf function even if it wasn’t reported in this list. Furthermore, any function that can be found in the model’s pre-training corpus (fundamentally the In-
ternet) can be potentially added to the prompt at any time if desired, which is impossible for other fixed-vocabulary methods.

heading: 6.1 Limitations
text: Although promising, the approach presented in this work still suffers from some key limitations that hold back its potential as a full Symbolic Regression method.
Size of the context window. LLMs are provided with a context window, which represents the maximum number of tokens they can process as input at the same time. For instance, Llama3, used for ICSR, has an 8k token context window. This limits the amount of information that we can include in the prompt, in terms of training datapoints and previously attempted functions with their errors. However, with context-window size increasing, commercially available LLMs like GPT-4 Turbo (Achiam et al., 2023) and Claude 3 (Anthropic, 2024), which process over 100k tokens, this issue is likely to be alleviated or lifted completely.
What to include in the prompt? Including all needed information in the prompt might not be enough, as some research suggests LLMs cannot fully utilize extended contexts (Liu et al., 2024c). In practice, we observe that when too many points are included, the model often continues generating points, especially with two-dimensional functions. Limiting training points in the prompt to 40 (chosen empirically) helps, while all input points are still used for coefficient optimization. Some directions to help the model leveraging the information in the data could be to sample the most informative subset of points to fit in the prompt, or present the LLM with higher-level descriptions of the points, rather than feeding them directly to the model. Finally, we hypothesize that presenting the data in different modalities, such as images of the points and plots of the functions, by using multimodal foundation models, might be helpful to incorporate all information available. We experimented with Vision-Language Models, but our attempts in that direction, reported in Section A of the Appendix, were not fruitful so far.
Dimensionality. Using an LLM for higher dimensional inputs is possible, but dimensionality exacerbates the issues presented above. As the number of variables grows, so does the space dedicated to the input points in the prompt, which
will naturally confuse the model and obfuscate the structure in the datapoints even further. Specifically fine-tuning an LLM on this kind of examples might show some improvement, but scaling this approach for higher dimensional problems remains a challenge.

heading: 7 Conclusion
text: We show that LLMs paired with the ICSR approach are able to perform Symbolic Regression tasks on classical SR benchmarks. The proposed method matches or outperforms a variety of established SR baselines, while producing simpler expressions that more closely resemble the complexity of the ground truth equations and result in better out of distribution performance. This work exposes yet another task that LLMs can be leveraged for, thanks to specialized techniques such as ICSR, and shows promise for integrating these models with mathematical reasoning methods.

heading: 7.1 Future Work
text: As this is one of the first works published on this topic, much work remains to be done. LLMs allow the inclusion of domain-specific natural language information into the prompt, as explored by Shojaee et al. (2024). The natural language interface could be further exploited by employing explicit Chain of Thought-like (Wei et al., 2022; Kojima et al., 2022) techniques, allowing the model to output even more well-informed guesses at every step and resulting in an interpretable method. Another interesting direction would be to consider tree-based search algorithms on top of the LLM, analogously to the TPSR (Shojaee et al., 2023) approach. As our work proves the intrinsic ability of LLMs to perform SR without taking into consideration any additional inputs, we have hope that future work can build upon ICSR to further leverage foundation models for SR.

heading: Acknowledgements
text: We are grateful to Alexander Ilin and Alberto Zabeo for the fruitful discussions. We thank AaltoIT (IT Services of Aalto University, Finland) for provided support with computational resources. This work was supported by the Research Council of Finland (Flagship programme: Finnish Center for Artificial Intelligence FCAI, and grants 352986, 358246) and EU (H2020 grant 101016775 and NextGenerationEU).

heading: B Hyperparameters
text: We report the hyperparameters used with LLMs (Table 5). As reported in the main text, for ICSR we sample ns = 10 initial seed functions and repeat the optimization loop for 50 iterations, using an acceptance threshold of 0.99999 and repeating the coefficient fitting for 5 times with different initializations. For DSR and uDSR we set the computation budget for the number of expressions to evaluate to 200K and extend the vocabulary as {add, sub, mul, div, sin, cos, exp, log, sqrt, n2, abs, n3, n4} and {add, sub, mul, div, sin, cos, exp, log, sqrt, abs, poly} correspondingly. For the NeSymRes model we evaluate the model checkpoint that has been obtained with the training set of 100M expression skeletons. The actual number of the equations in the training set is even larger since the values for the coefficients are resampled on each training batch. The beam size in NeSymRes is set to 10 and the number of restarts for the external coefficient optimizer is 10, while for E2E model the beam size is 100 but the coefficient optimizer is applied just once. E2E doesn’t benefit from restarting the external coefficient optimizer as much since E2E predicts the whole equation including the values of the coefficients. The predicted coefficients can be further improved by numerical optimizer but they serve as good initial values. For all other implementation details, we follow the default hyperparameters provided in the following repositories: gplearn2, DSR/uDSR3, NeSymReS4 and E2E/TPSR5. 2https://github.com/trevorstephens/gplearn 3https://github.com/dso-org/ deep-symbolic-optimization 4https://github.com/SymposiumOrganization/ NeuralSymbolicRegressionThatScales 5https://github.com/deep-symbolic-mathematics/ TPSR

heading: C Prompts
text: The prompt used to generate the seed functions is reported in Figure 5, while the prompt used during the optimization loop is reported in Figure 6 and the one used for the random guessing baseline is reported in Figure 7. For the ICSR-V extension presented in Appendix A we add a brief description of the provided plots as well as the image.

heading: D Benchmark functions
text: The list of functions and point ranges for all the benchmarks can be found in Table 6. The range for training and testing points was taken from the original source where available. Nguyen and Constant do not include a range for the testing points, so we used the same range as the training points but with more sample points. U[min, max, num] indicates points randomly sampled from a uniform distribution between the min and max values, while [min, max, num] indicates a range of equispaced points from min to max. The training points are sampled from U[min, max, num] once and then kept fixed across the random seeds and all tested methods to ensure consistency.

heading: E Sample results
text: We present a sample of one solution for each function in the benchmarks found by our method, to qualitatively investigate the generated expressions. The observations are seen in blue, the true function is seen in red and the model’s guess is seen in green (Figures 8, 9, and 10 and 11). Some of the failures of the models are apparent: in areas where there is a low density of training points the model sometimes makes guesses that ignore the overall trend, as seen, for example, in the R3 equation (Figure 10). The Keijzer benchmark is also much harder in the last 5 equations, with only 20 randomly sampled points to cover a complex 2D space, which can lead to some failures (e.g., in Keijzer 14).

heading: Future Work
text: As our work proves the intrinsic ability of LLMs to perform SR without taking into consideration any additional inputs, we have hope that future work can build upon ICSR to further leverage foundation models for SR