SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training
Authors: Kazem Meidani
Parshin Shojaee
Chandan K. Reddy
Amir Barati Farimani
Venue: ICLR 2024 spotlight (2024)
Keywords: Symbolic Mathematics
Pre-training
Transformers
Symbolic Regression
Deep Learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We introduce a multi-modal pre-training framework to enable mutual understanding between mathematical symbolic expressions and their numeric counterparts.

Abstract: In an era where symbolic mathematical equations are indispensable for modeling complex natural phenomena, scientific inquiry often involves collecting observations and translating them into mathematical expressions. Recently, deep learning has emerged as a powerful tool for extracting insights from data. However, existing models typically specialize in either numeric or symbolic domains, and are usually trained in a supervised manner tailored to specific tasks. This approach neglects the substantial benefits that could arise from a task-agnostic multi-modal understanding between symbolic equations and their numeric counterparts. To bridge the gap, we introduce SNIP, a Symbolic-Numeric Integrated Pre-training model, which employs contrastive learning between symbolic and numeric domains, enhancing their mutual similarities in the embeddings. By performing latent space analysis, we observe that SNIP provides cross-domain insights into the representations, revealing that symbolic supervision enhances the embeddings of numeric data and vice versa. We evaluate SNIP across diverse tasks, including symbolic-to-numeric mathematical property prediction and numeric-to-symbolic equation discovery, commonly known as symbolic regression. Results show that SNIP effectively transfers to various tasks, consistently outperforming fully supervised baselines and competing strongly with established task-specific methods, especially in the low data regime scenarios where available data is limited.

Published as a conference paper at ICLR 2024
SNIP: BRIDGING MATHEMATICAL SYMBOLIC AND NUMERIC REALMS WITH UNIFIED PRE-TNAINING
Kazem Meidani" Parshin Shojaee* 2 Chandan K. Reddy 2 Amir Barati Farimani 1,3
1 Department of Mechanical Engincering, Carnegie Mellon University
2 Department of Computer Science, Virginia Tech
3 Machine Learning Department, Carnegie Mellon University
ABSTRACT
In an where symbolic mathematical equations are indispensable for modeling complex natural phenomena, scientific inquiry often involves collecting observa- tions and translating them into mathematical expressions. Recently, deep learning has emerged as a powerful tool for extracting insights from data. However, ex- isting models typically specialize in either numeric or symbolic domains, and are usually trained in supervised manner tailored to specific tasks, This approach neglects the substantial benefits that could arise from task-agnostic modal understanding between symbolic equations and their numeric counterparts. To bridge the gap, we introduce SNIP, a Symbolic-Numert Integrated Pre-training model, which employs contrastive learning between symbolic and numeric do- mains, enhancing their mutual similarities in the embeddings. By performing latent space analysis, observe that SNIP provides cross-domain insights into the representations, revealing that symbolic supervision enhances the embeddings of numeric data and vice versa. We evaluate SNIP across diverse tasks, including symbolic-tor nommmrii- mathematical property prediction and numeric-to-symboli equation discovery, commonly known symbolic regression. Results show that SNIP effectively transfers to various tasks, consistently outperforming fully su- pervised baselines and competing strongly with established task-specific methods, especially in the low data regime scenarios where available data is limited
INTRODUCTION
Throughout the history of science, symbolic mathematics has been unreasonably effective in rep- resenting natural phenomena (Wigner 1960). Complex patterns of natural systems, represented as numeric data observations, can be elegantly abstracted using mathematical formulas. Mathematical symbolism has given us the language to describe, understand, and predict the natural world. The challenge of bridging the gap between the numeric observations and their mathematical symbolic representations has been consistent focus in many scientific and engineering domains. Recogniz- ing and exploring this connection crucial, as it promises to drive advancements in various fields.
In recent years, deep learning has demonstrated promising capabilities in learning from symbolic mathematics language as well as extracting insights from numeric data observations. Transformer models (Vaswani et al., (017), in particular, have emerged as frontrunners in this field, effectively capturing patterns within mathematical expressions and solving complex tasks such as differential equations and function integration (Lample & Charton 2020: Welleck et al. (022). However, these models, while powerful, are not inherently designed to handle numeric data inputs. While some pre- trained symbolic regression models have been introduced map numeric datasets to their governing mathematical expressions in a supervised manner (Biggio et al. 2021 Kamienny et al., (022), gap still remains in developing a task-agnostic pre-training model capable of mutual understanding between the modalities of symbolic mathematical equations and their corresponding numeric data.
Multi-modal pre-training models, exemplified by groundbreaking models like Contrastive Language-Image Pre-training (CLIP) (Radford et al. 2021), have found a significant place in the
"qqual Contribution. Contact email: mmeidani Ganiaamm@ddnreee.mm.dddu
Coode and model are available at: https://github.com//cc s-mbolic-mmmhehheeeeeeee waw.mmmmmaaatii... cs/
Mu i modal - -Math -athh-rttraaiiinggg

---

Published as a conference paper at ICLR 2024
deep learning landscape. CLIP has particularly set new standards in vision-language tasks, bridg- ing the understanding between visual content and natural language descriptions. Expanding beyond traditional vision- Laiionla domains, recent studies have broadened multi-modal pre-training to in-
clude other modalities, such as audio and tabular data (Liu et al. 2021 Zhang et al. 2023: Hager et al., (223). Additionally, previously untouched scientific domains, like molecular representation, are also benefiting from these advancements in multi-modal representations (Su et al. 2022 Cao
al. (023). Nevertheless, the symbolic-numette domain remains relatively unexplored. Consider- ing the foundational role of symbolic mathematics in science and the ubiquity of numeric data, an in-depth exploration of their mutual learning is not only timely but essential.
In this work, we present Symbolic- -ummeric Integrated Pre-training (SNIP) to connect the two often distinct worlds of symbolic mathematical expressions and their corresponding numeric manifesta- tions. The architecture of SNIP, depicted in Fig. [ incorporates dual Transformer encoders, with each encoder dedicated to learning the symbolic or numeric representations of mathematical func- tions. Subsequently, mask-agnosti contrastive objective is employed to enhance the similarity be- tween (symbolic, numeric) pairs of data. The multi-modal pre-training of SNIP provides capabilities to understand and generate cross-modal content. Our experiments show that SNIP achieves remark- able performance in cross-modal mathematical understanding and prediction tasks. Additionally, by combining SNIP with an equation generation decoder and exploiting its interpolatable latent space, we can effectively harness SNIP's mutual knowledge for the task of numeric-to-symboot equation discovery (known as symbolic regression), achieving competitive results with state-of-the-art base- lines. The major contributions of this work can be summarized as follows:
Proposing SNIP, a pioneering pre-training method that integrates mathematical symbolic and nu- meric domains through joint representation learning. This approach captures mutual relationships, delivering embeddings that are informed and enhanced by both domains.
Evaluating SNIP in cross-modal comprehension across different mathematical property prediction tasks. Our results indicate that SNIP outperforms the fully supervised baselines, particularly in low data regime scenarios. Visualizing the latent embeddings also confirms that SNIP's pre- trained representations reveal patterns linked to these cross cross-modal mathematical properties.
Leveraging SNIP for numeric-to-s toymmboli equation generation task, commonly known as sym- bolic regression. In this task, after training an expression generation decoder on top of SNIP's numeric encoder, we exploit the high-quality semantic within SNIP's continuous and low- dimensional latent representations to perform latent space optimization with the objective of find- ing equations with balanced accuracy-complexin Results show that SNIP achieves state-of-the- art performance on the well-known SRBench (La Cava et al. 2021 benchmark.
2 RELATED WORK
Large-scale Pre-training. Our work is built upon an extensive body of research advocating the advantages of pre-training large models on large datasets (Zhou et al. 2023] Zong et al., (023). Initially, pre-training was single-modal, with self-supervised learning (SSL.) as key paradigm that used data as its own supervision, especially useful where labeled data was limited (Balestriero et al.) 2023). This paved the way for the emergence of multi-modal pre-training where models are trained to understand relationships across different modalities (Wang et al. 2023). ). Vision and language have traditionally played the two main characters of pre-training models. For instance, CLIP (Radford) et al. 2021), ALIGN QJa et al. 202T). I). and FLAVA (Singh et al. 2022) utilize image-caption pairs to construct jointly learned embedding spaces. These models are trained to align the embeddings of corresponding image-caption pairs while distancing unrelated pairs. The success of multi-modal pre-training in vision and language spurred its adoption in other domains. For example, recent works have extended this approach to videos, audio, and even tabular data (Liu et al.) 102 Dong al., 2022. Hager et al. 2023). Specialized scientific domains have also embraced this paradigm. For instance, different models have emerged to learn joint representations of molecules (Su et al.) 2022 et al. (023). Our work introduces a fresh perspective, intertwining symbolic mathematics with numeric observations. To this end, we use multi-modal pre-training's 's potential to deepen the symbolic-numeric mutual understanding.
Deep Symbolic Mathematics. Recently, deep learning models have made significant performance in the field of mathematical reasoning (Saxton et al., 2019 Lu et al. 2023). The Transformer mod- els, originally designed for NLP tasks (Vaswani et al. 2017), ,, have been repurposed with remarkable success in the domain of symbolic mathematics. It has powered models that can integrate func-
tions (Lample& & Charton 2020 Welleck et al. 2022 prove mathematical theorems (Lample al.)
2

---

Published as a conference paper at ICLR 2024
Hrreo) suci-a ba. #RW Mi
Symbelic Expressions
Profie
Tokenice
Fosttonn embeddiing
Symbolic
Encoder
Numeric
Observations
Tohustion
Embeadder
Numeric Encodar
Figure 1: The SNIP Framework: schematic representation the dual-encoder pre-training scheme for mutual learning between symbolic equations and their numerical observations. Both symbolic and numeric encoders work in tandem, capturing the paired similaritics and essence of their respective modalities.
2022). and perform numerical calculations, such as arithmetic operations (Charton 2022 Jelassi al. 2023). These achievements underscore the flexibility and potential of deep learning models in abstract reasoning. Beyond pure symbolic reasoning, there is also a growing interest in supplement- ing these models with numerical knowledge for improved mathematical understanding. For exam- ple, recent works have studied to enhance language models with numeric representations, aiming to improve their skills in mathematical word problem-solving (Peng et al.] 2021 Liang et al. 2022] Thawani et al. 2021 Geva et al., 2020). Some recent studies have also explored different strate- gies for tokenizing and encoding numeric data, such as using multi-ho or continuous representation
of numbers (Biggio al., 2021: Becker et al. 2023 Golkar et al. 2023). Our work contributes a new angle to this growing field by integrating symbolic and numeric understanding in a unified multi-moal pre-traine pre-training framework. By doing so, we not only capture the abstract representations of mathematical symbolic concepts but also their tangible numeric behaviors.
Symbolic Regression. Symbolic regression (SR) concentrates discovering mathematical expres- sions for complex systems and representing data patterns in interpretable symbolic form. It has broad implications in both science and engineering, facilitating the modeling of diverse physical phenomenaa (Cranmer et al.) 2020 Rudy et al. 2017: Meidani & Barati Farimani] 2023). Genetic Programming (GP) algorithms laid the foundation for SR. offering methods search the vast space of mathematical expressions (Schmidt & Lipson| 2009 Cranmer 2023). The ascent of deep learning subsequently gave rise to neural network-centric methods to reinforce SR's representational capabil- ities (Petersen et al.) 2021). Some pioneering works also combined the evolutionary strengths of GP with the adaptability of neural networks, aiming for a better SR search (Udrescu & Tegmark 2020)] Mundhenk al. 2021). However, these methods often struggle with challenges such as computa- tional intensity, limited semantic depth, and the necessity to reinitiate search for different datasets. Inspired by the success of pre- pre-trained Transformers in NLP, recent works introduced pre- trained models for SR (Biggio et al. 2021 Kamienny et al. 2022: Shojaee et al. 2023), using synthetic data and pre-trained priors for equation generation. Our multi-modal pre-trained model, SNIP, advances this research towards a more insightful SR direction, leveraging rich encodings that harmoniously bridge symbolic equations with their numeric counterparts.
3 PRE-TRAINING
As depicted in Fig. T the SNIP architecture comprises two Transformer encoders, each tailored for learning the symbolic or numeric representations of mathematical functions. These symbolic and numeric encoders are jointly trained with a task-agnostic contrastive learning objective to predict correct pairings within batch of (symbolic, numeric) examples. During pre-training, SNIP receives synthetically created symbolic equations and their associated numeric data as inputs to the symbolic and numeric heads, respectively. total, SNIP pre-trained on approximately 60 million synthetic paired examples.
3.1 NUMERIC ENCODER
The numeric encoder's foundation is rooted in the recent advancements of Transformer models for encoding numeric observations into latent spaces (Kamienny et al. 2022; D' A'Acooli et al., [022] Biggio al. 2021). In this framework, the numeric encoder represee-eeeeeenntee as Σ -iiieeggaates an
3

---

Published as a conference paper at ICLR 2024
embedder, multi-layer Transformer, and an attention pooling approach, to map numeric observa- tions (z. y) into a condensed latent vector Zy.
Tokenization. Following (Charton 2022 Kamienny et al. 2022). numeric inputs are tokenized using base-10 floating-poie notation. They are rounded to four significant digits and subsequently represented sequences of three tokens: sign, mantissa (0-9999 range), and exponent (E-100 to E100). For instance, the number 5.432 is tokenized as [+, 5432, E-3].
Encoding. Given a batch of N numeric input points (xt y) E RD+1, each is represented by 3 + 1) tokens. With increasing D and N, the input sequence length grows, challenging the quadratic complexity of Transformers. To address this, we employ an embedder, as suggested by (Kamienny) al. 2022), before the Transformer encoder. This embedder maps each input point to a unique embedding space. The resulting embeddings, with dimension demh. are then fed into the encoder. For the numeric encoder, we utilize a multi-layer Transformer architecture (Vaswani et al.) 2017). Notably, due to the permutation invariance of the N input points for each batch sample, we exclude positional embeddings, aligning with the approach in (Biggio et al. 202I), This encoder variant is denoted as EncY. The representation at its I-th layer is given by V.- Enc. (Vi-1), where I ranges from to Ly, and Ly signifies number of layers within the numeric encoder.
Attention-based Distillation. To distill the information from the Transformer's output into com- pact representation for the whole sequence of observations, we employ an attention-based pooling mechanism, following (Santos et al. 2016). Let Ay denote the attention weights, which are com- puted as: Av = softmax (Wa Viv where Wa E [Rd is a learnable weight matrix, and we take the transpose of VLv E sequence-leved representation, Zv, is then obtained by: Zv = Ay. VLy- This attention mechanism allows the model to focus on the most informative parts of the data points, effectively compressing the information into a fixed-size embedding.
to apply softmax along the sequence dimension N. The compact
3.2 SYMBOLIC ENCODER
The symbolic encoder in our framework also draws inspiration from recent advancements in Trans- former models for encoding symbolic mathematical functions, as demonstrated in works such as (Welleck et al. 2022 Lample & Charton 2020). Here, the symbolic encoder- denoted as E is a composite entity parameterized by 1/, encapsulating the embedder, a multi-layer Transformer, and attention-based pooling mechanisms. Given an input symbolic expression f. this encoder outputs a condensed representation Zs.
Tokenization. Mathematical expressions are tokenized by prefix order of their trees, following the principles outlined in (Lample & Charton 2020). This process employs self-contained tokens to represent operators, variables, and integers, while constants are encoded using the same method- ology as discussed in Sec. 3.1 representing each with three tokens. In alignment with (Lample & Charton 2020), we use special tokens (BOS)) and [{EOS)] to mark sequence start and end.
Encoding. Given batch of symbolic expressions with M tokens, each symbolic input is repre- sented as S, = E (BOS); :., : EiiEoS)] + S where Su E (M Here, E refers to the embedding matrix, t, denotes the i-th token, M signifies the number of tokens in the symbolic expression, demh is the embedding dimension, and Spos represents the positional embed- ding matrix. the symbolic encoder, we use a Transformers model with the same architecture as in Sec. 3.1 This variant of the encoder, denoted as Enca. processes the symbolic inputs. The 1-th layer representation is described as S = Ency (S,-1), where varies from 1 to Ls. and Lg indicates number of layers within the symbolic encoder.
ivention-based Distillation. The symbolic encoder also employs attention based pooling, as in Sec. 3.1 This mechanism computes weighted sums distill information from the symbolic expres- sion into compact representation Zs- = As Sty using attention weights Ag through softmax along the symbolic sequence.
3.3 UNIFTED PRE-TRAINING OBJECTIVE
Our work introduces a multi-modal symbolic-numeric pre-training approach, SNIP, which aims to facilitate mutual understanding of both domains, enabling advanced tross-modal reasoning.
Training Objective. SNIP's pre-training objective is inspired by the joint training used in CLIP (Radford et al. 2021). Incorporating both numeric and symbolic encoder, the model optimizes symmetric entropy loss over similarity scores. It employs a contrastive loss (InfoNCE (Oord) al. 2018) objective) to learn the correspondence between numeric and symbolic data pairs. Specif-
4

---

Published as a conference paper at ICLR 2024
ically, this approach learns to align embeddings of corresponding symbolic-numeric pairs while distancing unrelated pairs. The objective function can be defined as:
-= Σ (log NNN(CC(2))) 2s, Zv) + log (C 2s));
(1)
(o,s)EB
where represents the batch of (symbolic, numeric) data pairs, NCE Zy) and NCE(Ev, Zs) denote the contrastive losses symbolic-t-nnmeeti and sumeric-to-symbolle similarities, respec- tively. The symbolic-to-numeric contrastive loss, NCE Zs, Zv): is calculated as:
exp Zs- Zz)
NCE( Zs, Zv)
(2)
Σze(Zī exp 2s:2
where 7 is temperature, z represents positive SNIP numeric embeddings that overlap with SNIP symbolic embedding Zg, and Z are negative numeric embeddings implicitly formed by other nu- meric embeddings in the batch. A symmetric equivalent, NCE( Zs) also defines the numeric- to-symbolic contrastive loss. More implementation details are provided App. B
3.4 PRE-TRAINING DATA
In our SNIP approach, pre-training relies on vast synthetic dataset comprising paired numeric and symbolic data. We follow the data generation mechanism in (Kamienny et al. (022), where each example consists of N data points (zr, y) E RD+1 and a corresponding mathematical function f. where y fzz) Data generation proceeds in several steps, ensuring diverse and informative training examples. More details about each of the following steps are provided in App. A
Sampling of functions. We create random mathematical expressions using a process detailed in (Kamienny et al. Lample & Charton (020). This process involves selecting an input dimen- sion D, determining the number of binary operators, constructing binary trees, assigning variables to leaf nodes, inserting unary operators, and applying random affin transformations. This method ensures diverse set of functions for training.
Lample&
Sampling of datapoints. After generating a function, we sample N input points and find their corre- sponding target values. To maintain data quality, we follow guidelines from (Kamienny al. (022), discarding samples with inputs outside the function's 's domain or exceptionally large output values. Our approach includes drawing inputs for each expression from various distributions, enhancing training diversity. The generation process of datapoints also involves selecting cluster weights and parameters, sampling input points for each cluster, and normalization along each dimension. To emphasize on the function's 's numeric behavior rather than the range of values, we also normalize the target values y between (0, 1).
4 USING SNIP FOR CROSS-MODAL PROPERTY PREDICTION
To evaluate SNIP's capability for modal comprehension between symbolic and numeric do- mains, we conducted targeted experiments. These tests aimed to assess the model's aptitude for predicting specific numeric mathematical properties based on the symbolic inputs-a non-trivial task requiring mutual understanding of both domains. For this purpose, we identified set of math- ematical properties; details can be found in App. C In this section, we focus on two numeric prop- erties for one-dimensional datasets: Non-Convexity Ratio (NCR), and Function Upwardness. The NCR approximates function convexity with values between NCR=0 (fully convex) and NCR=1 (fully concave). Upwardness quantifies the function's directionality by assessing the segments where data increases within the training domain, ranging from UP=-1 1 for strictly decreasing functions to UP=1 for increasing ones. Due to space limitations, only results for NCR and Upwardness are discussed here. A complete list of properties with their detailed prediction results and corresponding chance levels, as well as their SNIP's pre-trained representations, are provided in App.
4.1 MODELS AND TRAINING
To assess property prediction top of SNIP's embeddings, we employ a predictor head that passes MIP to yield the predicted values. We adopt a Mean Squared Error (MSE) loss function for training on continuous properties. We consider three key training configurations to probe the efficacy of SNIP's learned representations:
these embeddings through a hingle-hidden-Bee
Supervised Model: Utilizes the same encoder architecture as SNIP but initializes randomly.
SNIP (frozen): Keeps the encoder weights fixed, training only the predictor head.
SNIP (finetuned): Initializes encoder from pretrained SNIP, allowing full updates during training.
For a fair comparison, all model variants are trained on identical datasets comprising 10K equations and subsequently tested on a distinct 1 Keequation evaluation dataset. These datasets are generated using the technique described in Sec. 3.4
5

---

Published as a conference paper at ICLR 2024
0.1
0.0
0.4
Supervised
SNIP (Frazen)
(inetune)
Without Pretraining
Pretrained. Beeore Finetuning
After Finetuning
Pretrained,
107
Training Samples
Figure 2: 2D I-SNE representations of the encoded vectors across three model variants, colored for (top) Non con-Convexity Ratio and (bot- tom) Function Upwardness prediction tasks.
Figure 3: R² scores NCR prop- erty prediction task vs. the number of training samples.
4.2 RESULTS
Quantitative Results. Table presents the Normalized Mean Squared Error (NMSE) and accuracy metric Accg.1 for all three models across the tasks of pre- dicting NCR and Upwardness. Acco.1 reflects the percentage of predic- SNIP (finetuned)
Table Results of using SNIP for property prediction.
Non-Convexity Ratio
Upwardness
Model
NMSE
Acca.
NMSE
AAcco.1
Supervised
0.5299
0.565
0.5356
0.563
Here,
SNIP (frozen)
0.0731
0.861
0.0540
0.847
0.0683
0.0400
0,901
tions within absolute tolerance = 0.1
the true normalized values: Acc, Ni..r 1 p. 7} where p, and P. are the true and pre- dicted property values for the i-th example. Results reveal significant gap in performance between the purely supervised model and those benefiting from SNIP's prior knowledge. This performance gap be attributed SNIP's pre-trained, semantically rich representations, enabling enhanced generalization to unseen functions. Additionally, fine-tuning the SNIP encoder results in marginal performance gains, indicating the model's capability to adapt different downstream tasks.
Qualitative Findings. To delve deeper into the power of SNIP's representations, we compared its pre-tinetuning and post-finetnning latent spaces against that of supervised model lacking pre- training. using t-distributed Stochastic Neighbor Embedding (t-SNE) van der Maaten & Hinton 2008]. The visualizations are color-coded by the corresponding properties (Fig. 2p. Consistent with the quantitative outcomes, the supervised model's latent space, shown in Fig. 2(a), exhibits lim- ited structural coherence. contrast, SNIP's latent space Fig. bb) shows pronounced clustering and distinct property trends. Notably, further fine-tuning of the encoder for these prediction tasks, depicted Fig. 2(c), results in a more structured latent space, marked by clearer linear trends in properties. This finding underscores SNIP's quantitative advantages and its flexibility in adapting to downstream tasks.
Low Data Regime Analysis. We evaluated how training sample size influences the test R² 1 NMSE scores for predicting NCR, assessing three model variants on fixed 1K-smplle test set (Fig. 3). In low data regime scenarios with low just 100 training samples, the pervised model's score fell sharply to 0.292, while both SNIP variants maintained scores above 0.745. Upon increasing the training sample size 1M, all models showed improvement; however, SNIP variants continued to lead. We observe that the supervised baseline model might approach SNIP's performance with more training data, which reasonable, since this model is specialized only for the prediction of this property. However, SNIP's value lies in its flexibility the pre-trained representations can be efficiently adapted to new tasks. These results emphasize SNIP's superior generalization from limited data, underscoring the SNIP's rich semantic encodings.
5 USING SNIP FOR SYMBOLIC REGRESSION
SNIP aims to synergize symbolic and numeric reasoning through mutual learning, offering en- hanced capabilities for tasks that require both numeric-symbolic understanding and generation. A paramount task in this context is Symbolic Regression (SR), which identifies interpretable symbolic equations represent observed data. Essentially, SR transforms numeric observations into un- derlying mathematical expressions, thereby making it a numeric-to-symbolic generation task. The significance of SR extends to revealing functional relations between variables and offers ideal

---

Published as a conference paper at ICLR 2024
Training
Mapping Layers
Decoder
Latenk-yy
Fretrined Dosodar
(b) inference
SMP Chocolrr
Augmented Population
Generral on
Updote Populonion
Latent Space Optimization
Figure 4: Using SNIP for Symbolic Regression: (a) Training includes adding an expression generation module atop SNIP's numeric encoder; (b) Inference aims to enhance expressions by optimizing within SNIP's inter- polatable latent space.
benchmark for evaluating SNIP's pre-trained numeric representations. Recent advancements in SR leverage encoder-decoder Transformer frameworks (Biggio et al., 202T Kamienny et al., 2022). Therefore, to effectively undertake SNIP for SR, we perform the following two steps: First, training an expression generation decoder top of the SNIP's numeric encoder for generating the symbolic functions. Second, conducting latent space optimization (LSO) within SNIP's interpolatable latent space, enriched by pre-training, to further enhance the equation generation.
5.1 SR MODEL ARCHITECTURE AND TRAINING
We build the SR model upon SNIP's ' numeric encoder Eg which transforms numeric data into mantically rich embeddings. On top this encoder, we implement an expression generation module Gw that integrates an expression decoder D3 and mapping network 9,, to generate symbolic ex- pressions: 9. = Do o (Z)
Expression Decoder. To use SNIP for SR, overlay expression generation decoder Da. after SNIP's s numeric encoder (shown Fig. 4 a)). This decoder, which utilizes a multi-layer Transformer
BBigio al.] 2021 Kamienny et al. 2022 is trained map numeric encodings into symbolic
expressions, aiming to minimize the divergence between the predicted f and actual functions f.
Mapping Network. Inspired by the ClipCap approach (Mokady et al., (02T) in the field of image captioning, which integrates CLIP's pre-trained image embeddings with GPT-2 pre-trained text gen- eration model through a learnable mapping network, we adopt a similar strategy for SR. As shown Fig. (a), to facilitate integration with E2E's (Kamienny et al., (022) pre-trained SR decoder (DBSE), we introduce learnable Mapping Network 9y, This module translates SNIP's numeric embeddings Zy into a compatible input for D Specifically, g Rdomm + R M Xdemb reshapes SNIP embeddings into a sequence with maximum length M. This approach lets us leverage the existing pre-trained SR decoder without the need for training from scratch.
Training. The training objective is minimize the token matching cross-entropy loss symbolic expressions: L(f.f)
between the predicted f and ground-truth IT Σ, log the j-th token in f. given the preceding true tokens. Here, the decoder is initialized from pre-trained weights (Kamienny et al. (022)) and trained jointly with the mapping network to learn numeric-to- symbolic expression generation. More details on the model designand training implementation can be found in App. B
1
where P(t 11..)t
is the conditional probability of
5.2 SEMANTIC LATENT INSIGITT FOR SR
Traditional SR methods rely searching within the vast equation landscape, dealing with the dual challenges of combinatorial complexity and limited prior knowledge (Burlacu et al.) 2020 Schmidt
& Lipson 2009). Recent approaches incorporate deep learning to better navigate this space, integrat-
ing learned numeric-to-symbolic priors into the search process (Udrescu & Tegmark 2220 Petersen
2021 Biggio et al. 2021 Kamienny et al. 2022). Yet. these are also often constrained by et al.
their reliance on the function search techniques at the decoding stage (Mundhenk et al. 2021 Holt
et al.
2023 Landajuela et al., 2022), perpetuating the limitations. For example, in the Genetic Programming (GP) function search techniques, mutation and breeding steps across 'winning' sub- expressions are prone to significant deviations in function's numeric behavior. This emphasizes the

---

Published as a conference paper at ICLR 2024
necessity for better search strategy attuned to the semantics of the function. Recently, alternative strategies, like latent space learning of symbolic functions through Variational Autoencoders (VAEs) (Popov et al. 2023; Mežnar et al. 2023), trained exclusively for symbolic function reconstruction, do show promise but fall short by neglecting numeric behaviors essential for SR tasks.
In contrast, SNIP offers a novel solution through task-agnostic joint learning paradigm. This joint learning approach imprints the latent space with a wealth of integrated symbolic and nu- meric semantics that serve a high-dimensional 'semantic fin- gerprint' for various function behaviors and their inherent simi- larities. Therefore, unlike the latent space in (Popov et al. 2023 Mežnar et al., 2023), SNIP's task-agnostic latent space embod- ies robust numeric-symbolic prior, providing an ideal landscape for SR search. By augmenting SNIP's numeric encoder with an expression generation decoder (as shown in Fig 4. we can cre- ate a generative latent space-2 crucial asset for the numeric-to- symbolic generation task of SR. Our empirical investigations on the generative latent space further enrich this narrative. The in- nate interpolatability of SNIP's latent space, as demonstrated in Fig suggests a meaningful correlation between latent space representations and their corresponding numeric behaviors. In this figure, for a source function Z (blue curve) and a destination function Z (orange curves), we linearly interpolate within the numeric encoded vectors to obtain Zint. This interpolated embedding is decoded into a symbolic function f 9_ (Z(7t)) Upon computing f over dataset æ, we find that the interpolated function exhibits a behavior that is semantically in between the source and destination functions.. This is a significant advantage for nuanced search and explorations during the symbolic discovery process. Moreover, the fixed dimension demb of this space, which is substantially lower than the combinato- rial optimization space of equations, streamlines the search process. Given these attributes, SNIP's generative latent space stands compelling candidate for more effective approach to SR.
Figure 5: Interpolatability of SNIP numeric latent space.
5.3 SNIP LATENT SPACE OPTIMIZATION
As shown in Fig. 5 SNIP latent space interpolation shows a meaningful correlation with the func- tions' numeric pattern. This observation compels us to undertake a more comprehensive explo- ration of the latent space. Specifically, to fully harness the expressive capabilities of pre-trained SNIP embeddings in the context of SR, we employ Latent Space Optimization (LSO) as outlined in Fig. 4b). (b). This optimization process involves stochastic search over latent space Zv, with the objective of maximizing numerical fitness accuracy. To benefit from both prior knowledge of pre- trained model and capabilities of search method, we initialize the search population by augmenting the given dataset into a partitioned population P = (P1, P2, Pa). Specifically, P1 contains encod- ings from random sub-samples, with size n < N of the original data; Pa includes encodings from sampled inputs with their target values y perturbed by random Gaussian noise (perturb and then encode); and P3 includes perturbed encodings from a fixed sampled data (encode and then perturb). Each agent p with representation Zú is evaluated using fitness function based on the R fitting metric. Candidate symbolic functions are generated for each agent by feeding encodings to the ex- pression generation module f, = G. The functions' constants are then refined using BFGS (Fletcher 1987), with goal of optimizing the R² score against training data (Kamienny et al., 2022). Then, updates to the latent population are carried out using gradient-free optimizer, which accommodates the nonfiifferentiable characteristies of the function generation evaluation metrics. This latent optimization process runs for I' iterations or until achieving predefined Ragp criterion. The optimal symbolic function fO is then evaluated a holdout test set. Overall. LSO leverages SNIP's s rich latent space to efficiently transform symbolic regression's 's combinatorial search into continuous optimization of fitting performance. Details on the LSO algorithm and implementation are in App. E. An ablation study analyzing the impact of LSO and choice of optimization algorithm is also provided in App. E.55
5.4 EVALUATION ON SRBENCH
Datasets. SNIP was assessed on PMLB datasets Olson et al.) 2017) outlined in SRBench (La Cava
ct al. 2021), including: 119 Feynman equations (Udrescu & Tegmark 2020). 14 ODE-Strogatz chal-
lenges (La Cava et al. 2016), and 57 Black-box regression tasks without known underlying func-
8

---

Published as a conference paper at ICLR 2024
Complextyy Ranni
SBR2P
Complouity Rank:
Complesity
di sertena
Mean Accuracy (8? 9919 Rank
Median R Rank
Mean Accurscy IR >00 0.99) Rank
(a) Strogatz datasets
(b) Black-box dalasets
(c) Feynman detasets
Figure 6: Pareto plots comparing Ht² and equation complexity of all methods across SRBench datasets: (a) Strogatz, (b) Black-box, and (c) Feynman. Using SNIP for SR yields strong comple-xy trade- off, evidenced by its first Pareto-front locating in all datasets. Here, each point depicts method's median ranking within the data group, with lines/colors signifying Pareto dominance. The "*" marks SR methods in the Black-box datasets.
tions. For specifics each dataset, refer to App. D Leveraging the E2E's SR decoder (Kamienny ct al. 2022) for our decoder initialization, which IS trained for D S 10, we similarly constrained SNIP's pre-training and evaluation to datasets with continuous features and dimensionality D S 10. Also, since the range of target values is important, especially for predicting the constants, we do normalize y for this task. More details the experiment settings provided in App. E
Results. Fig. 66 illlustrates SNIP's performance against the recent end-to-end (E2E) transformer SR model (Kamienyy et al., (022) and all the SRBench baselines. The Pareto plots exhibit rankings for Fitting Accuracy against Model Complexity. The model's accuracy is evaluated using It2 and its complexity is evaluated as the number of nodes in the expression tree of the generated equa- tion (La Cava et al.) (021I. Here, SNIP shows a strong accuracy complexxtt balance, placing on the first Pareto-front across all datasets. On Strogatz datasets, SNIP demonstrates top-tier accuracy of 0.928, outperforming all the leading baselines. For Black-box datasets, SNIP again shows com- petitive accuracy while achieving lower complexity (47.52) than the competitive Operon baseline (64.95). On Feynman datasets, SNIP locates the Pareto frontier, offering better complexity than Operon (31.63 vs. 69.87) and better accuracy than AlFeynman (0.882 vs. 0.798) baselines. More detailed results on the SRBench datasets can be found in App. E
6 DISCUSSION AND CONCLUSION
We introduced SNIP, multi-modal symbolic-numeric pre-training model that learns how to as- sociate the symbolic and numeric aspects of mathematical functions. We showed that SNIP ex- hibits remarkable capabilities in estimating cross-modal mathematical properties, particularly in low data regime scenarios, outperforming fupev-sue models. Also, by leveraging the latent space that SNIP constructs-capturing both functional behaviors and symbolic forms- model demonstrates competitive performance in symbolic regression, even when compared to leading GP baselines. While SNIP showcases robustness and versatility in integrating symbolic and numeric learning, it has notable limitations. It struggles with data patterns that cannot be clearly expressed as closed-form mathematical functions. Also, its performance is tied to the pre-defined data generation protocol, adopted from (Lample & Charton 2020; Kamienny et al. 2022). which sets constraints factors such as input dimensionality, and the vocabulary of mathematical operators. For example, the current protocol limits input dimensions D < 10 due to the exponential increase in expression complexity at higher dimensions. Exploring higher-dimensiona. settings is an interesting avenue for future research that would likely require significant updates to the data generation protocol. Despite these limitations, SNIP has wide range of capabilities, presenting a powerful tool in the intersec- tion of symbolic and numeric mathematics. Future research can focus on potential applications of SNIP, from using numeric guidance in cymbolic-t--ymbolli tasks such as function integration to using symbolic guidance for numeric-to-numeric tasks such as zero-shot extrapolation and super- resolution. Also, the SNIP's learned representations could serve a foundation for innovative evaluation metries of symbolic-numerie proximity, as well as efficient data and feature valuation.

---

Published as a conference paper at ICLR 2024
REFERENCES
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. cookbook of self- supervised learning. arXiv preprint arXiv:2304.1222 2023.
Sören Becker, Michal Klein, Alexander Neitz, Giambattista Parascandolo, and Niki Kilbertus. Predicting ordinary differential equations with transformers. In Andreas Krause, Emma Brun- skill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Pro- ceedings of the 40th International Conference on Machine Learning, volume 202 of Proceed- ings Machine Learning Research, pp- 1978-2002. PMLR, 23-29 Jul 2023. URL https: //proceedings.. /tppooeeinng..........r.s//////.
Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Paras- candolo. Neural symbolic regression that scales. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pp. 936 945. PMLR, 18 24 Jul 2021. URL hetps: html
mlr wtw..pppp./..v...... wttp:///ww.............e/s//////
Bogdan Burlacu, Gabriel Kronberger, and Michael Kommenda. Operon c++:: An efficient ge- netic programming framework for symbolic regression. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, GECCO '20, pp. 1562-1570, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450371278. doi: htt.stt:::// //doi. www.oogg//.... 1
10.1145/33799993399999
Zhonglin Cao, Rishikesh Magar, Yuyang Wang, and Amir Barati Farimani. Moformer: Self- supervised transformer model for metal organic framework property prediction. Journal of the American Chemical Society, 145(5):2958-290) 2967, 2023. doi: 10. 1021//acs.2e1 141200. 1420. URL
https: /top::://doi... www.ioorg/1... 1021/ 2cl142440
PMID: 36706365.
Francois Charton. Linear algebra with transformers. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL
Miles Cranmer. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.01582, 2023.
Miles Cranmer, Alvaro Sanchez- Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. ISBN 9781713829546.
Stéphane D Ascoli, Pierre-Alexandre Kamienny, Guillaume Lample, and Francois Charton. Deep symbolic regression for recurrence prediction. Kamalika Chaudhuri. Stefanie Jegelka, Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Con- ference on Machine Learning, volume 162 Proceedings of Machine Learning Research, pp. 4520 4536. PMLR, 17-23 Jul 2022. URL https: httpa:::/pooceeeiiee mlr press/v162/ d-ascoli22a. html
Xiao Dong, Xunlin Zhan, Yangxin Wu, Yunchao Wei, Michael C Kampffineyer Xiaoyong Wei, Minlong Lu, Yaowei Wang, and Xiaodan Liang. M5product: Self- belf-harmonized contrastive learn- ing for e-commercial multi-modal pretraining. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 21252 22262,, 2022.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 889 898, Melbourne, Australia, jul 2018. Association for Computational Linguistics. doi: 10.18653/v1/P 8- 1082. BtLhhttps:/ /cc ac antholooo www.hhoool....gg//ggooo 1082
Roger Fletcher. Practical Methods of Optimization. John Wiley & Sons, New York, NY, USA, second edition, 1987.
10

---

Published as a conference paper at ICLR 2024
Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Proceedings the 58th Anmual Meeting of the Association for Computational Linguis- tics, pp. 946-958, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.acl-main.83 URL https:ttpp..... wataacaaanttooooooooooo www.oogggg000... acl -ma mm.inn.. 89
Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, et al. xval: A con- tinuous number encoding for large language models. arXiv preprint arXiv:2310.02989, 2023.
Paul Hager, Martin J Menten, and Daniel Rueckert. Best of both worlds: Multimodal contrastive learning with tabular and imaging data. P Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 2.392-23993 2023.
Samuel Holt. Zhaozhi Qian, and Mihaela van der Schaar. Deep generative symbolic regression. In The Eleventh International Conference on Learning Representations, 2023. URL https:
/ //opereviees nwt. wotts//foummmmmmm wow.rrrmmmmm..oooooo.oooaaa
Samy Jelassi, Stéphane d' A'scoli, Carles Domingo-Enrich, Yuhuai Wu, Yuanzhi Li, and François Charton. Length generalization in arithmetic transformers. arXiv preprint arXiv:2306.1544 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research. pp. 4904-4916, PMLR, 18-24 Jul 2021. URL https: attss://proceeednng .mlr.
Sxess/v139/jia2/ htm.
Kamienny, Stéphane d' Ascoli, Guillaume Lample, and Francois Charton. End- to-end symbolic regression with transformers. In Advances in Neural Information Processing Systems, 2022.
William La Cava, Kourosh Danai, and Lee Spector. Inference of compact nonlinear dynamic models by epigenetic local search. Engineering Applications Artificial Intelligence, 55:292-300 2016. ISSN 0952-1976. doi: http://dbbboogg///.oool//ggeaapp .004. URL hhtpps: //www.
sciencedirect. htt.a//c................/.tettaa
William La Cava, Patryk Orzechowski. Bogdan Burlacu, Fabricio de Franca, Marco Virgolin. Ying Jin, Michael Kommenda, and Jason Moore. Contemporary symbolic regression methods and their relative performance. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL httpp:/s/dtttetttee-eeecccaaappp
nww. htt.pp.....peeee http:cccppppe/2//1///////// Pac.bddcc---aaprrr-ee roundl. pdf
f
Guillaume Lample
Gample and François Charton, Deep learning for symbolic mathematics. In International n forum?
Conference on Learning Representations, 2020. URL/ttps::/
1d=S1e2YeHPDS
Guillaume Lample, Timothee Lacroix, Marie-Anne Lachaux, Aurelien Rodriguez, Amaury Hayat, Thibaut Lavril. Gabriel Ebner. and Xavier Martinet. Hypertree proof search for neural theorem proving. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (cds.), Advances in Neural Information Processing Systems, volume 35, pp. 26337- 26349. Curran Associates, Inc., 2022. URL htt.ahccces/ppppeeeeenn..n.errrr htwp://eeeee...o......eeeeooooao
Ei B1111/88800688111122223383333333
Mikel Landajuela, Chak Lee, Jiachen Yang, Ruben Glatt, Claudio P. Santiago, Ignacio Aravena, Ter- rell Mundhenk, Garrett Mulcahy, and Brenden K. Petersen. A unified framework for deep sym- bolic regression. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /tte::://opeeeevii/ /oppnreveew.... net/ det.orrmmmmmdann/noaaaee
11

---

Published as a conference paper at ICLR 2024
Zhenwen Liang, Jipeng Zhang. Lei Wang, Wei Qin, Yunshi Lan, Jie Shao, and Xiangliang Zhang. Mwp-bert: Numeracy-augmenter pre-training for math word problem solving. In Findings of NAACL 2022, pp. 997-1009,,02
Jing Liu, Xinxin Zhu, Fei Liu, Longteng Guo, Zijia Zhao, Mingzhen Sun, Weining Wang, Hanqing Lu, Shiyu Zhou, Jiajun Zhang, et al. Opt: Omni-perception pre-trainer for cross-modal under- standing and generation. arXiv preprint arXiv:2107.00249 2021.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Kai-Wei Chang. A survey of deep learning for mathematical reasoning. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14605-14631, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18663311/2022.allllmmoo
URL.
http:.ogg///00 2023.ac1-long..0
Kazem Meidani and Amir Barati Farimani. Identification of parametric dynamical systems using integer programming. Expert Systems with Applications, 219:119622, 2023. ISSN 0957-4174.
doi: https://doi.org/10/ 100.0.0116............... eswa. 19622.
Sebastian Mežnar, Sašo Džeroski, and jupučo Todorovski. Efficient generator of mathematical ex- pressions for symbolic regression. Machine Learning, Sep 2023, ISSN 1573-0565, doi: 10. $109944-22--04400-2 URLpttts: /ttp::///ooii....
doi oog. 1ttp:///..000011101110000 023-
Seyedali Mirjalili, Seyed Mohammad Mirjalili, and Andrew Lewis. Grey wolf optimizer. Ad-
vances in Eigineering Software, 69:46-61, 2014. ISSN 0965-9978. doi: https///doi.or//10.
1016fj.ahvengsott22033.2..000 URL https::/wwwtt http///wwww.ccccireeeettttt. htt.:e/ccciieett..cocccceeeeeeee www./ciencceeeeeeeeeeeooooceeeee article/p11/00965555 701.669997888111 /001///000059977101181333
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2 1 2021.
Terrell N. Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P. Santiago, Daniel faissol, and Brenden K. Petersen. Symbolic regression via deep reinforcement learning enhanced genetic programming sceding. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL httpss://pperrevieee net/Eorum? 2i=t htw.waoo1ddyyy
Randal Olson, William La Cava, Patryk Orzechowski, Ryan Urbanowiez, and Jason Moore. Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData Mining, I0(1):36, Dec 2017. ISSN 1756-0381, doi: 10. 10.1188655 URLhttps://doi.
orq /0g/100.200 11.. 11e.111661111 3000://1110000000010111000000010
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXivv/807.03746 2018.
Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. Mathbert: A pre-trained model for mathematical formula understanding. arXiv preprint arXiv:2105.00377, 2021.
Brenden K Petersen, Mikel Landajuela Larma, Terrell N. Mundhenk, Claudio Prata Santiago, Soo Kyung Kim, and Joanne Taery Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations, 2021. URL /ttttps://operreeet net http//orumm?/ htt.uummmm.mmmmmmmh0000022
Sergei Popov, Mikhail Lazarev, Vladislav Belavin, Denis Derkach, and Andrey Ustyuzhanin. Sym- bolic expression generation via variational auto-encoder. PeerJ Computer Science, 9:e 2,,124,,200 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 8748 076387763.3. PMLR, mlr, wwwpsppessssp....... /assss/v13. /139/ radford21a. 18 28-24 Jul 2021. URL /tttp:://prcceet html
12

---

Published as a conference paper at ICLR 2024
J. Rapin and O. Teytaud. Nevergrad A gradient-free optimization platform. /ttpa/om..ttps://GG11111 https://GitHub com/F Bww.//aaeokeeeeeeeeeeeeeeeeeeeee New.eergeeeaeaaeaaa 2018.
Samuel H. Rudy, Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Data-driven discov-
ery of partial differential equations. Science Advances, 3(4)x1602614, 2017. doi: 10.1126,
sciadv.160261.. URL lttps:: /ettps://ww..r wwwssciieeseeee org/ wwtp:///o/////.......ooo 16622614
Cicero dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou. Attentive pooling networks. arXiv preprint (622::600000309 2016.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical rea- soning abilitiee of neural models. In International Conference on Learning Representations, 2019.
URL
ew net forum 7 d= gR5iR5FX
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science Advance, 324(5923):81-855 2009, ISSN 0036-8075. doi: 10.1 126/science. 165893.
Parshin Shojaee, Kazem Meidani, Amir Barati Farimani, and Chandan K. Reddy, Transformer- based planning for symbolic regression. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2022.
Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji- Rong Wen. molecular multimodal foundation model associating molecule graphs with natural language. arXiv preprint wwXiv:2209.05493 2022.
Kenichi Tamura and Mareus Gallagher. Quantitative measure of nonconvexity for black-box tinuous functions. Information Sciences, 476:64-82, 82, 2019. ISSN 0020-0255. doi: https://doi. org/10.1016/j 1016/j 201. 10.009. URL https://wwwl www.ncedieeett.cc.oomo www.s/icm//eeeeeeeee..ecee..eeee
Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. Representing numbers in NLP: survey and a vision. In Proceedings of the 2021 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, pp. 644-656, Online, June 2021. Association for Computational Linguistics. doi: 10.11853/v1100 1/202 .nacc--man.55..
URL
wrt.or//////// 2021 naacl caaancl-main55.
Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for sym- bolic regression. Science Advances, 6 6(16))eayy263), 2020. doi: 10. 10.11266ccadv..ayy266 URL aay2631
https: www. www.esciiiencc..me org. doi/ abs
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma-
chine Learning Research, 986))2577-2605, 2008. URL http::ttp/// http://iinnrrrrg.peeee wwt.ppppp.//../////
varddermaatennea.. html
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30, 2017.
Xiao Wang. Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research, pp. 1-36, 2023.
Sean Welleck, Peter West, Jize Cao, and Yejin Choi. Symbolic brittleness in sequence models: On systematic generalization in symbolic mathematics. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):8629888 8637, Jun. 2022. doi: 10. v36.. 20841. URL httpp:://ojs.ooo org/ http:/ndexx.cco pww.AA//pppp////////a/aa AAAT /ttp///////rriiceeeeeeee///////0
13

---

Published as a conference paper at ICLR 2024
Eugene P. Wigner. The unreasonable effectiveness of mathematics in the natural sciences. richard courant lecture in mathematical sciences delivered at new york university, may 11, 1959. Com- munications on Pure and Applied Mathematics, 13(1):1-14, 1960. doi: https://doi.orr//10000// cpa.3160130102. URL httpp:: /apps://onniiin inel ibbrar.r.... wi httpcccc/ccaaassoee cpa. 316131110
1002/
Yiyuan Zhang. Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xi- angyu Yue. Metaarrrnsformmr: A unified framework for multimodal learning. arXiv preprint arXiv:2307. 10802, 2023.
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419 2023.
Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales. Self-supervised multimodal learning: A survey, arXiv preprint arXiv:2304.00 01008, 2023.
14


Reviews:
[{"review_id": "k86wK05aTG", "reviewer": "Reviewer_7mJN", "summary": "The paper proposes a new model for pre-training an embedding space for mathematical expressions. The obtained embeddings aim at encoding both symbolic and numerical properties of the corresponding expressions.  The paper introduces a dual encoder-scheme to learn such joint embeddings, inspired by similar schemes in multi-modal representation learning.  \nThe paper focuses on two tasks: property prediction and symbolic regression.   \n\nFor property prediction, a simple extension of the embedding architecture is proposed, where an additional head is exploited to predict numeric and symbolic properties of the input equations. The results show that pretrained embeddings (possibly, fintuned) provide a much better prediction in terms or R^2 and NMSE.   \n\nFor symbolic regression, a much more elaborate scheme is proposed. The existing encoding architecture is mapped with a pretrained-decoder using a technique similar to ClipClap. Second, an inference latent space optimization process is exploited to exploit the interpolability of the pertained latent space. The results show that SNIP is in the Pareto front for all the three investigated datasets.", "questions": "1) It is unclear whether there are other methods that exploit pretraining for mathematical expressions, or other semi-supervised approaches. In both the comparison, it is clear that some of the methods have been exposed to a very different amount of (unsupervised) data. So one may wonder whether there are other methods of exploiting such data than the one proposed by the authors.  \n\n2) In order to apply the proposed method to symbolic regression, a LSO process is added on top of the standard training. It is unclear what is the impact of such step on the quality of the generation and whether competitors employ similar strategies and to which extent. Do the authors have performed an ablation study or have intuitions about the impact of the LSO?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and easy to follow.  \n\nThe method is simple, sound and elegant.\n\nGenerating a large number of expressions automatically is easy: these settings are the ones that may benefit more from a pretraining strategy.", "weaknesses": "From the current shape of the paper, it is a little bit hard to investigate the novelty. The majority of techniques are inspired by similar techniques in multi-modal representation learning, and it is not clear whether there has been similar endeavors in the context of mathematical expressions representation learning or closer domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5yVNtLDI1W", "reviewer": "Reviewer_RHHe", "summary": "The paper introduces SNIP, a Symbolic Numeric Integrated Pre-training approach, meant to unify numeric and symbolic domains via contrastive learning. The proposed method follows the rich literature on multi-modal pre-trained models whose main goal is to combine different modalities such that their representations are appropriately linked and correlated in the latent space. \nSNIP is based on two encoders, one for numerical data and one for symbolic expressions. Contrastive learning is used to unify the resulting representations. Once this pre-training phase is completed, the numerical encoder can be extracted and combined with a symbolic decoder responsible for predicting the symbolic expression associated with the input numerical values, a.k.a. performing end-to-end symbolic regression. Thanks to the rich structure acquired by the numerical encoder during pre-training, latent space optimisation can be performed to enhance the results of symbolic regression. The results show that SNIP's encoders contain cross-domain information about symbolic and numerical inputs and that symbolic regression benefits from the pre-training phase.", "questions": "See weaknesses part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In terms of clarity the paper is generally well written and easy to follow.\n\nIn the context of end-to-end symbolic regression performed by Transformer models, the problem of unifying numerical and symbolic representations is very relevant. As such, I believe SNIP represents an interesting step in this direction.\n\nThe experiments in Section 4 and Appendix C effectively demonstrate that the single encoders' representations contain cross-domain information, e.g. the symbolic encoder's latent space is informative about numerical properties associated with the evaluations of the input expression. In addition, the example in Fig. 5 seems to suggest that the latent space possesses a continuous structure, in the sense that nearby points can be decoded into expressions that have a similar numerical evaluations.\n\nThe results in Section 5 show that the proposed approached outperforms E2E, which is the closest baseline deep learning model for symbolic regression. The model performs comparably with state-of-the art genetic-programming-based symbolic regression methods.", "weaknesses": "- Some parts could be better explained and more details should be provided. For example, in Section 4 it is not clear how the task is performed. Looking at the Appendix, it looks like the prediction of the NCR and Function Upwardness is made by extracting the embedding from the symbolic encoder only as the goal would be to assess whether the symbolic representations contain numerical information as well. If my interpretation is correct, I believe this should be explained more clearly in the main body.\nAnother source of confusion is the way symbolic regression is performed. In particular, it is not clear to me if the decoder is trained or not. Are the authors keeping it frozen and training only the mapping network? \n\n- The experiments performed in Section 4 are conducted on 1-d symbolic expressions. Although this setting is already insightful, I would be curious to see whether similar tests can be conducted on higher-dimensional expressions.\n\n- While Transformer-based approaches to symbolic regression provide advantages in terms of inference time compared to genetic programming approaches, they may suffer from some of the limitations characterizing standard language models. One such limitation could be that they may hallucinate and output a symbolic expression that does not fit the numerical data at all. I would have liked to see some investigations on this aspect, for example an analysis on the extent to which output symbolic expressions are numerically aligned with the input dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IQqF8UjzE2", "reviewer": "Reviewer_PqVT", "summary": "The paper proposes SNIP, a joint encoder for numeric and symbolic mathematical data, -- i.e. functions and their values. SNIP consists of two transformer-based encoders, one for numeric sequences, and one for symbolic functions (including numeric pre-factors: e.g. 6.21cos(2.1x+3.25)). At the output of each encoder, an attention pooling mechanism converts the encoded sequence into a fixed-dimension embedding. \n\nThe numeric encoder architecture is very similar to Kamienny (2022). Floating point numbers are discretized into three tokens (following Charton 2022), D-dimensional vectors (represented as 3D tokens) are reduced to a single embedding using a feed-forward network (as in Kamienny), and sequences of N embeddings are fed, without positional encoding, into a 8 layer transformer with 16 heads and 512 dimensions. \n\nThe SNIP architecture is pre-trained on generated pairs of functions and their values, similar to the generated training sets used for end-to-end symbolic regression (Biggo, d'Ascoli, Kamienny). The authors use a symmetric contrastive loss, and show that the trained embeddings account for qualitative properties of the encoded functions, such as their growth and convexity (Figure 2, and appendix C ans D).\n\nThe model is then tested on two tasks: \n* predicting qualitative properties of functions, by training a one layer classifier on top of a pre-trained model (either frozen or trainable)\n* symbolic regression, by adding an autoregressive 16-layer transformer decoder at the output of the  pre-trained numeric encoder, and performing pre-factor fine-tuning of the predicted solution (i.e. tuning the constants in the functional expression) by minimizing mean square error over perturbed input data (noisy and/or subsampled).\n\nFor property prediction, the pre-trained model achieves high accuracy after fine-tuning on a small sample of functions (1000, to 10000). \nFor symbolic regression, the pre-trained model achieves state-of-the-art performance on the SRBench evaluation dataset. It outperforms better previous end-to-end approaches (Kamienny) and is comparable to the best evolutionary methods (Operon, SBP-GP).", "questions": "**Section 3 - encodings**\n* Alternative encodings for real numbers been proposed in previous works. Biggio feeds the floating point representations into the embedding, Becker et al. propose the two-bit encoding (https://arxiv.org/abs/2307.12617). Recently (after the submission deadline), Golkar et al. introduced XVAL (https://arxiv.org/abs/2310.02989), an improvement over the three token embedding. These alternatives should be referenced in the paper (either in section 3 or in the related works). A discussion of their relative merits would be useful.\n* In several previous works,, symbolic functions are encoded as \"skeletons\", where pre-factors are replaced by constant tokens, e.g. CST cos(CST x + CST) instead of  6.21 cos (2.01 x + 3.25). Could they be used in SNIP, instead of encoding prefactors (as 3 token sequences)?\n* Can you provide dataset sizes, notably the number of pre-training examples, in the main? \n* Section 3.1 p.3: the base 10 floating point encoding, using three tokens per numbers, was introduced in Charton (2022) \n* Section 3.2 p.4: the prefix enumeration of trees was introduced in Lample (2020)\n\n**Section 4 - predicted properties**\n* $R^2$ makes little sense when measuring scalar quantities. Also, $R^2=1-MSE$, so the columns in table 1 are redundant. Could you add scalar accuracy metrics, such as the percentage of test examples predicted within a certain tolerance of their actual values? \n* Add the chance level for the indicators in table 1.\n* The comparison between pre-trained and unsupervised model (figure 3) is unsatisfactory: it overlooks the fact that the pre-trained model already saw 60 million examples. The analysis is interesting but needs to be improved.\n* Several properties were measured in this evaluation, but only average results are presented. Can detailed results for each property be provided? (maybe in the appendix)\n\n**Section 5 - Symbolic regression**\n* The decoder is significantly larger that what was used in previous works (16 layers, vs 8 in Kamienny and 6 in Biggio), why is this so? Can you provide ablation results for different architecture choices (layers, but also dimensions and heads)?\n* Figure 11.a, in Appendix E, strongly suggests that SNIP is learning different (independent?) models for different dimensions D. Is there a benefit in training SNIP on many dimensions at the same time? Or should different models be trained on different dimensions? \n* Please provide results on a held-out dataset of generated functions, and scaling results for different problem dimensions (D), function complexity (e.g. family of operators used), number of input (N), precision achieved (in the MSE sense).\n* The late space optimisation experiments would benefit from a more detailed description in the main (perhaps at the expense of the introduction, which is a bit repetitive). In particular, the impact of LSO on model performance should be presented and discussed.\n* A gradient-free library like Nevergrad (https://facebookresearch.github.io/nevergrad/) could be used to provide better perpective on the potential impact of LSO. At present, only one algorithm is used, with little justification.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Joint training on functions and their values is a promising idea, and the paper proposes a working architecture. The results on symbolic regression achieve a new state of the art for end-to-end transformer-based methods, on par with the best evolutionary tools. \n\nThe paper is clearly written, and the method is sufficiently described. Experiments are convincing.", "weaknesses": "* The architecture is complex and no ablation studies are provided. In particular, the model is larger than previous works on symbolic regression with transformers. SNIP has 8 and 16 layers in the encoder and decoder vs 6 and 6 in Biggio, and 4 and 8 in Kamienny. Without an ablation study, it is difficult to know if the improved performance is due to the new architecture, or is just an effect of scaling. \n\n* Most evaluations focus on symbolic regression, which is also the pre-training task. Therefore, the claim that SNIP can be used on many mathematical tasks is not well-supported. To prove that the model can be applied to other tasks, evaluations on symbolic to numeric tasks, like function evaluation, or on pure symbolic or numerical computations (eg symbolic integration, linear algebra) would be needed.\n\n* The test sets for symbolic regression are small, and somewhat limited. Feynman has 119 functions, many of them redundant, most of them very simple (first order approximations of physical laws). Strogatz has 14, all in low dimension. Evaluating from a larger test set of generated functions (held-out from the training set) would greatly improve the paper. \n\n* There are no ablations on results. How does model performance scale with problem dimension, function complexity, the number of inputs, desired precision?\n\n* Evaluation on function properties (section 4) is weak. The metrics used are redundant ($R^2=1-NMSE$), and not appropriate (what is the point of $R^2$ when only one number is predicted?). Additional metrics are needed, such as the number of correct predictions (e.g. model prediction within 1% of correct values). \n\n* The comparison with unsupervised training (section 4) is inconclusive. The pre-trained models used 60 million examples, comparing them with an unsupervised model trained on 10,000 is unfair. In fact, the 87% accuracy achieved by the unsupervised model after only one million examples is quite surprising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iICQgJlRge", "reviewer": "Reviewer_8R9V", "summary": "This paper introduces SNIP, a multimodal pre-training method whose aim is to combine the symbolic and numeric properties of functions into a structured and meaningful latent space.\nThe authors show that the resulting embeddings can be used for a variety of downstream prediction tasks such as predicting the degree of convexity or upwardness of a function. They also combine their model with a decoder and a latent space optimiser to produce a powerful symbolic regression method.", "questions": "- There is a lack of details in some parts of the paper:\n    - Finetuning (section 4): (i) It isn’t clear to me whether the properties (NCR and upwardness) are predicted based on the symbolic or numeric data. (ii) What is the dimension ? I don’t see how one can define upwardness for multi-dimensional functions so I assume D=1?\n    - Interpolability plot (figure 5):  are these graphs qualitative plots, or are they actual functions in the latent space ? I would like the authors to comment on this.\n- I’m not sure the fine-tuning task with 100 examples can be called few-shot. Few-shot usually refers to only a handful of examples. I would instead call this the “low data regime\". \n- In Fig 3, it looks like supervised model is going to catch up with SNIP… This is a bit problematic, ideally we would need to see results with more training examples to make sure SNIP remains better asymptotically. Authors the authors may argue that SNIP is much better in the low data regime, I don’t think this is the most important asset of SNIP, as we typically have infinite data in these kind of tasks anyway.\n- I find it a bit weird to use a mapping network to convert the latent vector to a sequence. I understand that the authors want to mimic the cross-attention of the decode as in Kamienny et al, but intuitively, I would have instead tried using the latent vector as a “BOS” token, then use cross-attention with the output embeddings of the encoder (before they are pooled). This could be worth trying in the future.\n- I don’t find the results section of sec 5.4.2 particularly well presented, especially in paragraphs “strogatz”, “black box” and “Feynman” where many numbers are presented which can directly be read from the graph — since the orderings of the models are similar on all three datasets, it would be better to have a more qualitative description extracting the key takeaways", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall this a very enjoyable and valuable paper. I particularly appreciated:\n- Novelty: although the paper builds on top of existing approaches, the CLIP-like training is a clever contribution which gives a lot of value to the paper.\n- Impact: I believe this approach has a real potential to impact the way we perform symbolic tasks.\n- Clarity: the paper is extremely pleasant to read. Although some parts lack details (see weaknesses), the paper is very well-written, easy to follow, and the figures are excellent.\n- Content: I really enjoyed the experiments the authors present. The embeddings plot of Fig. 2 and interpolability plot of Fig. 5 give a lot of intuition, and the applications to cross-modal property prediction\n- Experimental soundness: the authors put a lot of effort into evaluation. They rigorously benchmarked their model using SRBench (which many SR papers avoid still today), and have an extensive set of additional results in the appendix.\n- Code availability: the authors provide an anonymous link towards their code.", "weaknesses": "In some parts, the authors are too hasty in their presentation of the technical details, missing out some important informations (see the questions below for details, and some other minor issues). Apart from this, I do not have many concerns about this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)