## 2 Transformer Symbolic Regression  
### Symbolic Regression  
Symbolic regression is a technique aimed to automatically discover mathematical expressions or functions from data sets [12].  
Set target dataset \(X=\{x_{1},x_{2},x_{3},\ldots,x_{n}\}\) and label \(Y=\{y_{1},y_{2},y_{3},\ldots,y_{n}\}\), symbolic regression aims to find a function \(f(\cdot)\) so that \(f=\operatorname*{argmin}(eval(f(X)))\) where \(eval(\cdot)\) indicates the evaluation of formula \(f(\cdot)\). For example, \(eval(\cdot)=\left[f(X)-Y\right]^{2}\).  
Symbolic regression is considered an extension of traditional regression methods, traditional regression methods need to assume that the data follows a polynomial distribution or some other distribution while symbolic regression doesn't have to. Symbolic regression can automatically discover nonlinear and higher-order relationships in the data, and can also be applied to multivariable problems to discover interactions between multiple variables. It can be used to uncover underlying patterns behind data to help researchers find real models and mechanisms.  
Symbolic regression is mainly realized by genetic algorithm and deep reinforcement learning algorithm.  
Traditional symbolic regression algorithms mainly use genetic algorithms to search for feasible mathematical expressions.  
The specific steps of using the genetic algorithms to search for the optimal formula are as follows:  
1. Set the initial space of arithmetic variables, operations and end conditions during the running process;
2. Initial formula set as population;
3. Evaluate the formulas based on evaluation \(eval(\cdot)\) such as the mean square error between the predicted values of formulas and the data labels;
4. Generate a new population by replication, crossover, and mutation operations;
5. Repeat steps 3 and 4 until the end condition is met, and sort the generated formula set based on the evaluations to choose the best formula.  
Symbolic regression based on genetic algorithm has some problems, it cannot take advantage of the inherent characteristics of the dataset to search for suitable formula, so the search process will be too long, the search is too inefficient, and it is easy to fall into local optimal.  
In recent years, deep reinforcement learning has made great progress in the field of optimization solutions. Many studies have applied deep reinforcement learning to solve symbolic regression problems. This kind of method can search symbols from the symbol space (the space of variables and operations), construct a mathematical expression from the symbols, and get a reward according to the fitness of the mathematical expression to optimize the strategy function. Deep reinforcement learning can achieve very good results in symbolic regression problems [13].  
Deep learning-based symbolic regression methods often model formulas as binary trees, the nodes of the tree are called symbols which are divided into variables in green and operations in blue of mathematical, as shown in Figure 3, the binary tree can be converted into mathematical formulas by depth-first search of mid-order traversal, and it is \(\cos(x\times y)+\tan(x)/\ln(x+y)\). Therefore, the problem of symbolic space exploration is regarded as the generation problem of a formula binary tree.