# Bias and Variance Analysis of Contemporary Symbolic Regression Methods
## 4 Results
### Error and Model Size  
SRBench [8] shows that GP-based methods provide most accurate results on the test partition for the given Feynman datasets. To demonstrate, which algorithms perform better than the others, we rank the algorithms by their average accuracy on each problem. We assign rank one to the algorithm with the lowest error for a specific problem and noise level combination, rank two to the second best, and so on. The same procedure is done for model size. The distributions for accuracy and model size over all noise levels are shown in Figure 3. The median and the interquartile range of all error values and their corresponding ranks, broken down by noise level, are shown in Table 1. Table 2 shows the same statistics for the size of all models. The box of FFX in Figure 3b is cut off to prevent distortion of the axis, but the distribution is described in detail in Table 2.  
Table 1 shows that most algorithms perform similar across all noise levels. One exception is AIFeynman [19]. As also described in the original paper by [PERSON] and [PERSON] [19], AIFeynman outperforms other algorithms on problems without noise but struggles with low noise levels of \(\sigma\in[0.001,0.01]\) and performs worst at \(\sigma=0.1\). Operon  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & & \(\mathbf{\sigma=0.000}\) & \(\mathbf{\sigma=0.001}\) & \(\mathbf{\sigma=0.010}\) & \(\mathbf{\sigma=0.100}\) & **All Noise Levels** \\ \hline \multirow{2}{*}{AllFeynman} & RMSE & \(9.7\times 10^{-15}/1.1\times 10^{-2}\) & \(4.3\times 10^{-4}/1.6\times 10^{-2}\) & \(6.0\times 10^{-3}/3.0\times 10^{-2}\) & \(4.1\times 10^{6}/1.5\times 10^{4}\) & \(3.2\times 10^{-3}/6.6\times 10^{-2}\) \\  & Rank & \(2/3\) & \(3/4\) & \(4/4\) & \(10/4\) & \(4/7\) \\ \hline \multirow{2}{*}{Operon} & RMSE & \(4.3\times 10^{-4}/8.5\times 10^{-3}\) & \(7.4\times 10^{-4}/1.0\times 10^{-2}\) & \(6.9\times 10^{-3}/2.9\times 10^{-2}\) & \(4.3\times 10^{-2}/1.5\times 10^{-1}\) & \(4.6\times 10^{-3}/4.1\times 10^{-2}\) \\  & Rank & \(2/3\) & \(2/3\) & \(2/3\) & \(3/5\) & \(2/3\) \\ \hline \multirow{2}{*}{GP-GOMEA} & RMSE & \(3.5\times 10^{-3}/8.2\times 10^{-2}\) & \(3.0\times 10^{-3}/5.5\times 10^{-2}\) & \(9.1\times 10^{-3}/6.0\times 10^{-2}\) & \(4.0\times 10^{-2}/1.3\times 10^{-1}\) & \(1.2\times 10^{-2}/9.5\times 10^{-2}\) \\  & Rank & \(3/2\) & \(3/2\) & \(3/2\) & \(2/3\) & \(3/2\) \\ \hline \multirow{2}{*}{AFP-FE} & RMSE & \(3.4\times 10^{-3}/1.6\times 10^{-1}\) & \(7.9\times 10^{-3}/1.7\times 10^{-1}\) & \(9.3\times 10^{-3}/1.8\times 10^{-1}\) & \(3.8\times 10^{-2}/2.0\times 10^{-1}\) & \(1.3\times 10^{-2}/1.8\times 10^{-1}\) \\  & Rank & \(5/1\) & \(5/1\) & \(4/2\) & \(4/3\) & \(4/1\) \\ \hline \multirow{2}{*}{EPLEX} & RMSE & \(3.4\times 10^{-2}/2.6\times 10^{-1}\) & \(3.6\times 10^{-3}/2.1\times 10^{-1}\) & \(2.1\times 10^{-2}/1.7\times 10^{-1}\) & \(4.4\times 10^{-2}/1.9\times 10^{-1}\) & \(3.4\times 10^{-2}/2.0\times 10^{-1}\) \\  & Rank & \(5/3\) & \(5/2\) & \(5/3\) & \(4/2\) & \(5/3\) \\ \hline \multirow{2}{*}{AFP} & RMSE & \(2.6\times 10^{-3}/3.8\times 10^{-1}\) & \(3.0\times 10^{-2}/3.4\times 10^{-1}\) & \(3.8\times 10^{-2}/3.5\times 10^{-1}\) & \(6.5\times 10^{-2}/4.1\times 10^{-1}\) & \(4.0\times 10^{-2}/3.7\times 10^{-1}\) \\  & Rank & \(6/2\) & \(7/2\) & \(6/2\) & \(6/2\) & \(6/2\) & \(6/2\) \\ \hline \multirow{2}{*}{gplearn} & RMSE & \(8.3\times 10^{-2}/4.0\times 10^{-1}\) & \(8.1\times 10^{-2}/3.9\times 10^{-1}\) & \(8.8\times 10^{-2}/4.0\times 10^{-1}\) & \(1.0\times 10^{-1}/4.0\times 10^{-1}\) & \(8.9\times 10^{-2}/4.0\times 10^{-1}\) \\  & Rank & \(7/3\) & \(7/3\) & \(8/3\) & \(7/3\)and GP-GOMEA provided the most accurate models across all noise levels. The difference between those two algorithms is not significant, as the pairwise \(p\)-values of the error rank distributions in Table 3 show. Algorithms with restricted search spaces such as AIFeynman, ITEA, DSR and FFX performed worst and (mostly) without significant differences. An outlier in algorithms with restricted search space is AIFeynman for problems without noise.