# Bias and Variance Analysis of Contemporary Symbolic Regression Methods
## 5 Conclusions and Outlook  
In this study, we analyze the bias and variance of ten contemporary SR methods. We show, how small differences in training data affect the behavior of the models besides differences in error metrics. We use the models that were trained in the SRBench benchmark [8], as they provide a well-established setting for a fair algorithm comparison. We show that both bias and variance increase with the test error of models over most algorithms. Exceptions are the algorithms AIFeynman, TTEA and FFX, whose error is primarily caused by bias. This is expected as stronger restrictions of the search space should lead to more similar outputs despite changes in training data but also to a consistent, systematic error in all models. Another assumption for the high bias for AIFeynman and FFX are their non-evolutionary heuristic search, which induces bias. Our experiments prove our expectation that larger models tend to have smaller bias, except from FFX. However, despite the common assumption that larger models are susceptible to high variance, we could not observe a clear connection between those two properties in our analysis. The connection between variance and model size is distorted by the different median accuracy of the methods. Given that high accuracy is achieved both by algorithms with small and large models, also small variance occurs in algorithms with models of any size. E.g., Operon and GP-GOMEA provide the smallest error and variance across all noise levels, however, the size of their models differs clearly. While GP-GOMEA's average model size ranks between the other algorithms, Operon found the largest models of all GP-based methods. However, this still implies, that even though Operon's models are large and might look very different between multiple algorithm runs, their behavior on the Feynman datasets are very consistent. This work is a first step towards the analysis of bias and variance in the symbolic regression domain. Although bias and variance belongs to the basic knowledge in machine learning, recent studies for other machine learning methods challenge the common understanding of this topic. Therefore, we also suggest further research in this direction for symbolic regression. The most obvious extension of our work would be an analysis on real world problems, as this work was restricted to generated benchmark data of problems with limited dimensionality. While the presence of actually unknown noise, an unknown ground truth and usually a too small number of observations in such scenarios make a fair comparison hard, it would give even further insight in the practicality of SR methods. Moreover, our analysis, especially regarding variance, was limited by the high accuracy of multiple algorithms on the given synthetic problems. Further research regarding the relationship between variance and model size would benefit from harder problems, where all algorithms yield a certain level of error. This would allow a more in-depth analysis of variance and highlight more differences between algorithms, especially between GP  
Figure 8: Relation between median variance and median model size per algorithm over all problems. GOMEA and Operon. Another SR-specific aspect for further research is the analysis of the symbolic structure of SR models. While this work only focuses on the behavior and output and therefore the semantic of models, another important aspect for practitioners is whether the formulas found in SR are similar from a syntactic perspective. Conceptualization, L.K. and S.W. ; methodology, L.K. ; software, L.K. ; validation, L.K. and G.K.; data curation, L.K. ; writing--original draft preparation, L.K. ; writing--review and editing, G.K. and S.W. ; visualization, L.K. ; supervision, G.K and S.W. ; project administration, G.K.; funding acquisition, G.K. All authors have read and agreed to the published version of the manuscript.