Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper presents a benchmark for contemporary symbolic regression, constructing a set of regression problems on which SR methods are tested. The authors use a variety of datasets, some from PMLB, some other SR databases, and use R2 testing for establishing accuracy. Overall, genetic programming-based models seem to perform best, while AIFeynman perform best for finding solutions for synthetic problems. 
strengths: Overall, the paper is clearly-written, with extensive datasets for benchmarking. The authors do a good job at bringing together SR methods and comparing their performance, clearly explaining drawbacks and limitations. The work has good contributions despite the limitations and is a good exposition of presented methods. 
weaknesses: The paper presents limitations in the using real-world data from models, as authors also explain. The presentation of the results in the main text is clear, yet a more in-depth analysis of different datasets and variance across them would be helpful in further understanding the benchmark (although a more extensive analysis is presented in the Appendix). The applications could be strengthened in the exposition as well. 
correctness: The evaluation methods present limitations but they are explained by the authors. 
clarity: The paper is very clearly-written, well-structured, and good result visualization. 
relation to prior work: The methods used are explained and grounded into previous literature; the set of methods could be expanded in the future, but seems sufficient for the benchmark presented. 
additional feedback: None.
review_title: Benchmark for contemporary symbolic regression

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors introduce an open-source benchmarking platform for symbolic regression and benchmark an array of different methods on over 200 regression problems. The benchmarking code and data is readily accessible and appears to be straightforward to use (although I have  not tried). The benchmarking results reveal that the best performing methods for real-world regression combine genetic algorithms with parameter estimation and/or semantic search drivers. In the presence of noise, the authors find that deep learning and genetic algorithm-based approaches perform similarly.  
strengths: The benchmarking code appears to be easy to apply and the data is made available. The proposed way of benchmarking allows more detailed comparison of symbolic regression methods than other available benchmarking software. 
weaknesses: The concrete advantage over commercial symbolic regression benchmarking platforms (the paper mentions Eureqa and Wolfram) should be stated more clearly. It is also not exactly clear to me how the algorithms used by these platforms differ from the proposed solution and why the proposed solution is more ‘straightforward’ as mentioned in the paper. 

Overall, I think the paper spends too little time describing the proposed benchmarking method and its advantages. The summary of existing algorithms which spans a large part of the paper could be shortened to make room for these explanations. The paper’s focus is placed mainly on the benchmarking results and not on the benchmarking methodology. Shifting the focus a little could be beneficial to the wider community given this is a dataset / benchmark track.

correctness: To the best of my knowledge, the proposed methodology and experimental setup are correct.
clarity: The paper is clearly written and fairly well motivated.
relation to prior work: The authors briefly compare the proposed benchmarking against commercial software for symbolic regression benchmarking, however the differences and advantages could be made clearer.
additional feedback: No
review_title: Benchmarking symbolic regression algorithms 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper is motivated by the need for an accessible, reproducible, and realistic benchmark for symbolic regression (SR) research to help the field evaluate new methods, agree on the quality of existing ones, and reduce reliance on toy problems. To that end, it contributes SRBench, a repository of datasets and 14 SR methods constructed to allow integration of future work. The paper states that this represents “the largest and most comprehensive SR benchmark effort to date.” Relative to previous work, the paper also contributes particular attention to the multiple criteria at play in evaluating SR methods. Because SR involves not just prediction but learning an generative form, these criteria include prediction accuracy, model simplicity, and whether the method learns the true underlying model. For this reason, SRBench augments the existing Penn Machine Learning Benchmark (PMLB) of datasets without specified ground truth with a second set of 130 synthetic datasets from physics that have known ground truth.  The paper also uses the benchmark to compare the performance of the 14 SR methods. It describes statistically significant differences among the algorithms in terms of both prediction and ground-truth, though some algorithms do well for one task and not the other. For the black-boxed prediction tasks, it also compares SR methods to non-SR machine learning methods and finds some SR methods can produce models that are both better and simpler. 
strengths: First, a disclaimer: I am not an expert in symbolic regression or genetic programming, so this review is written from the perspective  of a technically literate audience that is interested in but not well-versed in the details of the symbolic regression algorithms presented here. As such, I cannot vouch for the technical accuracy of the descriptions of the algorithms in Section 3. 

That said, I appreciate the organization of Section 3. The descriptions clearly reference back to Table 1, which includes both sources for each method and links to its code on GitHub. Clicking, for example, on the “Operon” row link yields a well-formatted Github page with building and usage instructions and example code. These would be valuable resources to the expert or to a person looking to get into this area. 

More broadly, the paper and SRBench represent a resource for the SR community. Although the benchmark draws on existing datasets and approaches,  the project of gathering those together into an accessible, reproducible, standardized framework;  incorporating 14 SR methods; and making the whole thing open to future contributions, represents extensive, valuable work. The experimental results may also be useful, both in guiding practitioners in their algorithm choices and as results that other researchers may seek to replicate or extend. 
weaknesses: 
The introduction and conclusion note that the benchmark provides a “large and diverse” set of regression problems and the paper repeatedly mentions the need to assess “real-world performance,” but it does not make it clear what kind of variety the datasets cover or what ‘real’ means here. The datasets derived from physics equations are clearly more ‘real’ than contrived toy examples, but are they a good representation of the main ‘real’ context that SR is applied to? The authors note in the conclusion that future improvements might incorporate “more realistic applications settings” but the paper would be improved if it gave a bit more attention to this point earlier on. 

In particular, the introduction gives little sense of what is at stake in a broader sense in the use of SR methods. I would recommend including some mentions here of the kinds of areas where SR methods have been or could be applied (or perhaps list a few example tasks, more specific than “physics, engineering, statistics…”). In what contexts is it advantageous or necessary to learn an analytical model rather than only predict? Or, do SR algorithms sometimes do better at prediction than ML algorithms and are they being adopted in ML applications? It would also be helpful to have a summary of what the datasets cover. Clearly, the ground-truth datasets come from physics, but what kinds of application areas are the PMLB datasets drawn from? Does it have any relevant gaps, known limitations, or conscious omissions? This can all be fairly brief, but it would help both motivate the paper and make the limitations and social implications of the benchmark more specific. 

There is additional detail on the data in the appendix, but it remains unclear to me (A) whether SR is actively being applied, say, in criminal justice applications (the example dataset context mentioned in the appendix) and (B) whether any information warning of potential biases or contextual factors for particular datasets is provided with the datasets.

correctness: I note again that I am not qualified to judge the correctness of the descriptions of the algorithms in section 3. The datasets are drawn from existing resources, but I have little further sense (from the paper at least) of how they were constructed. The design of the comparison between methods and presentation of the output appear reasonable, and Table 2 (+ the appendix) helpfully summarizes some important settings used. I have no reason to doubt that the resulting comparison of algorithm performance given those settings is inaccurate.
clarity: The paper is clearly written, with good organization and logical flow. 

Figure 1 – reads clearly.  It is easy to evaluate and compare algorithms on multiple criteria at once. 

Figure 2- this plot is a little less clear on its own, though the explanation in the text helps.  It is not clear to me whether the shapes and colors of the points have any significance.

Figure 3 – reads clearly. 

relation to prior work: Yes, the authors compare their work to existing commercial platforms, noting that their benchmark provides better ability to control experiment parameters across methods and thus determine what is really driving performance differences. They note other recent benchmarking work but distinguish their work by its attention to both solution rates and prediction accuracy rather than one or the other. The authors also motivate the paper with references to a 2012 discussion of the need for better benchmarks in genetic programming and refer in a general way to existing bad practices, which it claims have continued since 2012. They do not name particular examples here, but this may be appropriate as it avoids scapegoating particular papers or authors.
additional feedback: There are two comparisons happening in the paper – an internal one among SR algorithms (some of which have roots in ML methods) and an external one to non-SR ML algorithms. This contrast might be made more explicit. In any case, I would be interested to read more about how SR and ML approaches are similar and different and what SR algorithms add, even on tasks that ML methods perform well on. I leave this here because I do not think adding this to this paper is essential. 

----------

*** Tidbits that did not affect my overall review of the paper: ***

This sentence does not make sense to me (pg. 18): “In that regard, we do not foresee it as creating additional ethical issues  around their use, unless our work inspires a significant number of people to research symbolic regression.”

There is a typo on page 6. I believe where it reads “problems (i.e. such, in which the underlying…),” it should simply read “(i.e. in which the…)”
review_title: Straightforward paper introducing benchmark and related experimental results for symbolic regression

Meta-review: All reviewers support accepting the paper, especially after taking the author feedback into account. One concern was how the proposed algorithms compare to those implemented in commercial software platforms such as Wolfram. The authors correctly point out that the closed source nature of these platforms makes a comprehensive comparison difficult. A limited comparison could still be a valuable addition to the benchmark so that researchers can compare their algorithms to the commercial state-of-the-art. In any case, the paper is of high quality and I recommend accepting it.
Decision: Accept