Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper tackles a symbolic regression (SR). SR is a task to explain observed data using mathematical expressions. The authors state that there are five individual methods and the proposed method combines them. The experimental results on SRBench, a benchmark for SR, demonstrate that the proposed method performs better than the existing methods.
strengths_and_weaknesses: #### Strengths
- Combining the individual methods for SR is interesting. The view is also helpful in understanding the current trend of the SR methods.
- The overall architecture and experimental procedure are well described in the main text and supplementary material.
- The experimental result shows that the proposed method, uDSR achieves better performance than the existing ones.

#### Weaknesses
- Since the proposed method is a combination of the existing methods, its technical novelty is limited.
- There is another method that combines several types of SR methods [a] that is shown to outperform current state-of-the-art methods. Some comparisons to uDSR were necessary.
- Figure 4 is great in terms of ablation study, but it seems that the combination of all five modules achieves statistically significant improvements over the other variations.

[a] Mundhenk et al. Symbolic Regression via Neural-Guided Genetic Programming Population Seeding. NeurIPS, 2021.
questions: The reviewer would like the authors to solve the three weaknesses above.

limitations: There are some descriptions of technical limitations, not societal ones.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper provides a holistic review of symbolic regression approaches, and picked 5 most popular ones for algorithm amalgamation to construct a new symbolic regression algorithm. The new symbolic regression algorithm is claimed to maximize the benefits of each and cover each other’s weaknesses. 

strengths_and_weaknesses: 
This paper studies an important problem, symbolic regression. The algorithm does not contain novel parts, but it offer novel way for us to understand the symbolic regression algorith especially what is the core parts and how could we combine them. The writting of this paper is clear enough. This paper provides thorough ablation studies, which showed different combinatorial ablations of uDSR components.

However, certain perspectives remains to be improved, including demonstration, and results explanation. I've put the corresponding suggestions and questions in the following, and would raise my assessment if they are properly addressed.
questions: 
Below are some questions and suggestion regarding algorithm description and evaluation.

A concrete example is missing. Though the authors showed example formulas in line 221 and introduced the benchmark, it is important for the symbolic regression paper to introduce at least one example distilled symbolic rule, so as to make it easier for the reader to link what the authors are describing to which parts in the final distilled algorithm. Especially for this hybridizing work, could the author examplify which part in the symbolic rule is generate by which sub-problem? Especially the CINDy part, as how the vector based algorithm connects to symbolic optimization remains unclear to me, and could be made clear if explained via examples.

However, it did not show the possible downside of hybridizing different algorithms. Algorithm combination in a chain shaped form is known demonstrate more variance and less bias, so will the combined algorithm underperform the one single algorithm? ("If certain single algorithm is good enough, why combining with the other worse algorithm won't downgrade it?") 

Though the author explained what the "on" and "off" mean in line 320, it is still unclear how different is the hybridized algorithm will be from some original algorithm, if all other sub-algorithms are turned off. Based on the results in fig 4, it seems that one single algorithm achieved ~ 0.6, which seems a bit lower than the results reported in the original papers. The ablation study focus on checking the components of the proposed algorithm, while the critical concern regarding this algorithm amalgamation model is what if one choose single sub-algorithm, and how does it compare with the hybridyzed algorithm. It is encouraged that the author evaluate more explicitely on how does the hybridized compared to single ones from the explanation side, and evaluate the result they got compared to the originally reported ones.


The workflow of the proposed hybridyzation is clearly explained both in Algorithm 1 and Fig 1. But the integration of the strength of the NSRS remains unclear to me. It seems that the NSRS has been downgraded to only generate expression with one specific (X,Y) problem. Is the proposed algorithm still possible to generalize across new problems?

limitations: Based on my limited assessment, this work stays free of possible negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a new symbolic regression method that unifies 5 existing symbolic regression methods and assesses the approach using SRBench a recently proposed benchmark framework for symbolic regression. It also discusses contribution of each component to the unified framework through  comprehensive ablation studies (48 combinations of the components used in the framework) and provides some recommendations regarding the component choice based on the ablation studies.

strengths_and_weaknesses: Overall, this study provides interesting findings of symbolic regression, and the reviewer found that the study may be beneficial for the community while there are some things to be addressed/clarified.

# Strengths
1. From the motivations described and ablation studies in the paper, the proposed, unified framework for deep symbolic regression seems carefully designed, and it is not like just a combination of random existing methods.
2. The ablation study to discuss which component of the proposed framework contributes to the improvement seems very comprehensive. The community may also learn from recommendations based on the ablation study.
3. Although the reviewer has some concerns in the SRBench-based evaluation in this study (see **Weaknesses**), it seems that the proposed framework provides improved performance for datasets used in SRBench, comparing to existing symbolic regression methods.


# Weaknesses
1. Some of the evaluation metrics used in this study (symbolic/accuracy solution rate, $R^2$, accuracy solution) should be clarified and justified in the paper (i.e., the paper should be self-contained). While these metrics are used in SRBench, the reviewer is concerned how essential for SR each of these metrics is.
2. It seems that NeSymReS (referred to as NSRS in this study) and SINDy are both missed in evaluation as baselines while those are components of the proposed method and the baseline results for the other components (AIF, DSR, and GP) are provided.
3. The reviewer is afraid that the choices of ranges of variables in the instance generation for pretraining are very heuristic and potentially leaked from the datasets used for evaluation in this study. According to Algorithm 3 in Appendix, the ranges of sampled values for variables seem overlapping those of Feynman datasets used in this study e.g., most of them are within range of 1 to 10 (See FeynmanEquations.csv and BonusEquations.csv available at [the original dataset provider](https://space.mit.edu/home/tegmark/aifeynman.html))
questions: - Use of the inverse function (L223) does not look like a good option in practice, but how can the author assure that it can be generalized and applied to real-world problems? e.g., what if $y - \sin(x_1) \leq 0$ in $\bar{f}^{-1}(y) = \log(y - \sin(x_1))$
- How many of the 252 datasets could the proposed method not finish the inference within the 24 hours walltime?
- How did this study obtain the results of the 14 baseline methods? If it just refers to the SRBench paper and shows the copy of the results, the authors should clearly describe so in the paper and explain whether or not the noise-injected datasets (Fig. 2) are identical to those in the SRBench paper. If the authors reran the baselines (, which would be preferred), the authors should mention so and explain computing resource used for the baselines.
- “the highest Pareto ranking,” (L302) should be clarified in the paper.
- If the authors can provide (pre)training time and inference time for each configuration in Fig. 4, it will be interesting and support some of the statements such as L351-352.
- “For real-world problems with physical units,” (L354-355) what datasets in the authors’ mind for instance? i.e., If none of the datasets used in this study is categorized into it, how did the authors find the dimensional analysis on the SR problem recommended?
- Table 2 makes the reviewer think if SINDy is contributing to the framework as its only one property is already covered by AI Feynman.

# Misc
- Figure 1 should be placed around Section 4, where the figure is first referred.
- **NSRS** should be replaced with **NeSymReS** to respect the name used in the original study (Biggio et al., 2021)
- “where F is a nonlinear function subject to certain constraints (see below)” Please specify where it can be found.
- The repeated use of “thesis” in this paper sounds weird. Maybe it means either “statement”, “claim”, or “argument”?
limitations: While the authors state in the checklist that several limitations are discussed in Section 6, the reviewer found only one limitation explicitly mentioned (L374-375). The reviewer did not find any potential negative societal impact of this work discussed in this work.
As explained in **Weaknesses** above, the main limitations of this work are the evaluation metrics (not well explained or justified either) and datasets (looks overlapping the ranges of values in generated datasets used for pretraining).
review_title: null

Meta-review: metareview: The paper presents a novel deep symbolic regression approach that is a hybridization of existing methods, showing state-of-the-art performance on the SRBench. Let me stress that being a hybrid is no reason to reject a paper as creating hybrids can be a very creative contribution. And for me this is the case here. the hybrid is not just a "mixture" but actually a very creative rewiring of components of the underlying approaches. This is also supported by the ablation study. Moreover, several of the 
issued raised were clarified well in the rebuttal.  BTW, the authors may also want to cite other approaches for equation discovery, see e.g.

Jure Brence, Ljupco Todorovski, Saso Dzeroski:
Probabilistic grammars for equation discovery. 
Knowl. Based Syst. 224: 107077 (2021)

Will Bridewell, Pat Langley, Ljupco Todorovski, Saso Dzeroski:
Inductive process modeling. Mach. Learn. 71(1): 1-32 (2008)

But this only for making the paper more self-complete. 
confidence: Certain
Decision: Accept