Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Deep Generative Symbolic Regression (DGSR), a novel method for the task of Symbolic Regression. DGSA uses a set transformer to encode the input data into permutation-invariant representations and uses an autoregressive decoder to generate the closed-form equations. The model is pre-trained on datasets sampled from a given prior. During inference time, the decoder is further refined on the test dataset. For both pre-training and refining, the model is updated using reinforcement learning (policy gradient or Priority Queue Training) to minimize the MSE between the ground-truth output y and f(x) of the predicted equation f. Experiments show that this method performs better than the prior works.

strength_and_weaknesses: Strength:
Technically, I think the best design of DGSR is to use as the training target the MSE between the ground-truth y and f(x) of the predicted function f. Although this target is nondifferentiable and requires some form of RL, it enables (1) the model could learn to generate any equations that are equivalent to the ground-truth equation and (2) the model could be refined on the test dataset during inference. 
Compared to NGGP, the proposed DGSR model is pre-trained and therefore provides better initial equations for genetic programming during inference.
Compared to pre-trained encoder-decoder transformers with CE loss, DGSR could learn to generate equivalent equations and could be refined during inference.
Therefore, DGSR achieves the best performance as shown in the experiments.

Weakness:
1 I suspect that the use of the set transformer is not important. Among the three invariant properties mentioned in sec. 3.1, why is the first property a good assumption ("an encoding function g that is permutation equivariant to the individual variables in [xi1, . . . , xid] = Xi of the points in X")? I think many examples are not permutation-invariant to all variables. 
2 To validate how each new component contributes to the improvement, more ablations are expected.
(1) Replace the set transformer encoder as a plain transformer or just replace your network as GPT3 and maintain the same training and refining.
(2) Pre-train your network with CE loss without refinement. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. 
It seems to be novel to combine pre-training and RL for symbolic regression. 
If code is released, the results should be reproducible. 
summary_of_the_review: Overall, I think this is a good paper. The proposed approach is technically sound and achieves good results. The main concern is the missing ablations. I'd like to improve the rating if more ablations are added.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors improve the state-of-the-art in symbolic regression. They observe a few challenges with the prior art. The most important challenges that the authors solve are 1) an encoder that is invariant to the permutation of the datapoints in the dataset; 2) a loss function that encourages learning the invariances of the equation; 3) an efficient inference procedure for sampling functions of the modeled posterior $p(f|\mathcal{D})$ by refining the posterior through policy gradients. The results demonstrate that the authors' method Deep Generative Symbolic Regression (DGSR) samples more unique functions, samples mostly valid functions and has a better complexity (how quickly it finds the true function) and recovery rate than the prior art on popular benchmarks, and is able to extrapolate to larger number of inputs than the number of inputs in the pre-training stage.  The authors augment these results with thorough ablations studies and experiments to understand why DGSR works. 
strength_and_weaknesses: Strengths:

1. Well-motivated and useful improvements over the prior art. 

2. State-of-the-art results and thorough ablations. 

Weaknesses:

1. Why do you think NESYMRES generates valid equations almost perfectly? Is there a way to improve your DGSR method to increase the valid sequences? For example, adding a regularization term that excludes invalid equations in the inference stage of your algorithm?

2. What do you think might be the directions for improving DGSR. Please discuss this in more depth.
clarity,_quality,_novelty_and_reproducibility: The paper is very clear on its motivation and experimental setups. The quality of the paper is stellar: the authors have spent a lot of care for each detail. The contributions are somewhat original, largely combining known approaches to fight with the issues of the prior art in symbolic regression.
summary_of_the_review: This is a very strong methodological paper and it will be helpful to future work in symbolic regression. I recommend accepting this paper, because the contributions are useful and non-trivial.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors approach SR via deep, permutation-invariant, generative models, a framework they call DGSR.  They evaluate their framework on numerous problem sets, and have good empirical  performance. 

strength_and_weaknesses: Strengths: 
  - The approach appears novel.
  - Numerous datasets tested  
   - Empirical results are solid

Weaknesses: 
- Many distracting typos in the paper, including, but not limited to: 
  - Page 1: Use "e.g.," in the examples of discrete and continuous components
  - Page 1: "Firstly" -> "First"
  - Page 2: "for all samples n" -> "for all samples i" 
  - Page 2: "Defined with y_i ..." is not a sentence. 
  - Page 2: "pre-fix" -> "prefix" 
  - Page 3: "dataset invariant" -> "dataset-invariant"
  - Page 3: "a Inference" -> "an Inference" 
  - Page 3: "equation f," -> "equation f" 
  - Page 3: "That is" -> "That is," 
  - Page 3: "Specifically" -> "Specifically,"
  - Page 3: "decoder," -> "decoder" 
  - Page 4: "An Bayesian" -> "A Bayesian" 
  - Page 4: "equations P1, intuitively" -> "equations P1.  Intuitively," 
  - Page 4: "Where" -> "where" 
  - Page 4: Delete "Although" 
  - Section 3.2: Put parentheses around "P2", "P3", etc. 
  - Table 1 caption: delete space before "?" 
  - Page 5: "Valipour et al. (2021) proposes" -> "... propose".  Same with Jin et al. and "uses"

- Other nagging questions:
  - Page 2: Does "closed form" imply no summations or recursion? 
  - So are you only focusing on the structure search and not the parameters?  What happens if you find a structure that is not very close but you get misled by some parameters that mask the mistake, and you miss a better structure? 
  - Why is 10d the number of samples?
  - Is recovery rate only evaluated on structure, or also on parameters?
  - Table 2: Why not normalize average eq evals by number of correct solutions? Right now it looks like GP is superior.
  - Please enumerate limitations and future work in Section 6, not bury them in the appendix.

clarity,_quality,_novelty_and_reproducibility: The quality appears solid and the approach novel. However, the numerous typos reduce clarity. 

summary_of_the_review: Overall, I like the paper, but would prefer to see a cleaner presentation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a pre-trained generative model for solving Symbolic Regression for multivariate problems in low dimension (from 2 to 12 dimensions). 

A sequence to sequence model, a set transformer encoder and an autoregressive decoder (as in Biggio 2021), is trained on generated datasets of values (X, f(X)) of random functions f, using the (weighted) mean squared error of model predictions. At inference, model predictions are refined by training the decoder, again with a mean squared error loss, while keeping the encoder weights frozen. Finally, a top-k Monte Carlo search is performed to find a best solution (according to the mean squared error).

The authors evaluate their model on a set of benchmarks (over 300 functions overall). The approach achieves state-of-the-art performance on the recovery rate (percentage of correct solutions retrieved).
strength_and_weaknesses: **Strength**

Symbolic Regression is an important field, and the authors propose an interesting architecture, with promising results (albeit on very small test sets). The paper is interesting, and clearly written. 

**Weaknesses**

The tests sets are limited to very small benchmark suites. For instance, the results from table 2 (the main claims about the method success)  are based on 54 equations, and the test set substantiating P3 has only 8 equations (Feynman d=5). The authors should back up their claims with results from large generated datasets (e.g. using the Lample & Charton generator they use for pre-training data set). Without additional tests, claim P3 should be removed, because it is not substantiated.

The recovery rate, the only evaluation criterion presented in the main, makes very little sense in practical applications. First, it limits tests to artificial situations where the function to be found is already known to the experimenter. Second, it precludes the use of real constants in the formulas (which are always approximate). This is a standard problem in SR, and the main rationale for the approaches of Biggio and Kamienny (suggesting the authors are tackling a much simpler problem). Finally, it assumes an in-domain solution, that can be expressed within the pre-training model vocabulary - a very strong limitation. Alternate metrics exist, like R2, MSE on test data, or (better) extrapolation MSE on test data (i.e. for each test function split the datapoints into two subsets $(x,f(x))_{infer}$, used for inference fine-tuning, and $(x,f(x))_{test}$ used for testing). The authors should prefer them in the main. 

On claim P2 (inference speed), the authors have a point that their method is faster than RL and GP approaches. But I fail to see how it can be faster than the end-to-end approaches of Biggio, d'Ascoli and Kamienny, which use *no gradient based-retraining*. While this is not necessarily a problem (most practical uses of SR can accommodate slow inference), this should be mentioned more clearly in the paper. Also, could inference speeds (in seconds) be provided? 

On claim P1, the fact that symbolic transformers, trained on cross-entropy, can learn equivalence between different equations, has been observed in many previous papers. Lample (2019) mentions it (section 4.6), and D'Ascoli exploits this property to find alternate solutions (closed forms vs recurrences corresponding to the same data). D'Ascoli also observes that training from simplified formulas does not improve accuracy, which further suggests that equivalence has been learned. DGSR shows that training on MSE can also achieves this, a new result to my knowledge, but this is by no means a "first" (as table 1 suggests). This should be discussed, and claim P1 should be toned down. 

In section 3.1, the authors insist on the permutation equivariance of features (the d dimensions of the problem). While this might help with benchmarks such as SRBench, where many equations display this feature, I am not sure it is a good practical idea. In most physical models, features will not be permutation equivariant, because they have different units. To my knowledge, other SR approaches do not try to enforce such feature permutation equivariance. The authors should either drop it, or better justify its necessity. 


clarity,_quality,_novelty_and_reproducibility: **Clarity** - The main paper is clearly written, but a lot of important elements have to be looked up in the supplementary. Unfortunately, large parts of the supplementary material are a long version of the paper, with many duplicates (or close duplicates), which does not help clarity. The authors would be advised to structure the appendix for easier reference, and eliminate duplicates.

**Novelty** - The basic architecture (set transformer for encoder, and autoregressive decoder) is very close to Biggio (2021). The data generation comes from Lample (2019), and pre-training procedures and losses. To my knowledge, the gradient-based inference retraining proposed by the authors is a new idea. Note that previous authors use MSE-based fine-tuning of the raw predictions of their pre-trained model. For instance, D'Ascoli (2022) and Kamienny (2022) uses MSE error to guide beam search. As such, the "cannot refine" mention in table 1 seems misleading. The authors should mention fine-tuning in other transformer-based approaches, and discuss the specificities of their approach (there is an obvious trade-off between performance and speed, here). 

**Quality** - see weaknesses, in the previous section. In my opinion, the claims P1, P2, and P3 are insufficiently substantiated. 

**Reproducibility** - 
The information provided by the authors should allow one to reproduce their results.
summary_of_the_review: Overall, the paper is interesting and the results are promising, but the experimental methodology is very disappointing. The test sets are too small to back some of the claims (P3), the main metric is of little interest in practice (it expects the solution is already known, and doesn't account for approximate solutions, or out of domain situations). 

The comparison with other approaches is also unsatisfying. Inference speed, one of the basic claims of the authors, is only compared with the slowest models (GP and RL). The fact that other models learn invariance is not credited. 

I would encourage the authors to back up their claims with stronger test results. If this can be done during the rebuttal period, I will be happy to increase my recommendation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses the problem of symbolic regression --- finding a symbolic formula that fits given data.  A Novel approach is given based on a transformer to propose formulas and a genetic algorithm for searching further from proposed formulas.  The results were improved through interaction with the reviewers and seem significant.

There is some concern about novelty as many (but not all) of the ideas used in this paper have appeared earlier.  But the novel contributions and the upgrades in the paper since submission justify publication.
justification_for_why_not_higher_score: For me, the main issue for not giving a spotlight is the question about novelty.  I am also concerned about the use of synthetic data.  It suggests that there is no pressing set of applications waiting to benefit from these methods.
justification_for_why_not_lower_score: Symbolic regression is a conceptually natural problem and meaningful work in this area deserves to be published.
Decision: Accept: poster