Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper shows that transformer models can be used to accurately learn advanced mathematical computations from millions of examples.  The problems are drawn from the fields of differential equations and control theory.  The selected problems are ones that are solvable using known algorithms; however, these algorithms involve a sequence of advanced mathematical operations (e.g., differentiation, calculating the rank of a matrix, calculating the eigenvalues of a matrix, etc), for which no known simple shortcuts exist. For the experiments in this paper, for each problem a large number (50 million) of training examples are randomly generated, and are then used to train a transformer model.  Across these problems, the paper shows that the neural network is able to solve these problems at high accuracy (96-99.7% accuracy).

Strengths
- The paper ask a well-motivated question regarding whether neural networks are able to learn complex mathematical operations from examples.
- The paper is clearly written, and the experimental rigor/quality appears quite high.
- The empirical results are quite intriguing, and raises many interesting questions for future research.  For example, (1) how is a transformer model managing to attain such high accuracy on these problems, (2) what other complicated mathematical problems might be similarly learnable, (3) what are the practical implications to real systems of these results.
- The paper does a good job considering the various potential counterarguments to its conclusions in the discussion section (Section 5.4).  For example, the paper argues convincingly that (1) it’s unlikely the model is exploiting some trivial distinction between the positive vs negative examples because the examples are sampled randomly, and that (2) it’s unlikely the model is interpolating between solutions because the problem space is so much larger than the training set (and because small models which would be unable to memorize the training set also perform well).

 Weaknesses
- I would have appreciated more of a discussion about the computational cost of solving these problems mathematically vs. solving them with a neural network.  What is the computational complexity (Big-O) of each of the known mathematical algorithms for solving these problems?  Are there large computational savings from using a neural network?  What are the practical implications of the results shown in this paper?
- I think including a few more baselines would have been useful.  Also, a brief description of the FastText model would make the paper more self-contained.   One question: Why is the FastText model in Section 5.1 only trained with 2 million examples, while the transformer model is trained with 50 million examples?
- In the current experiments, the test set is drawn from the same exact (random) distribution as the training set.  I was very curious whether the model would have been able to attain high test-time accuracy, had the test examples been drawn from a different distribution.  In particular, I’d be curious how difficult it would be to construct a test distribution on which the current model performs terribly.  This line of questioning would be able to start better answering whether the model is solving the problem in a way that truly generalizes.
- I think the paper could be strengthened by adding additional error analysis to try to better understand the errors made by the transformer model.
- The paper says “training is performed on 8 V100 GPUs with float16 operations”: It would have been nice to hear more about the training process: For example, low long did training take for the various problems, and how was training distributed across the GPUs?
- I would have appreciated more of a discussion about the broader implications and significance of these results.

Overall, I thought the paper was very interesting and well executed, and I think it would make a very nice addition to ICLR 2021.
review_title: Clear, well-executed, interesting paper

final_score: rating: 7: Good paper, accept
review_content: =Quality=
High: well executed and motivated

=Clarity=
Well-situated wrt to related work.  Baseline needs to be explained more (see below).

=Originality=
The ML method is standard. The novelty is in setting up the datasets and evaluation metrics. Doing was technically complex. The datasets very valuable and will hopefully be open-sourced.

=Significance=
This paper demonstrates that neural networks are surpisingly good at the task of predicting certain properties of differential systems, such as their stability. This is a neat result and is sufficient for publication. However, the paper would have more impact if there was a concrete proof of concept of how such a tool could be used to improve the lives of practioners.

=Motivation/Introduction=
I agree with you that it is very impressive that neural networks can nearly solve these tasks. It would be helpful if you devoted more of the exposition to explaining how your classifiers could be used by practitioners to improve their workflows. Are there engineering applications, for example,  where the classifier could be used to quickly screen proposed systems?

Also, I'd like to better understand what the novel capability of your tool is. Is it faster than numerical methods for verifying these properties? More accurate? Applicable in situations where symbolic manipulation is impossible?

=Predicting Control feedback matrices=
I found this section very cool and encouraging for future work! I'd mention it in the intro more.


=Baseline classifier=
My primary hesitation with the paper is that there is not enough justification for why your baseline is sufficient. 

You need to provide far more details about the FastText baseline. Describing the name of the software package is insufficient. What is the actual model? 

"Such high performances over difficult mathematical tasks may come as a surprise, and one
might wonder whether the model is exploiting some defect in the dataset, or some trivial
property of the problems that would allow an easy way to correct solutions. We believe
this is very unlikely...because a trivial solution would be found by the text classification tool we use as a baseline."

This argument is weak without an explanation of what the baseline model is and what its inductive biases are.


Are there no heuristics from the application community that would serve as baselines? 


In the discussion, you mention in passing that "providing at train time intermediate results that would help a human calculator (frequencies for PDE, or Jacobians for stability) does not improve accuracy." It seems to me that a baseline method would be a simple machine learning model on top of some hand-crafted features of these intermediate results.


=Discussion section=
"in some of our problems, even a model with one layer and 64 dimensions
obtains a high accuracy, and such a small model would never be able to memorize that
many examples."
Either make this precise or remove it. At first glance, it seems to be that 64 dimensions has a lot of capacity. For example, 2^64 is much bigger than the size of your dataset.


=Open Sourcing=
Will you be able to open-source the datasets? Doing so would considerably increase the future impact of your work, as it would provide a benchmark for future ML methods.

=Evaluation set=
It would be cool if you could have a couple of anecdotes of applying your classifier to famous equations from papers, particularly ones where the derivations to prove stability, for example, were quite tedious. You argue that the test set is representative, since it is uniformly sampled, but are the equations of interest to the community in some corner of this space?

=Analysis/interpretation=

Do the attention patterns of the transformer reveal anything interesting?

What seems to characterize the equations that the model makes mistakes on?

Can you use your model to get  per-equation embeddings? Do they reveal interesting cluster structure in the data?

*** After reading the authors' responses ***
I have raised my score to a 7. I felt that some of the key questions, e.g. regarding generalization to mathematical expressions that are qualitatively different than the training data,  were answered well. This paper should not be reviewed as being methods-driven. It's about demonstrating a new way that deep learning could be transformative for engineering, by allowing engineers to screen proposed designs for stability, etc.
review_title: Really neat new dataset and application 

final_score: rating: 3: Clear rejection
review_content: This paper empirically demonstrated the effectiveness of neural networks for learning to predict different mathematical properties of dynamical systems, which achieve high accuracy on synthetic datasets generated by the authors.


Pros:
+ the paper is clearly written and easy to follow
+ the dataset seems to be carefully generated and is large.

Cons:
- The authors do not clearly state their methodology, including the concrete architecture of the transformer-based model, and the training loss for optimizing the model. As shown in the experiments, the accuracy of the proposed model is much higher than the baseline, but it is not clear what the major reason is, and why.
- As the dataset contains about 50 million samples, and only 10000 of them are held out for test and validation, which means the training dataset contains sufficient data and the result could just be overfitting. Besides, the authors do not describe clearly how they generate the dataset and what is the problem distribution. For some distribution (e.g., those with smaller variance), maybe 50 million is large enough and there won't be a generalization issue. As a well-known fact that neural network has universal approximation ability, it is not surprising that it can learn to predict the mathematical properties given enough data. It will be better if the authors could show a '#training sample VS test accuracy' curve to show how the method behaves differently given different numbers of training samples.

Overall this paper does not have enough technical contributions, so I vote for a clear reject.
review_title: Lack of methodological contribution

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper investigates the use of deep learning models, and specifically transformers, to learn mathematical properties of differential systems. Authors tackle three problems:

* Local stability: where the goal is predicting the stability of a given system at a given point, similar to Spectral Mapping Theorem.
* Control theory: where the goal is predicting controllability and computing the control feedback matrix as in Kalman condition.
* Stability of PDEs using Fourier transform: where the goal is to predict the existence and stability of a PDE given its differential operator.

Overall, this paper is well written and tackles an intersting problem with potential useful insights. I believe it can be improved by making the discussion more rigorous:

1. How general are the class of functions considered in each problem? For example, you generate random functions by sampling unary- binary trees. Does this give the guarantee that the majority of solutions for each of three problems can learned by deep learning methods?

2. How this work compares to previous ones such as (Lample and Charton 2020) that study learning symbolic mathematics using deep learning? Any distinct insight when investigating numerical and differential systems?

3. It is useful to see if generalizability of learned mathematical computation can be assessed by experiments, where training datasets are generated using some class of functions, and they are tested on a separated class of functions. Are such experiments feasible using unary- binary trees?


review_title: Review of Transformers for differential systems

Meta-review: This paper shows that transformer models can be used to learn certain advanced mathematical concepts such as the local stability of differential equations. Reviewers found this surprising and useful for engineers, and the evaluation was adequate. They also felt that it opens the doors to similar studies on other aspects of mathematics.
Decision: Accept (Poster)