Title: Extrapolation and learning equations
Authors: Georg Martius
Christoph H. Lampert
Venue: ICLR 2017
Keywords: Supervised Learning
Deep learning
Structured prediction

Abstract: In classical machine learning, regression is treated as a black box process of identifying a
suitable function from a hypothesis set without attempting to gain insight into the mechanism connecting inputs and  outputs.
In the natural sciences, however, finding an interpretable function for a phenomenon is the prime goal as it allows to understand and  generalize results. This paper proposes a novel type of function learning network, called equation learner (EQL), that can learn analytical expressions and is able to extrapolate to unseen domains. It is implemented as an end-to-end differentiable feed-forward network and allows for efficient gradient based training. Due to sparsity regularization concise interpretable expressions can be obtained. Often the true underlying source expression is identified.


TL;DR: We present the learning of analytical equation from data using a new forward network architecture.

Review discussion:
Hi,
I have a few questions:
1. Can you comment/speculate about the space of functions in which the EQL would work well? The sin, cos and other mappings seem to fit perfectly well for the functions considered in the paper, but I wonder how these will generalise to general equations, which could make the claim in the paper somewhat less significant.
2. Training EQL seems difficult -- can you comment on this, e.g. how to control generalisation/overfitting/interpolation/extrapolation trade-off in practice?
3. It would be good to compare to vanilla NNs with additional features, e.g. sin or cos or something that can capture your prior info about the underlying function.
Thanks.
Answer to 1: We would argue there is no such thing as a "general equation". Every equation one would like to learn lives in a context, which suggests suitable base functions. In the manuscript, this context is physical expressions (in particular classical mechanics), where sin/cos/etc show up naturally. If we aimed
at a very different field, e.g. quantum field theory, we would certainly need different base functions, but a domain expert would know what these are.
Note that we do not aim at learning arbitrary or black-box functions, since there is no hope to extrapolate for them. Or put differently, if the network is not equipped with the required base functions it cannot perform the task.
Answer to 2: We discuss this in the appendix A1: we aim at choosing a model that has a low validation error (i.e. interpolates well) and high sparsity (which indicates we found the correct formula).
However, these quantities live on different scales, and we do not want to introduce a trade-off parameter because it would have to be model selected and this cannot be done since we cannot measure extrapolation quality at training time.
Therefore, we use a rank-based selection (15).
Instead of the validation error and sparsity values themselves, we use the ranks of the respective solutions. Ranks are comparable quantities, so we can combine them into a single quantity and select without a tradeoff parameters.
In practice, we observe this criterion to work well, its drawback is that we need to produce multiple candidate networks in order to be able to define rankings in the first place.
Answer to 3: This is a good idea, although as mentioned above without multiplication units it will not extrapolate well as it
cannot represent the true functions. If we do include multiplications, we essentially recover EQN.
Best regards
Thank you for an interesting read. I have two questions after a first read-through:
1. Did you fix the type parameter I_i (see eq. 4) in advance? Does the choice of I_i affect the performance a lot?
2. Vanilla NNs with sigmoid units have been shown to be universal approximators.
So I guess the main selling point of the specific architecture you proposed is that your method is more efficient in say model size?
But I didn't see any experimental validation on this.
to 1: We set the I_i such that each base function is present the same number of times. We do not expect the choice to be important, as only a subset of the neurons is active in the final solution, which is selected automatically by the sparsity regularizer.
to 2: Sigmoid NNs are universal *interpolators*. Our network aims at learning not just an approximation, but the correct functional form, such that it can *extrapolate*. Figures 2, 3 and 4 show the difference: all method approximate the function well in the region where the training data is given, but only EQN identifies a function that extrapolates to new regions.
Best regards.
An interesting approach to making neural nets more intuitive with traditional basis functions.
I have one question though, the models seem to fit well for values around and tend to wander off after that. With neural nets having non-linearities like sigmoid, sine and cosine this is expected. Is there a work-around to tackle this shortcoming?
I am not quite sure I understand your questions correctly.
In case the algorithm finds the right functional form, it fits everywhere, not only where it was trained. In the plots you see that EQL typically
matches the true system's behavior, even outside the training domain.
If one doesn't find the right formula (like the MLP does), then further away values are indeed wrong. That's what we discuss as 'near/far extrapolation'.
I am not to which shortcoming you refer? To make MLP's extrapolate? This is not possible unless the underlying system is actually described by a few sums and sigmoids.
In case of the EQL, it may also happen that it does not find the right formula, as we show in some cases. This might be tackled by a different optimization method.
Thank you for an interesting read.
To my knowledge, very few papers have looked at transfer learning with **no** target domain data (the authors called this task as "extrapolation"). This paper clearly shows that the knowledge of the underlying system dynamics is crucial in this case. The experiments clearly showed the promising potential of the proposed EQL model. I think EQL is very interesting also from the perspective of interpretability, which is crucial for data analysis in scientific domains.
Quesions and comments:
1. Multiplication units. By the universal approximation theorem, multiplication can also be represented by a neural network in the usual sense. I agree with the authors' explanation of interpolation and extrapolation, but I still don't quite understand why multiplication unit is crucial here. I guess is it because this representation generalises better when training data is not that representative for the future?
2. Fitting an EQL vs. fitting a polynomial. It seems to me that the number of layers in EQL has some connections to the degree of the polynomial. Assume we know the underlying dynamics we want to learn can be represented by a polynomial.
Then what's the difference between fitting a polynomial (with model selection techniques to determine the degree) and fitting an EQL (with model selection techniques to determine the number of layers)?
Also your experiments showed that the selection of basis functions (specific to the underlying dynamics you want to learn) is crucial for the performance. This means you need to have some prior knowledge on the form of the equation anyway!
3. Ben-David et al. 2010 has presented some error bounds for the hypothesis that is trained on source data but tested on the target data. I wonder if your EQL model can achieve better error bounds?
4. Can you comment on the comparison of your method to those who modelled the extrapolation data with **uncertainty**?
We thank the reviewer for his comments.
To 1: The reason for relying on multiplication units is twofold: first, it is needed for model selection according to Occam’s Razor. We select for models with a small number of active units. For “simulating” multiplication with additive units a large amount of units are required with would favor instances with a different representation. The second reason is that the learned multiplication would only work in the training domain and would to extrapolate.
To 2: We need prior knowledge on the function class of the underlying dynamics. Again a generic polynomial is a universal approximator and can also be used for interpolation. True, EQL can also learn polynomials up to order (2*number of layers), BUT it also contains the trigonometric functions and their powers and products. The exhaustive search through all functional forms is quickly intractable. In order to extrapolate the underlying functional expression has to be identified. Simply fitting a polynomial will not do (and is likely to extrapolate badly).
To 3:
Thank you for the interesting view of interpreting extrapolation as domain adaptation with no data of the target task. Unfortunately, we do not think that the situation will lead to provable bounds, at least not comparable ones to Ben-David's, except if we make assumptions about the underlying data distribution. This we would like to avoid, especially in the context of modeling a physical system, as we do, where future data (conditions) will not even be i.i.d.. Note that even if we would have some (unlabeled) target data, Ben-David's bound would not be informative: by definition the support of the training data is disjoint from the data distribution during extrapolation, so the discrepancy terms in the bound would almost surely be maximal, and the bound vacuous.
To 4:
We assume you mean methods such as Gaussian Processes.
For the GP: Constructing a kernel that can express our hypothesis class of
functions would be very difficult, since it's not just linear
combinations of base elements, but also their concatenations.
Just products of inputs are not hard (polynomial kernel), but products of intermediate terms would, require some hierarchical construct, with kernel acting on the output of other kernels.
If we are not using our function class but general kernels then both mean extrapolation performance and “uncertainty” will grow very quickly when departing from the training domain.
The authors attempt to extract analytical equations governing physical systems from observations - an important task. Being able to capture succinct and interpretable rules which a physical system follows is of great importance.
However, the authors do this with simple and naive tools which will not scale to complex tasks, offering no new insights or advances to the field.
The contribution of the paper (and the first four pages of the submission!) can be summarised in one sentence:
"Learn the weights of a small network with cosine, sinusoid, and input elements products activation functions s.t. the weights are sparse (L1)".
The learnt network weights with its fixed structure are then presented as the learnt equation.
This research uses tools from literature from the '90s (I haven't seen the abbreviation ANN (page 3) for a long time) and does not build on modern techniques which have advanced a lot since then. I would encourage the authors to review modern literature and continue working on this important task.
We find of the criticism in this review mostly unjustified. A scientific review should be based on scientific contents. Citing Smith’s “Task of the Referee”, we would ask to be judged on questions such as “Is the goal of the paper significant?”, “Is the method of approach valid?”, “Is the execution of research correct?”, “Are the correct conclusion being drawn?”, etc. We believe the answer to all of these is yes, but we can accept if reviewers disagree. We do not find a statement “This research uses tools from literature from the '90s [...] and does not build on modern techniques which have advanced a lot since then” appropriate without indicating which ‘modern’ techniques are meant and how or why they would be more appropriate for the task of learning physical equations.
Despite the above, we would like to justify our choice of methods. We use the tools that we find are suitable for the problem, regardless when they were invented.
In particular, EQL relies on a differentiable artificial neural network, so why call it something else? We do not use ReLUs or lots of layers, not because we wouldn't be aware of them, but because these would not represent the function class we are interested in. Note that, in contrast to e.g. image classification or generation, we are not trying to learn a black box but an analytic expression.
Similar to many of the current Deep Learning works, who are based on methods from the 1990s with today’s computation power and data resources, we do apply recent techniques where appropriate, e.g. the Adam optimizer, which is from 2014.
Also, our model selection mechanism is novel and essential for the extrapolation task.
In summary: please judge the manuscript based on a) if it tackles an important task, b) if it achieves in solving the task, and c) if it enables other researchers to do the same. We believe the answer to all three questions is yes.
Thank you for an interesting perspective on the neural approaches to approximate physical phenomenon. This paper describes a method to extrapolate a given dataset and predict formulae with naturally occurring functions like sine, cosine, multiplication etc.
Pros
- The approach is rather simple and hence can be applied to existing methods. The major difference is incorporating functions with 2 or more inputs which was done successfully in the paper.
- It seems that MLP, even though it is good for interpolation, it fails to extrapolate data to model the correct function. It was a great idea to use basis functions like sine, cosine to make the approach more explicit.
Cons
- Page 8, the claim that x2 cos(ax1 + b) ~ 1.21(cos(-ax1 + π + b + 0.41x2) + sin(ax1 + b + 0.41x2)) for y in [-2,2] is not entirely correct. There should be some restrictions on 'a' and 'b' as well as the approximate equality doesn't hold for all real values of 'a' and 'b'.
Although, for a=2*pi and b=pi/4, the claim is correct so the model is predicting a correct solution within certain limits.
- Most of the experiments involve up to 4 variables. It would be interesting to see how the neural approach models hundreds of variables.
- Another way of looking at the model is that the non-linearities like sine, cosine, multiplication act as basis functions. If the data is a linear combination of such functions, the model will be able to learn the weights. As division is not one of the non-linearities, predicting expressions in Equation 13 seems unlikely.
Hence, I was wondering, is it possible to make sure that this architecture is a universal approximator.
Suggested Edits
- Page 8, It seems that there is a typographical error in the expression 1.21(cos(ax1 + π + b + 0.41x2) + sin(ax1 + b + 0.41x2)).
When compared with the predicted formula in Figure 4(b), it should be 1.21(cos(-ax1 + π + b + 0.41x2) + sin(ax1 + b + 0.41x2)).
We thank the reviewer for his comments.
Page 8: thanks for pointing this out. The approximation only holds for b=Pi/4. We change the manuscript accordingly. Typo: yes indeed the sign was missing.
Only 4 variables:  True, we only considered cases with up to 5 variables (robotic arm). We did not try it on systems with many more variables and we expect that lots of training data would be required to be able to uncover the true functional form in case all variables interact.
So interpolation will scale but extrapolation might not, at least not to hundreds.
Base functions, and universal approximator: It is correct that sine, cosine, and multiplication are the new base functions, however, the model is not just considering a linear combination of those, but also their functional composition (sine of cosine times …).
Since division is not a base function Eq. 13 cannot be represented exactly. In this case, EQL will just act as a normal approximator and performs well in the interpolation regime but fails on extrapolation like all other methods. EQL is surely a universal approximator because MLPs are a subclass of EQL.
Our responses to the reviews can be found in the individual comments below.

Meta-review:
role: PC
data: comment: This paper proposes using functions such as sin and cos as basis functions, then training a neural network with L1 regularization to obtain a simple estimate of functions that can extrapolate under some circumstances.
 
 Pros:
 - the paper has a wide-ranging discussion connecting extrapolation in regression problems to adjacent fields of system identification and causal learning.
 - the method is sensible enough, and should probably be a baseline in the time-series literature. It also seems like an advance on the hard-to-optimize Eureqa method.
 
 Cons:
 I agree with the authors that Reviewer 5's comments aren't very helpful, but this paper really does ignore or dismiss a lot of recent progress and related methods. Specifically:
 - The authors claim that cross-validation can't be used to choose the model, since it wouldn't encourage extrapolation - but why not partition the data in contiguous chunks, as is done in time-series methods?
 - The authors introduce an annealing trick to help with the L1 objective, but there is a rich literature on gradient-based optimization methods with L1 regularization that address exactly this problem.
 - The authors mostly consider toy data, limiting the potential impact of their method.
 - The authors don't compare against closely related methods developed to address the exact same setting. Namely, Schmit + Lipson's Eureqa method, and the Gaussian process methods of Duvenaud, Lloyd, Grosse, Tenenbaum and Ghahramani.
 - The authors invent their own ad-hoc model-selection procedure, again ignoring a massive literature.
 
 Given the many "cons", it is recommended that this paper not be presented at the conference track, but be featured at the workshop track.

Decision: Invite to Workshop Track