Venue: ICLR 2017 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The authors attempt to extract analytical equations governing physical systems from observations - an important task. Being able to capture succinct and interpretable rules which a physical system follows is of great importance. However, the authors do this with simple and naive tools which will not scale to complex tasks, offering no new insights or advances to the field. 

The contribution of the paper (and the first four pages of the submission!) can be summarised in one sentence: 
"Learn the weights of a small network with cosine, sinusoid, and input elements products activation functions s.t. the weights are sparse (L1)".
The learnt network weights with its fixed structure are then presented as the learnt equation. 

This research uses tools from literature from the '90s (I haven't seen the abbreviation ANN (page 3) for a long time) and does not build on modern techniques which have advanced a lot since then. I would encourage the authors to review modern literature and continue working on this important task.
review_title: Important task but marginal contribution

final_score: rating: 7: Good paper, accept
review_content: Thank you for an interesting perspective on the neural approaches to approximate physical phenomenon. This paper describes a method to extrapolate a given dataset and predict formulae with naturally occurring functions like sine, cosine, multiplication etc.                                                                                                                                                  
                                                                                                                                                                                                          
Pros                                                                                                                                                                                                      
- The approach is rather simple and hence can be applied to existing methods. The major difference is incorporating functions with 2 or more inputs which was done successfully in the paper. 
            
- It seems that MLP, even though it is good for interpolation, it fails to extrapolate data to model the correct function. It was a great idea to use basis functions like sine, cosine to make the approach more explicit.                                                                                                                                                                                        
                                                                                                                                                                                                          
Cons                                                                                                                                                                                                      
- Page 8, the claim that x2 cos(ax1 + b) ~ 1.21(cos(-ax1 + π + b + 0.41x2) + sin(ax1 + b + 0.41x2)) for y in [-2,2] is not entirely correct. There should be some restrictions on 'a' and 'b' as well as the approximate equality doesn't hold for all real values of 'a' and 'b'. Although, for a=2*pi and b=pi/4, the claim is correct so the model is predicting a correct solution within certain limits.      
                                                                                                                                                                                                          
- Most of the experiments involve up to 4 variables. It would be interesting to see how the neural approach models hundreds of variables.                                                                 
                                                                                                                                                                                                          
- Another way of looking at the model is that the non-linearities like sine, cosine, multiplication act as basis functions. If the data is a linear combination of such functions, the model will be able to learn the weights. As division is not one of the non-linearities, predicting expressions in Equation 13 seems unlikely. Hence, I was wondering, is it possible to make sure that this architecture is a universal approximator.                                                                                                                                                                                
                                                                                                                                                                                                          
Suggested Edits                                                                                                                                                                                           
- Page 8, It seems that there is a typographical error in the expression 1.21(cos(ax1 + π + b + 0.41x2) + sin(ax1 + b + 0.41x2)). When compared with the predicted formula in Figure 4(b), it should be 1.21(cos(-ax1 + π + b + 0.41x2) + sin(ax1 + b + 0.41x2)). 
review_title: Learning physical phenomenon

final_score: rating: null
review_content: Thank you for an interesting read. I have two questions after a first read-through:

1. Did you fix the type parameter I_i (see eq. 4) in advance? Does the choice of I_i affect the performance a lot?

2. Vanilla NNs with sigmoid units have been shown to be universal approximators. So I guess the main selling point of the specific architecture you proposed is that your method is more efficient in say model size? But I didn't see any experimental validation on this.
review_title: Choices of activation functions

final_score: rating: 6: Marginally above acceptance threshold
review_content: Thank you for an interesting read. 

To my knowledge, very few papers have looked at transfer learning with **no** target domain data (the authors called this task as "extrapolation"). This paper clearly shows that the knowledge of the underlying system dynamics is crucial in this case. The experiments clearly showed the promising potential of the proposed EQL model. I think EQL is very interesting also from the perspective of interpretability, which is crucial for data analysis in scientific domains.

Quesions and comments:

1. Multiplication units. By the universal approximation theorem, multiplication can also be represented by a neural network in the usual sense. I agree with the authors' explanation of interpolation and extrapolation, but I still don't quite understand why multiplication unit is crucial here. I guess is it because this representation generalises better when training data is not that representative for the future?

2. Fitting an EQL vs. fitting a polynomial. It seems to me that the number of layers in EQL has some connections to the degree of the polynomial. Assume we know the underlying dynamics we want to learn can be represented by a polynomial. Then what's the difference between fitting a polynomial (with model selection techniques to determine the degree) and fitting an EQL (with model selection techniques to determine the number of layers)? Also your experiments showed that the selection of basis functions (specific to the underlying dynamics you want to learn) is crucial for the performance. This means you need to have some prior knowledge on the form of the equation anyway!

3. Ben-David et al. 2010 has presented some error bounds for the hypothesis that is trained on source data but tested on the target data. I wonder if your EQL model can achieve better error bounds?

4. Can you comment on the comparison of your method to those who modelled the extrapolation data with **uncertainty**?
review_title: An interesting paper for domain adapatation with NO target domain data

final_score: rating: null
review_content: An interesting approach to making neural nets more intuitive with traditional basis functions.

I have one question though, the models seem to fit well for values around and tend to wander off after that. With neural nets having non-linearities like sigmoid, sine and cosine this is expected. Is there a work-around to tackle this shortcoming?
 
review_title: Domain of function approximation

final_score: rating: null
review_content: Hi,

I have a few questions:

1. Can you comment/speculate about the space of functions in which the EQL would work well? The sin, cos and other mappings seem to fit perfectly well for the functions considered in the paper, but I wonder how these will generalise to general equations, which could make the claim in the paper somewhat less significant.

2. Training EQL seems difficult -- can you comment on this, e.g. how to control generalisation/overfitting/interpolation/extrapolation trade-off in practice?

3. It would be good to compare to vanilla NNs with additional features, e.g. sin or cos or something that can capture your prior info about the underlying function.

Thanks.
review_title: on space of functions that EQL can learn/approximate, overfitting and comparison to adding sin/cos features to the inputs 

Meta-review: This paper proposes using functions such as sin and cos as basis functions, then training a neural network with L1 regularization to obtain a simple estimate of functions that can extrapolate under some circumstances.
 
 Pros:
 - the paper has a wide-ranging discussion connecting extrapolation in regression problems to adjacent fields of system identification and causal learning.
 - the method is sensible enough, and should probably be a baseline in the time-series literature. It also seems like an advance on the hard-to-optimize Eureqa method.
 
 Cons:
 I agree with the authors that Reviewer 5's comments aren't very helpful, but this paper really does ignore or dismiss a lot of recent progress and related methods. Specifically:
 - The authors claim that cross-validation can't be used to choose the model, since it wouldn't encourage extrapolation - but why not partition the data in contiguous chunks, as is done in time-series methods?
 - The authors introduce an annealing trick to help with the L1 objective, but there is a rich literature on gradient-based optimization methods with L1 regularization that address exactly this problem.
 - The authors mostly consider toy data, limiting the potential impact of their method.
 - The authors don't compare against closely related methods developed to address the exact same setting. Namely, Schmit + Lipson's Eureqa method, and the Gaussian process methods of Duvenaud, Lloyd, Grosse, Tenenbaum and Ghahramani.
 - The authors invent their own ad-hoc model-selection procedure, again ignoring a massive literature.
 
 Given the many "cons", it is recommended that this paper not be presented at the conference track, but be featured at the workshop track.
Decision: Invite to Workshop Track