Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

This paper presents a dataset of mathematical equations and applies TreeLSTMs to two tasks: verifying and completing mathematical equations. For these tasks, TreeLSTMs outperform TreeNNs and RNNs. In my opinion, the main contribution of this paper is this potentially useful dataset, as well as an interesting way of representing fixed-precision floats. However, the application of TreeNNs and TreeLSTMs is rather straight-forward, so in my (subjective) view there are only a few insights salvageable for the ICLR community and compared to Allamanis et al. (2017) this paper is a rather incremental extension.

Strengths

The authors present a new datasets for mathematical identities. The method for generating additional correct identities could be useful for future research in this area.
I find the representation of fixed-precision floats presented in this paper intriguing. I believe this contribution should be emphasized more as it allows the model to generalize to unseen numbers and I am wondering whether the authors see some wider application of this representation for neural programming models.
I liked the categorization of the related work.

Weaknesses

p2: It is mentioned that the framework is the first to combine symbolic expressions with black-box function evaluations, but I would argue that Neural Programmer-Interpreters (NPI; Reed & De Freitas) are already doing that (see Fig 1 in that paper where the execution trace is a symbolic expression and some expressions "Act(LEFT)" are black-box function applications directly changing the image).
The differences to Allamanis et al. (2017) are not worked out well. For instance, the authors use the TreeNN model from that paper as a baseline but the EqNet model is not mentioned at all. The obvious question is whether EqNets can be applied to the two tasks (verifying and completing mathematical equations) and if so why this has not been done.
The contribution regarding black box function application is unclear to me. On page 6, it is unclear to me what "handles […] function evaluation expressions". As far as I understand, the TreeLSTM learns to the return value of function evaluation expressions in order to predict equality of equations, but this should be clarified.
I find the connection of the proposed model and task to "neural programming" weak. For instance, as far as I understand there is no support for stateful programs. Furthermore, it would be interesting to hear how this work can be applied to existing programming languages such as Haskell. What are the limitations of the architecture? Could it learn to identify equality of two lists in Haskell?
p6: The paragraph on baseline models is rather uninformative. TreeLSTMs have been shown to outperform Tree NN's in various prior work. The statement that "LSTM cell […] helps the model to have a better understanding of the underlying functions in the domain" is vague. LSTM cells compared to fully-connected layers in Tree NNs ameliorate vanishing and exploding gradients along paths in the tree. Furthermore, I would like to see a qualitative analysis of the reasoning capabilities that are mentioned here. Did you observe any systematic differences in the ~4% of equations where the TreeLSTM fails to generalize (Table 3; first column).

Minor Comments

Abstract: "Our framework generalizes significantly better" I think it would be good to already mention in comparison to what this statement is.
p1: "aim to solve tasks such as learn mathematical" -> "aim to solve tasks such as learning mathematical"
p2: You could add a citation for Theano, Tensorflow and Mxnet.
p2: Could you elaborate how equation completion is used in Mathematical Q&A?
p3: Could you expand on "mathematical equation verification and completion […] has broader applicability" by maybe giving some concrete examples.
p3 Eq. 5: What precision do you consider? Two digits?
p3: "division because that they can" -> "division because they can"
p4 Fig. 1: Is there a reason 1 is represented as 10^0 here? Do you need the distinction between 1 (the integer) and 1.0 (the float)?
p5: "we include set of changes" -> "we include the set of changes"
p5: In my view there is enough space to move appendix A to section 2. In addition, it would be great to see more examples of generated identities at this stage (including negative ones).
p5: "We generate all possible equations (with high probability)" – what is probabilistic about this?
p5: I don't understand why function evaluation results in identities of depth 2 and 3. Is it both or one of them?
p6: The modules "symbol" and "number" are not shown in the figure. I assume they refer to projections using Wsymb and Wnum?
p6: "tree structures neural networks" -> "tree structured neural networks"
p6: A reference for the ADAM optimizer should be added.
p6: Which method was used for optimizing these hyperparameters? If a grid search was used, what intervals were used?
p7: "the superiority of Tree LSTM to Tree NN shows that is important to incorporate cells that have memory" is not a novel insight.
p8: When you mention "you give this set of equations to the models look at the top k predictions" I assume you ranked the substituted equations by the probability that the respective model assigns to it?
p8: Do you have an intuition why prediction function evaluations for "cos" seem to plateau certain points? Furthermore, it would be interesting to see what effect the choice of non-linearity on the output of the TreeLSTM has on how accurately it can learn to evaluate functions. For instance, one could replace the tanh with cos and might expect that the model has now an easy time to learn to evaluate cos(x).
p8 Fig 4b; p9: Relating to the question regarding plateaus in the function evaluation: "in Figure 4b […] the top prediction (0.28) is the correct value for tan with precision 2, but even other predictions are quite close" – they are all the same and this bad, right?
p9: "of the state-of-the-art neural reasoning systems" is very broad and in my opinion misleading too. First, there are other reasoning tasks (machine reading/Q&A, Visual Q&A, knowledge base inference etc.) too and it is not obvious how ideas from this paper translate to these domains. Second, for other tasks TreeLSTMs are likely not state-of-the-art (see for example models on the SQuAD leaderboard: https://rajpurkar.github.io/SQuAD-explorer/) .
p9: "exploring recent neural models that explicitly use memory cells" – I think what you mean is models with addressable differentiable memory.

# Update after the rebuttal
Thank you for the in-depth response and clarifications. I am increasing my score by one point. I have looked at the revised paper and I strongly suggest that you add the clarifications and in particular comments regarding comparison to related work (NPI, EqNet etc) to the paper. Regarding Fig. 4b, I am still not sure why all scores are the same (0.9977) -- I assume this is not the desired behavior?
review_title: A dataset paper with application of TreeLSTMs

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a model that predicts the validity of a mathematical expression (containing trigonometric or elementary algebraic expressions) using a recursive neural network (TreeLSTM).  The idea is to take the parse tree of the expression, which is converted to the recursive neural network architecture, where weights are tied to the function or symbol used at that node.  Evaluation is performed on a dataset generated specifically for this paper.

The overall approach described in this paper is technically sound and there are probably some applications (for example in online education).  However the novelty factor of this paper is fairly low — recursive neural nets have been applied to code/equations before in similar models.  See, for example, “Learning program embeddings to propagate feedback on student code” by Piech et al, which propose a somewhat more complex model applied to abstract syntax trees of student written code.

I’m also not completely sure what to make of the experimental results.  One weird thing is that the performance does not seem to drop off for the models as depth grows.  Another strange thing is that the accuracies reported do not seem to divide the reported test set sizes (see, e.g., the depth 1 row in Table 2).  It would also be good to discuss the Sympy baseline a bit — being symbolic, my original impression was that it would be perfect all the time (if slow), but that doesn’t seem to be the case, so some explanation about what exactly was done here would help.  For the extrapolation evaluation — evaluating on deeper expressions than were in the training set — I would have liked the authors to be more ambitious and see how deep they could go (given, say, up to depth 3 training equations).  


review_title: A review of "Combining  Symbolic Expressions..."

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: SUMMARY 

The model evaluates symbolic algebraic/trigonometric equalities for validity, with an output unit for validity level at the root of a tree of LSTM nodes feeding up to the root; the structure of the tree matches the parse tree of the input equation and the type of LSTM cell at each node matches the symbol at that node in the equation: there is a different cell type for each symbol. It is these cell types that are learned. The training data includes labeled true and false algebraic/trigonometric identities (stated over symbols for variables) as well as function-evaluation equalities such as "tan(0.28) = 0.29" and decimal-expansion equations like "0.29 = 2*10^(-1) + 9*10^(-2)".  I believe continuous values like "0.29" in the preceding expressions are encoded as the literal value of a single unit (feeding into an embedding unit of type W_{num}), whereas the symbols proper (including digit numerals) are encoded as 1-hot vectors (feeding into an embedding unit of type W_{symb}).
Performance is at least 97% when testing on unseen expressions of the same depth (up to 4) as the training data. Performance when trained on 3 levels (among 1 - 4) and testing on generalization to the held-out level is at least 96% when level 2 is held out, at least 92% when level 4 is withheld. Performance degrades (even on symbolic identities) when the function-evaluation equalities are omitted, and degrades when LSTM cells are replaced by plain RNN cells. The largest degradation is when the tree structure is replaced (presumably) by a sequence structure.
Performance was also tested on a fill-in-the-blank test, where a symbol from a correct equation was removed and all possible replacements for that symbol with expressions of depth up to 2 were tested, then ranked by the resulting validity score from the model. From the graph it looks like an accuracy of about 95% was achieved for the 1-best substituted expression (accuracy was about 32% for a sequential LSTM).

WEAKNESSES

* The title is misleading; "blackbox function evaluation" does not suggest what is intended, which is training on function-evaluation equations. The actual work is more interesting than what the title suggests.
* The biggest performance boost (roughly 15%) arises from use of the tree structure, which is given by an oracle (implemented in a symbolic expression parser, presumably): the network does not design its own example-dependent structure.
* What does the sympy baseline mean in Table 2? We are only told that sympy is a "symbolic solver". Yet the sympy performance scores are in the 70-80% range. If the solver’s performance is that weak, why is it used during generation of training data to determine the validity of possible equations?
* Given that this is a conference on "learning representations" it would have been nice to see at least a *little* examination of the learned representations. It would be easy to do some interesting tests. How well does the vector embedding for "2*10^(-1) + 9*10^(-2)" match the vector for the real value 0.29? W_{num} embeds a continuum of real values in R^d: what is this 1-dimensional embedding manifold like? How do the embeddings of different integers provided by W_{sym} relate to one another? My rating would have been higher had there been some analysis of the learned representations.
* We are told only that the "hidden dimension … varies"; it would be nice if the text or results tables gave at least some idea of what magnitude of embedding dimension we’re talking about.

STRENGTHS

The weaknesses above notwithstanding, this is a very interesting piece of work with impressive results. 
* The number of functions learned, 28, is a quantum jump from previous studies using 8 or fewer functions.
* It is good to see the power of training the same system to learn the semantics of functions from the equations they satisfy AND from the values they produce. 
* The inclusion of decimal-expansion equations for relating numeral embeddings to number embeddings is clever. 
* The general method used for randomly generating a non-negligible proportion of true equations is useful.
* The evaluation of the model is thorough and clear.
* In fact the exposition in the paper as a whole is very clear.
review_title: Impressive model for learning to encode symbols and numbers, and to evaluate functions, for determining the validity of algebraic/trigonometric equalities

Meta-review: Learn to complete an equation by filling the blank with a missing function or numeral, and also to evaluate an expression.  Along the way learn to determine if an identity holds (e.g. sin^2(x) + cos^2(x) = 1).  They use a TreeNN with a separate node for each expression in the grammar.

PROS:
1. They've put together a new dataset of equational expressions for learning to complete an equation by filling in the blank of a missing function (or value) and function evaluation.   They've done this in a nice way with a generator and will release it.

2. They've got two interesting ideas here and they seem to work.  First, they train the network to jointly learn to manipulate symbols and to evaluate them.  This helps ground the symbolic manipulations in the validity of their evaluations.  They do this by using a common tree net for both processes with both a symbol node and a number node.  They train on identities (sin^2(x) + cos^2(x) = 1) and also on ground expressions (+(1,2) = 3).  The second idea is to help the system learn the interpretation map for the numerals like the "2" in "cos^2(x) with the actual number 2.  They do this by including equations which relate decimals with their base 10 expansion.  For example 2.5 = 2*10^0 + 5*10^-1.  The "2.5" is (I think) treated as a number and handled by the number node in the network.  The RHS leaves are treated as symbols and handled by the symbol node of the network. This lets them learn to represent decimals using just the 10 digits in their grammar and ties the interpretation of the symbols to what is required for a correct evaluation (in terms of their model this means "aligning" the node for symbol with the node for number).

3. Results are good over what seem to us reasonable baselines

CONS:

1. The architecture isn't new and the idea of representing expression trees in a hierarchical network isn't new either.

2. The writing, to me, is a bit unclear in places and I think they still have some work to do follow the reviewers' advice in this area.

I really wrestled with this one, and I appreciate the arguments that say it's not novel enough but I feel that there is something interesting in here and if the authors do a clean-up before final submission it will be ok.
Decision: Accept (Poster)