Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a method for symbolic superoptimization — the task of simplifying equations into equivalent expressions. The main goal is to design a method that does not rely on human input in defining equivalence classes, which should improve scalability of the simplification method to a larger set of expressions. The solution uses a reinforcement learning method for training a neural model that transforms an equation tree into a simpler but equivalent one. The model consists of (i) a tree encoder, a recursive LSTM that operates over the input equation tree, (ii) a sub-tree selector, a probability distribution over the nodes in the input equation tree, and (iii) a tree decoder, a two layer LSTM that includes a tree layer and a symbol generation layer. The RL reward uses an existing method for determining soft equivalence between the output tree and the input tree along with a positive score for compressing. 

The main strengths of the paper are that (i) it targets the scalability problem in simplifying arithmetic expressions by reducing the amount of human effort involved in the process, (ii) it provides some evaluation comparing against methods that use pre-defined rules for transforming equations, (iii) it provides a baseline method against which future scalable models can be compared against.

The following are the main concerns I have with the paper

1) The model description should include more details about the design choices. 
For instance, the specific reward function, the central component of the model, is left rather under-discussed. The function form is sensible but why does the negative case use -0.1 as the reward scaling constant? Why not some other number? How is this tuned? 

2) As far as I can see there is no real ablation analysis that shows which of the components are actually useful. Is the sub-tree selector necessary? Is the curriculum training necessary? How much does the sub-tree embedding similarity loss contribute to the results? Even if each of these actually add value it will be useful to know how much. What about other design choices? If we trained a direct seq2seq model with linearized expressions instead of the tree structured inputs, would it work just as well? These are empirical questions that need to be answered to justify that the proposed model indeed is useful. 

3) The experimental details are sparse. In particular, there is no mention of how hyper-parameters of the proposed method are tuned. Are the performance numbers averages over a set of random seeds or is it simply the best performing number that has been reported? This is especially troublesome for a RL based model. 	

Overall the paper presents a particular model and strategy for training but lacks appropriate experimentation to establish their utility. 


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors present a framework for symbolic superoptimization using methods from deep learning. A deep learning approach operating on the expression tree structures is proposed based on a combination of subtree embeddings, LSTM RNN structures, and an attention mechanism. 

The approach avoids the exploitation of human-generated equivalence pairs thus avoiding human interaction and corresponding bias. Instead, the approach is trained using random generated data. It remains somewhat unclear how the corresponding random data generation influences general applicability w.r.t. other tasks, as the authors apply constraints on the generation process for complexity reasons. A corresponding discussion would be valuable here.

In Secs. 3 & 4, the authors present their specific modeling and learning approach. However, they do not report on modeling or learning alternatives. It would be interesting for the audience to understand, how the authors reached these specific choices, and how (some of) these choice influence performance and learning stability. For example, in Sec. 4.1, an additional loss term is introduced to further support the learning of embeddings. However, it might interesting to see comparative results quantitatively investigating the effect of this additional loss term. Also, as far as I can see, no information on the choice of hyperparameters (e.g. LSTM dimensions) are provided or analyzed w.r.t. their effect on the performance of the proposed approach.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper provides a novel approach to the problem of simplifying symbolic expressions without relying on human input and information. To achieve this, they apply a REINFORCE framework with a reward function involving the number of symbols in the final output together with a probabilistic testing scheme to determine equivalence. The model itself consists of a tree-LSTM-based encoder-decoder module with attention, together with a sub tree selector. Their main contribution is that this framework works entirely independent of human-labelled data. In their experiments, they show that this deep learning approach outperforms their provided human-independent baselines, while sharing similar performance with human-dependent ones.
Overall, the work in this paper has the potential to be a contribution to ICLR but lacks completeness and clarity. A number of experimental details are missing, making it difficult to understand the setup under which results were obtained. Moreover, the paper does not seem to have been revised with many grammatical issues that make it hard to read.
The following are major issues within the paper and should be addressed:
•	The paper does not mention the amount of compute given to their model, nor the amount of time taken to train. As the REINFORCE framework is generally quite computation-heavy, these are significant details. Without assessing the amount of compute and time allotted for training HISS, the comparisons to previous baselines lose a fair amount of meaning. The paper alludes to processes being ‘extremely time consuming’, but then does not provide any numbers.
•	They do not mention the data used to train the model weights. In the comparisons sections, some details on datasets are given, but these seem to refer to data for inference.
•	There are many grammatical errors that likely could have been detected with 	 revision. A handful of such errors would not affect the score, but they are so numerous as to make the paper much more difficult to understand.
Additionally, these are comments that slightly detract from the quality of the paper:
•	It’s unclear what to glean from Section 5.1, as the dataset and baselines seem to be fairly trivial. If their claim is to have the first nontrivial human-independent approach to simplifying symbolic expressions, there is no need to compare to baselines that can only handle small expressions.
•	Sections 5.3 and 5.4 contribute little to the paper. For 5.3, the model was trained to embed equivalent expressions close together, using L2 regularization. It is therefore unsurprising that equivalent expressions are then closer together than non-equivalent ones. The paper also does not provide a comparison to the method without this regularization, and so it’s unclear if this embedding similarity helps in any way.  For 5.4, the section is extremely short and contains very little content. Moreover, just as many of the variables in their provided examples oppose their conjectures as support them.
•	The most interesting figure provided is the rewrite rules discovered by the model. It would be even better if an additional column containing the rules discovered by Halide (the main baseline) were provided.
Overall, in my understanding, the primary point in favor of the paper is in being the first nontrivial human-independent approach to simplifying symbolic expression. That said, this is not my area of expertise, so I cannot judge novelty or importance as well as other reviewers.

review_title: Official Blind Review #3

Meta-review: This work introduces a neural architecture and corresponding method for simplifying symbolic equations, which can be trained without requiring human input. This is an area somewhat outside most of our expertise, but the general consensus is that the paper is interesting and is an advance. The reviewer's concerns have been mostly resolved by the rebuttal, so I am recommending an accept. 
Decision: Accept (Poster)