Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper uses neural networks to answer falling body physics questions by 1. Resolving the parameters of the problem, and 2. Figure out which quantity is in question, compute it using a numerical integrator and return it as an answer.
Learning and inference are performed on artificially generated questions using a probabilistic grammar.
Overall, the paper is clearly written and seems to be novel in its approach.

The main problems I see with this work are:
1. The task is artificial, and it's not clear how hard it is. The authors provide no baseline nor do they compare it to any real world problem. Without some measure of difficulty it's hard to tell if a much simple approach will do better, or if the task even makes sense.
2. The labler LSTM uses only 10 hidden units. This is remarkably small for language modeling problems, and makes one further wonder about the difficulty of the task. The authors provide no reasoning for this choice.

review_title: A nice approach to this problem, but inputs seem too artificial

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper build a language-based solver for simple physics problems (a free falling object under constant velocity). Given a natural language query sampled from a fixed grammar, the system uses two LSTM models to extract key components, e.g., physical parameters and the type of questions being asked, which are then sent to a numerical integrator for the answer. The overall performance in the test set is almost perfect (99.8%).

Overall I found this paper quite interesting to read (and it is well written). However, it is not clear how hard the problem is and how much this approach could generalize over more realistic (and complicated) situations. The dataset are a bit small and might not cover the query space. It might be better to ask AMT workers to come up with more complicated queries/answers. The physics itself is also quite easy. What happens if we apply the same idea on billiards? In this case, even we have a perfect physics simulator, the question to be asked could be very deep and requires multi-hop reasoning.

Finally, given the same problem setting (physics solver), in my opinion, a more interesting direction is to study how DNN can take the place of numerical integrator and gives rough answers to the question (i.e., intuitive physics). It is a bit disappointing to see that DNN is only used to extract the parameters while still a traditional approach is used for core reasoning part. It would be more interesting to see the other way round.

review_title: An interesting paper to read but could be made better

final_score: rating: null
review_content: Could the authors elaborate on how the dataset is generated? Although the grammar is given in appendix, the generation process is not mentioned in details. I wonder whether there is any bias in training/val/test. Authors could show distributions of different types of questions, etc. 

BTW, it seems that the problem will be more interesting if the numerical integrator is replaced by a DNN. Currently the core of reasoning is still done by manual designed procedure. I wonder whether the authors have tried on this direction. 

review_title: How the dataset is generated?

final_score: rating: null
review_content: Have the authors tried using a single LSTM to perform both labeling and classification? Also, is the model they use bidirectional?

For the analysis section, did the authors perform any analysis to determine the importance of each input word for classification?
review_title: Comparison w/ single network + attention

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors describe a system for solving physics word problems. The system consists of two neural networks: a labeler and a classifier, followed by a numerical integrator. On the dataset that the authors synthesize, the full system attains near full performance. Outside of the pipeline, the authors also provide some network activation visualizations.

The paper is clear, and the data generation procedure/grammar is rich and interesting. However, overall the system is not well motivated. Why did they consider this particular problem domain, and what challenges did they specifically hope to address? Is it the ability to label sequences using LSTM networks, or the ability to classify what is being asked for in the question? This has already been illustrated, for example, by work on POS tagging and by memory networks for the babi tasks. A couple of standard architectural modifications, i.e. bi-directionality and a content-based attention mechanism, were also not considered.
review_title: Rich data generation procedure but system specific and not well motivated

final_score: rating: null
review_content: From the text, it's hard to understand how hard the task is. 

Here are a few ways to answer this:
- Compare the perplexity of the grammar to real world questions of the same nature.
- Run the system on real examples.
- Compare to previous method, or to some simple baseline. 
review_title: How hard is the task?

Meta-review: The paper explores a model for solving simple physics problems (posed in automatically generated natural language). Whilst this an interesting problem, the reviewers worry that the problem is too simple because all problems are automatically generated. The paper should at least incorporate some reasonable baseline models and/or apply the proposed methodology on real physics problems.
Decision: Reject