Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Two questions:

1. It seems very hard to generate a dataset of ~200k programs that have all the variability that one would expect to find in real source code. Thus, it would be desirable to use models that exhibit strong generalization (i.e., they generalize well even when test data is systematically different from training data). For example, one might hope that even if the learned static analyzer were run on a program that was much longer and used more identifiers than any program seen during training, the system would still perform well. Have you considered evaluating models in terms of stronger types of generalization?

2. How is the threshold used to color figure 3 chosen? Was it chosen to optimize performance on that example? Have you considered building a quantitative evaluation for that component?


Also, you may be interested in the following citation, which also uses a trained language model to localize errors in code:
Campbell, Joshua Charles, Abram Hindle, and José Nelson Amaral. "Syntax errors just aren't natural: improving error reporting with language models." Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, 2014.
review_title: Evaluating strong generalization?

final_score: rating: 3: Clear rejection
review_content: The authors are trying to understand whether static analysis can be learned. As I hinted in my question, I think that all of the interesting complexity of static analysis has been removed in the toy language --- extraordinarily simple logic using a set can solve the problem posed, and an LSTM (unsurprisingly) can learn the extraordinarily simple logic (when given a differentiable set object). This extreme simplicity gives me no confidence that a more realistic static analysis problem can be solved.

LSTMs (and deep learning) have had remarkable successes in solving messy real-world language problems. It's certainly possible that LSTMs could solve static analysis -- but being technically timid is not the right way to go about it.
review_title: Too toy: doesn't effectively prove a point

final_score: rating: null
review_content: The paper is very clear, I don't have any questions.
review_title: No questions, just making this message go away.

final_score: rating: null
review_content: Given a set data structure, isn't the task of finding uninitialized variables fairly easy in the proposed language? You chase through the code paths and put initialized variables in the set, and check the used variables against the set.

Why is doing this with an LSTM interesting? It's just some simple logic... unless the LSTM is doing something with modeling the variable values (i.e., certain if statements could be mutually exclusive).. Not clear why this teaches us anything about LSTMs in more complex situations.
review_title: Isn't using a differentiable set "cheating" in some sense?

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper takes a first step towards learning to statically analyze source code. It develops a simple toy programming language that includes loops and branching. The aim is to determine whether all variables in the program are defined before they are used. The paper tries a variety of off-the-shelf sequence classification models and develops a new model that makes use of a ``differentiable set'' to keep track of which variables have been defined so far. Result show that an LSTM model can achieve 98% accuracy, and the differentiable set model can achieve 99.3% accuracy with sequence-level supervision and 99.7% accuracy with strong token-level supervision. An additional result is used whereby an LSTM language model is trained over correct code, and then low probability (where a threshold to determine low is tuned by hand) tokens are highlighted as sources of possible error.

One further question is if the authors could clarify what reasoning patterns are needed to solve these problems. Does the model need to, e.g., statically determine whether an `if` condition can ever evaluate to true in order to solve these tasks? Or is it just as simple as checking whether a variable appears on a LHS before it appears on a RHS later in the textual representation of the program?

Strengths:
- Learning a static analyzer is an interesting concept, and I think there is good potential for this line of work
- The ability to determine whether variables are defined before they are used is certainly a prerequisite for more complicated static analysis.
- The experimental setup seems reasonable
- The differentiable set seems like a useful (albeit simple) modelling tool

Weaknesses:
- The setup is very toy, and it's not clear to me that this makes much progress towards the challenges that would arise if one were trying to learn a static analyzer 
- The models are mostly very simple. The one novelty on the modelling front (the differentiable set) provides a small win on this task, but it's not clear if it is a useful general construct or not.

Overall:
I think it's an interesting start, and I'm eager to see how this line of work progresses. In my opinion, it's a bit too early to accept this work to ICLR, but I'd be excited about seeing what happens as the authors try to push the system to learn to analyze more properties of code, and as they push towards scenarios where the learned static analyzer would be useful, perhaps leveraging strengths of machine learning that are not available to standard programming languages analyses.
review_title: Interesting start, but I think not quite ready for publication

final_score: rating: 3: Clear rejection
review_content: The authors explore the idea of deep-learning a static analyzer. They do it with a toy programming language and a very simplified analysis problem -- just checking if all variables are initalized.

While the idea is interesting and might be developped into a tool in the future, the toy task presented in this paper is too simple to warrant an ICLR submission. Just detecting whether a variable is initialized in a string is a toy algorihtmic task, similar to the ones solved in a number of paper in recent years by models such as the Neural Turing Machine, Stack RNNs, Neural GPU, or Differentiable Neural Computer. All these architectures perform almost perfectly on a number of algorithmic tasks, so it is highly probable that they would also solve this one. Unluckily, the authors only compare to much more basic models, such as HMMs. Since the code for many of the above-mentioned models is available online, a paper without these baselines is not ready for ICLR. Moreover, there is a risk that existing models already solve this problem very well, making the contribution unclear.
review_title: Fine idea but small-scale and lacks crucial baselines

Meta-review: There is a general consensus that, though the idea is interesting, the work is not mature enough for a conference publication (e.g., the problem is too toy, not clear that really solves any, even artificial problem, better than existing techniques).
Decision: Reject