Task: Language Identification
Input (Spanish, English): Summary: The paper proposes a neural model for predicting Python syntax trees from text descriptions. Guided by the actual Python grammar, the model generates tree nodes sequentially in a depth-first fashion. Key ideas include injecting the information from the parent node as part of the LSTM input, a pointer network for copying the terminals, and unary closure which collapses chains of unary productions to reduce the tree size. The model is evaluated on three datasets from different domains and outperforms almost all previous work.  Strengths:  The paper is overall very well-written. The explanation of system is clear, and the analysis is thorough.  The system itself is a natural extension of various ideas. The most similar work include tree-based generation with parent feeding (Dong and Lapata, 2016) and various RNN-based semantic parsing with copy mechanism (Jia and Liang, 2016; Ling et al., 2016). [The guidance of parsing based on grammar is also explored in Chen Liang et al., 2016 ([Link] where a code-assist system is used to ensure that the code is valid.] Nevertheless, the model is this paper stands out as it is able to generate much longer and more complex programs than most previous work mentioned.   Weaknesses:  The evaluation is done on code accuracy (exact match) and BLEU score. These metrics (especially BLEU) might not be the best metrics for evaluating the correctness of programs. For instance, the first example in Table 5 shows that while the first two lines in boxes A and B are different, they have the same semantics. Another example is that variable names can be different. Evaluation based on what the code does (e.g., using test cases or static code analysis) would be more convincing.  Another point about evaluation: other systems (e.g., NMT baseline) may generate code with syntactic error. Would it be possible to include the result on the highest-scoring well-formed code (e.g., using beam search) that these baseline systems generate? This would give a fairer comparison since these system can choose to prune malformed code.  General Discussion:  * Lines 120-121: some approaches that use domain-specific languages were also guided by a grammar. One example is Berant and Liang, 2014, which uses a pretty limited grammar for logical forms (Table 1). In addition to comparing to that line of work, emphasizing that the grammar in this paper is much larger than most previous work would make this work stronger.  * Lines 389-397: For the parent feeding mechanism, is the child index being used? In other words, is p_t different when generating a first child versus a second child? In Seq2Tree (Dong and Lapata, 2016) the two non-terminals would have different hidden states.  * Line 373: Are the possible tokens embedded? Is it assumed that the set of possible tokens is known beforehand?  * The examples in the appendix are nice.  ---  I have read the author response.
Output (English): en