This paper introduces pointer-network neural networks, which are applied to referring expressions in three small-scale language modeling tasks: dialogue modeling, recipe modeling and news article modeling. When conditioned on the co-reference chain, the proposed models outperform standard sequence-to-sequence models with attention.  The proposed models are essentially variants of pointer networks with copy mechanisms (Gulcehre et al., 2016; Gu et al., 2016; Ling et al., 2016), which have been modified to take into account reference chains. As such, the main architectural novelty lies in 1) restricting the pointer mechanism to focus on co-referenced entities, 2) applying pointer mechanism to 2D arrays (tables), and 3) training with supervised alignments. Although useful in practice, these are minor contributions from an architectural perspective.  The empirical contributions are centred around measuring perplexity on the three language modeling tasks. Measuring perplexity is typical for standard language modeling tasks, but is really an unreliable proxy for dialogue modeling and recipe generation performance. In addition to this, both the dialogue and recipe tasks are tiny compared to standard language modeling tasks. This makes it difficult to evaluate the impact of the dialogue and recipe modeling results. For example, if one was to bootstrap from a larger corpus, it seems likely that a standard sequence-to-sequence model with attention would yield performance comparable to the proposed models (with enough data, the attention mechanism could learn to align referring entities by itself). The language modeling task on news article (Gigaword) seems to yield the most conclusive results. However, the dataset for this task is non-standard and results are provided for only a single baseline. Overall, this limits the conclusions we can draw from the empirical experiments.   Finally, the paper itself contains many errors, including mathematical errors, grammatical errors and typos: - Eq. (1) is missing a sum over $z_i$. - "into the a decoder LSTM" -> "into the decoder LSTM" - "denoted as his" -> "denoted as" - "Surprising," -> "Surprisingly," - "torkens" -> "tokens" - "if follows that the next token" -> "the next token" - In the "COREFERENCE BASED LANGUAGE MODEL" sub-section, what does $M$ denote? - In the sentence: "The attribute of each column is denoted as $s_c, where $c$ is the c-th attribute". For these definitions to be make sense, $s_c$ has to be a one-hot vector. If yes, please clarify this in the text. - "the weighted sum is performed" -> "the weighted sum is computed" - "a attribute" -> "an attribute" - In the paragraph on Pointer Switch, change $p(z_{i,v} |s_{i,v}) = 1$ -> $p(z_{i,v} |s_{i,v}) = 0$. - In the "Table Pointer" paragraph, I assume you mean outer-product instead of cross-product? Otherwise, I don't see how the equations add up.   Other comments: - For the "Attention based decoder", is the attention computed using the word embeddings themselves or the hidden states of the sentence encoder? Also, it applied only to the previous turn of the dialogue or to the entire dialogue history? Please clarify this. - What's the advantage of using an "Entity state update" rule, compared to a pointer network or copy network, which you used in the dialogue and recipe tasks? Please elaborate on this. - In the Related Work section, the following sentence is not quite accurate: "For the task oriented dialogues, most of them embed the seq2seq model in traditional dialogue systems while our model queries the database directly.". There are task-oriented dialogue models which do query databases during natural language generation. See, for example, "A Network-based End-to-End Trainable Task-oriented Dialogue System" by Wen et al.
Output: en