Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper addresses the task of dealing with named entities in goal oriented dialog systems. Named entities, and rare words in general, are indeed troublesome since adding them to the dictionary is expensive, replacing them with coarse labels (ne_loc, unk) looses information, and so on. The proposed solution is to extend neural dialog models by introducing a named entity table, instantiated on the fly, where the keys are distributed representations of the dialog context and the values are the named entities themselves. The approach is applied to settings involving interacting to a database and a mechanism for handling the interaction is proposed. The resulting model is illustrated on a few goal-oriented dialog tasks.


I found the paper difficult to read. The concrete mappings used to create the NE keys and attention keys are missing. Providing more structure to the text would also be useful vs. long, wordy paragraphs. Here are some specific questions:

1. How are the keys generated? That are the functions used? Does the "knowledge of the current user utterance" include the word itself? The authors should include the exact model specification, including for the HRED model.

2. According to the description, referring to an existing named entity must be done by "generating a key to match the keys in the NE table and then retrieve the corresponding value and use it". Is there a guarantee that a same named entity, appearing later in the dialog, will be given the same key?  Or are the keys for already found entities retrieved directly, by value?

3. In the decoding phase, how does the system decide whether to query the DB?

4. How is the model trained?

In its current form, it's not clear how the proposed approach tackles the shortcomings mentioned in the introduction. Furthermore, while the highlighted contribution is the named entity table, it is always used in conjunction to the database approach. This raises the question whether the named entity table can only work in this context.

For the structured QA task, there are 400 training examples, and 100 named entities. This means that the number of training examples per named entity is very small. Is that correct? If yes, then it's not very surprising that adding the named entities to the vocabulary leads to overfitting. Have you compared with using random embeddings for the named entities?

Typos: page 2, second-to-last paragraph: firs -> first, page 7, second to last paragraph: and and -> and


review_title: Poor presentation

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes to generate embedding of named-entities on the fly during dialogue sessions. If the text is from the user, a named entity recognizer is used. If it is from the bot response, then it is known which words are named entities therefore embedding can be constructed directly. The idea has some novelty and the results on several tasks attempting to prove its effectiveness against systems that handle named entities in a static way.

One thing I hope the author could provide more clarification is the use of NER. For example, the experimental result on structured QA task (section 3.1), where it states that the performance different between models of With-NE-Table and W/O-NE-Table is positioned on the OOV NEs not present in the training subset. To my understanding, because of the presence of the NER in the With-NE-Table model, you could directly do update to the NE embeddings and query from the DB using a combination of embedding and the NE words (as the paper does), whereas the W/O-NE-Table model cannot because of lack of the NER. This seems to prove that an NER is useful for tasks where DB queries are needed, rather than that the dynamic NE-Table construction is useful. You could use an NER for W/O-NE-Table and update the NE embeddings, and it should be as good as With-NE-Table model (and fairer to compare with too).

That said, overall the paper is a nice contribution to dialogue and QA system research by pointing out a simple way of handling named entities by dynamically updating their embeddings. It would be better if the paper could point out the importance of NER for user utterances, and the fact that using the knowledge of which words are NEs in dialogue models could help in tasks where DB queries are necessary.
review_title: Recognizing and Updating NE Embeddings on the Fly

final_score: rating: 3: Clear rejection
review_content: Properly capturing named entities for goal oriented dialog is essential, for instance location, time and cuisine for restaurant reservation. Mots successful approaches have argued for separate mechanism for NE captures, that rely on various hacks and tricks. This paper attempt to propose a comprehensive approach offers intriguing new ideas, but is too preliminary, both in the descriptions and experiments. 

The proposed methods and experiments are not understandable in the current way the paper is written: there is not a single equation, pseudo-code algorithm or pointer to real code to enable the reader to get a detailed understanding of the process. All we have a besides text is a small figure (figure 1). Then we have to trust the authors that on their modified dataset, the accuracies of the proposed method is around 100% while not using this method yields 0% accuracies?

The initial description (section 2)  leaves way too many unanswered questions:
- What embeddings are used for words detected as NE? Is it the same as the generated representation?
- What is the exact mechanism of generating a representation for NE EECS545? (end of page 2)
- Is it correct that the same representation stored in the NE table is used twice? (a) To retrieve the key (a vector) given the value (a string)  as the encoder input. (b) To find the value that best matches a key at the decoder stage?
- Exact description of the column attention mechanism: some similarity between a key embedding and embeddings representing each column? Multiplicative? Additive?
- How is the system supervised? Do we need to give the name of the column the Attention-Column-Query attention should focus on? Because of this unknown, I could not understand the experiment setup and data formatting!

The list goes on...

For such a complex architecture, the authors must try to analyze separate modules as much as possible. As neither the QA and the Babi tasks use the RNN dialog manager, while not start with something that only works at the sentence level

The Q&A task could be used to describe a simpler system with only a decoder accessing the DB table. Complexity for solving the Babi tasks could be added later.

review_title: Some good ideas, but lack of detailed explanation impacts understanding

Meta-review: This work deals with the important task of capturing named entities in a goal-directed setting. The description of the work and the experiments are not ready for publication; for example, it is unclear whether the proposed method would have an advantage over existing methods such as the match type features that are only mentioned in Table 3 for establishing the baseline on the original bAbI dialogue dataset, but not even discussed in the paper.
Decision: Reject