RGTI:Response generation via templates integration for End to End dialog
Authors: Yuxin Zhang
Songyan Liu
Venue:  (2020)
Keywords: End-to-end dialogue systems
transformer
pointer-generate network

Abstract: End-to-end models have achieved considerable success in task-oriented dialogue area, but suffer from the challenges of (a) poor semantic control, and (b) little interaction with auxiliary information. In this paper, we propose a novel yet simple end-to-end model for response generation via mixed templates, which can address above challenges. 
In our model, we retrieval candidate responses which contain abundant syntactic and sequence information by dialogue semantic information related to dialogue history. Then, we exploit candidate response attention to get templates which should be mentioned in response. Our model can integrate multi template information to guide the decoder module how to generate response better. We show that our proposed model learns useful templates information, which improves the performance of "how to say" and "what to say" in response generation. Experiments on the large-scale Multiwoz dataset demonstrate the effectiveness of our proposed model, which attain the state-of-the-art performance.

Under review as a conference paper at ICLR 2020
RGTI: GENERATION VIA TEMPLATES INTE- GRATION FOR END TO END DIALOG
Anonymous authors Paper under double-blind review
ABSTRACT
End-to-end models have achieved considerable success in task-oriented dialogue area, but suffer from the challenges of (a) poor semantic control, and (b) little in- teraction with auxiliary information. In this paper, we propose a novel yet simple end-to-end model for response generation via mixed templates, which can address above challenges. In our model, we retrieval candidate responses which contain abundant syntactic and sequence information by dialogue semantic information related to dialogue history. Then, we exploit candidate response attention to get templates which should be mentioned in response. Our model can integrate multi template information to guide the decoder module how to generate response better. We show that our proposed model learns useful templates information, which im- proves the performance of "how to say" and "what to say" in response generation. Experiments on the large-scale Multiwoz dataset demonstrate the effectiveness our proposed model, which attain the state-of-the of-the------ performance.
INTRODUCTION
Task-oriented dialogue is aim to help users to complete task in specific field such as restaurant reservation, or booking film tickets. The traditional approach is design pipeline architectures which have several modules: natural language understanding, dialogue manager and natural lan- guage generation (Wen et 2017). It's easy control, but the dialogue system becomes more and more complicated. With the development of deep learning, end-to-end methods have shown hope- ful results and received great attention in academic. They input user queries and generate system responses, which is relatively simple. However, the disadvantage is that end-to-end approaches are difficult to control generated results.
Task-oriented responses should have correct entities and grammatical expressions, which means solving problems of "what to say" and "how to say". If a user wants to ask for a restaurant of moderate price range, good task-oriented dialogue system should return the response with right restaurants whose prices cannot be high or low and use the proper wording which in clear and unambiguous expression. Researches get right entities by looking up the knowledge base(KB). Sukhbaatar et al. (2015) introduce KB in the form of hidden states. Madotto et (2018) use atten- tion and copy mechanism to produce words from KB. Most research on generating smooth response has been carried out in using templates. However, templates usually are designed by domain experts needs huge cost and is difficult transfer different domains. Wen et al. (2015) use Semantically Controlled LSTM (SC-LSTM) to control semantic results of language generation. [Su et al.) (2018) propose hierarchical architecture using linguistic patterns to improve response generation. These models though achieving good performance, suffer from the problems as templates fixed or poorly controllability of semantic.
in advance (Walker et al., (2007b)
In order to alleviate such issues, we propose an end-to-end response generation model via templates integration RGTI, which is composed a encoder encoding the retrieval candidate responses with the form of triple into template representation and a decoder that integrates templates to generate the target response. Instead of encoding the templates directly, we construct a hierarchical encoding structure to make the templates contain semantic information and sequence information. During the decoder phase, we exploit a mixed decoder via templates and dialogue history introducing copy mechanism to generate better response.

---

We now describe the RGTI framework composed of two parts: encoder for templates in triple form as well as dialogue history and mixed decoder generation and copy mode, as shown Figure 1. The dialogue history X = (r,,)... the expected output, where n.m are the corresponding lengths. We first retrieval relevant candidate Under review as a conference paper at ICLR 2020
We empirically show that RGTI achieve advanced performance using the triple encoder and mixed decoder. In the Human-human multi aulti-domain dialogue dataset Budzianowski et al.] (2018). RGTI is able to surpass the previous state-of-the art automatic evaluation, which further confirms the effectiveness of our proposed encoder-decoder model.
Dialog History
GET _.sarrccc
User: Would like moderate
pinase.
"query": {
System: I found de luca cucina and har and rivereide
does
sound good for you?
"query string":
Belief
"fields" : ["ser"]
gurns centtre moder european, price
tange-nto deraliel
"query": "I'm looking for cheap place to stay with free wifi
inform:
Osamen laca cucina and bar, ramecriversdd brasserie)
User Lets a table for on Monday at 16645
System Diay. that worked the will recarved
MJEOG3RS.
there anything else can do for you?
Belief sattee
(value count] results match your request- narrow it down what area are you looking stay in, and what price range?
food-modem
canger moderate)
Diviog Act restaurant- booking tnt-t--oonnn thock
have
any particular
User: I'm looking for cheap place stoy with free
found [value count) which pricerange) price range. they
pricerange) and [hotel area]
Belief
range internet-yes)
NEED TO PREDICT:
[hotel which do you prefer?
System: We howe 10 pomibiiriie foryou.Do you have only additional Proferences
ok. have several hotels particular location?
Figure I:
Dialogue Example.The The left part of the figure is the raw dialogue contexts, based on the above dialogue history obtain candidate reference responses which in the lower right part of the figure by retrieving the dialogue.
2
OUR MODEL
..)) is the input, and the system response Y (M1, .... Vm) is
responses to the target responses. Then the encoder block uses multi-head attention to encode the candidate responses in triple form into templates representation. Next, the copy augmented decoder uses gating mechanism for words selection from two sources: utterances and templates, while generating target response.
Dialog Corpus
Encoder Block
Attention Layer
Decoder Block
Response
Useri
We have 10 possibiltties
canen atene
and riveriid
Utterance
Sequence Decoder
for You. Do you any additional preferences
Attention
Encoder
10:00
Systericbey
novevvr retuutes
h1
Fueee dor
with phoopppioc
Template Decoder
Gate
Template Search
Template Encoder
Figure 2: The proposed architecture for task-oriented dilogue systems: RGTI.
2

---

Under review as a conference paper at ICLR 2020
2.1 RETRIEVAL REFERENCE RESPONSE
First we should retrieval reference responses from training sets. The benefit of this approach is we can get more semantically coherent responses. For getting the better reference responses, we use dialogue state, dialogue act and semantic information of responses search criteria. ElasticSearch been used, which can get faster and more acurrate results.
2.2 ENCODER BLOCK
We use multi-head attention to encode dialogue history X and retrieval reference responses R. which automatically extract important information from sentences. If we feed the retrieved sentences directly into the model, the noise is large and not conducive to model learning, so we assume the closet part the entity in the response the most important. We divide the entity and its most relevant parts into triple form (head, entity, tail). Head means the part of the sentence before the entity, and tail means the after part, for example if response is "enjoy your stay value-place, goodbye", the corresponding head is "enjoy your stay", entity is "value-place", tail "goodbye". Using this form, we can pay attention to word order and obtain the relationship between the sentence structure and the entity in the sentence. We treat the encoded retrieval responses as template Q, which is a refined expression of retrieval results.
2.3 MIXED DECODER
Vocabulary Distribution
Softrax
Generate_ dist P_gen
P_gen
MultiHead Attention Distribution
Liee
Add & Norm
Copy Distribution
Feed Forward
Add Norm
Copy_dist ((1-P_gon)
Mult-Head Attention
Multi-ited Attertion
Masked Multi-lead Attention
Final Distribution
Isput Embedding
Output Embedding
Figure 3: Mixed Decoder,
We predict tokens of target response based on two mixed modes, generate-mon mode and copy- Generate-mode is generate words from the vocabulary directly and copy mode is to copy words from templates. Accordingly, location copying layer. Then we use the gated mechanism of the above output layers to get the final generated words. The probability of generating target word yı is the sum of multiple probabilities at each time step t:
model uses two output layer: sequence prediction layer, template
P(velse Vt-1))
Ppr(uelse St. 11-1. 6e) Pm Pm(pr(sa 11-1)+
Peo Ut-1 Mq) Pm (ye, Ce yt-1)

---

Under review as a conference paper at ICLR 2020
Metric
DSTC2
WOZ2.0
KVRET
MultiWOZ
Dialogues
1612
600
2425
8438
Total turns
23,354
4,472
12,732
113,556
Total tokens
199,431
50,264
102,077
1,490,615
Avg turns per dialogue
14.19
7.45
5.25
13,46
Avg tokens per turn
8.54
11.25
8.02
13.13
total unique tokens
986
2,142
2,842
23689
Slots
8
24
Value
212
1847
1363
4510
Table 1: Statistics of and MultiWOZ, part of the data above is derived from Budzianowski et al. 2018
where pr is the predict-mode, and means copy words from template, Pm((1) shows the probability for choosing different modes. The probabilities of these modes are calculated as follows:
1
Ppr (Bcl))) (pr(())
Z
(yol)))))
1
Σ (ooolic)
Z
j=Q==M
where to is score function choose mode, Z stands for the normalization term of modes. Z = expr + Ex (
Specifically, the score functions of two modes are given by:
(Vu++ u;) = u.f Wrrlst, Ctem]
( = xz) DNN((,, 8t, histQ)
EXPERIMENTS
3.1 DATASET
To verify the results of our model, we use recently proposed MultWWOZ dataset (Budzianowski et al. 2018) to carry out experiments. The MultiWOZ the largest existing numan-human conversational corpus spanning over seven domains (attraction, hospital, police, hotel, restaurant, taxi, train), which contains 8438 multi-turn dialogues and the average length of each dialogue is 13.68. Different from current mainstream task-oriented dialogue datasets like WOZ2.0 (Wen et al., 2017).and DSTC2 which contain less than 10 slots and only a few hundred values. There are almost 30(domain,si) pairs and over 4500 possible values in MultiWOZ dataset. Each dialogue consists of dialogue goal and a representation of multiple pairs of users and system utterances. In each turn the dialogue, there still two kinds of annotations, one is belief state and the other is the dialog action which are used to mark the status of the current conversation and the potential actions of the user.
3.2 TRAINING DETAILS
We trained model end-to-end using Adam optimizer (Kingma & Ba, 2014).an .and multi- step learning rate with milestones in 50, 100, 150. 200, the learning rate starts from 1e-3 and the parameter y is 0.5. All the embeddings are initialized randomly, and beam-search strategy used during the decoding state. The tlppe--arameter such as hidden size is 512 and the dropout rate is 0.2.
3.3
EVALUATION
3.3.1 ENTITY F1
The main evaluation metric is Fl score which is the harmonic mean of precision and recall at word level between the predicted answer and ground truth. By comparing the ground truth system re- sponses with the set of entities to select useful entities, this metric evaluate the ability generate

---

Under review as a conference paper at ICLR 2020
utterance
interested that expenstiv price range.
template1
[resturaat_anee]
you
template2
the- (restaurant, aarea you reserve you
[restaurant_
would
to reservation
Truth
Irestaurant_
city freetaurant reservation
price range. would you
How one moderate prico range?
template1
you profer the-> [hotel Lanea]-> or the)
template2
you profer tho-> [hotol_rroa]->>
(value count) options
you
[hotel_area) or [notel_area] part
Ground Truth
we [elun] entries your preferences. you profer [notelareea] sreal, or [hotel_area]
Figure 4: Examples of RGTI generation result.
Model
BLEU
Entity F1
LSTM
Two-stage requence-to-sequenee Architecturee(SSquiciiy
Effectively Incorporating Knowledge Bases(Mem2Scq) Global-to-local Memory Pointer Networks (GLMP) Hierarchical Disentangled Self-Attention (HDSA)
Our proposed model (RGTI)
26.89%
Table 2: BLEU and Entity Fl score of baselines and proposed model (RGTT) on Multiwoz dataset. Retrained using docker container provided by the authors with exactly same
relevant entities and to capture the semantics of the dialog (Eric & Manning 2017) and (Eric et al. 2017
3.3.2 BLEU
We also use BLEU score in our evaluation which is often used to compute the word overlap between the generated output and the reference response. The early BLEU metries was used the field of translation In recent years, it has also referred to the BLEU metrics in the end-to-end task-oriented dialogue and in the field chat-bots.
3.4 ABLATION STUDY
We perform ablation experiments on the test set to analyze the effectiveness of different module in our model. The results of these experiments shown the Table below. As one can observe from the Table, our model without copy mechanism has 2.5% BIEU drop in generate results. On the other hand, RGTI without templates means that we do not consider the relevant templates information and thus lead to reduction in BLEU. Note that if we remove the templates, then 0.5 % increase can be observed the table, which suggests that instead of applying context history makes the corresponding text noise have more side effects than his positive influence.
RELATED WORKS
dialog systems are mainly explored by following two different approaches: piplines and end-to-end. For the dialogue systems of piplines (Williams Young, 2007; Wen et al., 2017), modules are separated into different trained modets: language understanding g (Young et al., 2013; Chen al., 2016), which is used to understand human intention, dialogue state tracking

---

Under review as a conference paper at ICLR 2020
(Lee Stent, 2016; Zhong al., 2018) which is for estimating user goal at every step of the dialogue, dialogue management (Su et al., 2016), and natural language generation (Sharma et al,,201) which aimed realize language surface form given the semantic constraint. These approaches achieve good stability via combining domain-specific knowledge and slot-filling techniques, but additional human labels needed. On the other hand, end-to-end approaches have shown promising results recently. BiLSTM Zhao et al. (2017) use recurrent neural networks to generate final responses and achieve good results the previous state-of-art model about memory mechanism strengthen the rea- soning ability by incorporating external knowledge into the neural network, Sequicity and HDSA represent the conversation history as belief span or dialog action which is used as compressed infor- mation for downstream model generate system response considering current dialog state.
4.1 TEMPLATE
Using templates guide response generation is common method in task-oriented dialogue area. Stent et al. (2004) and Walker et al. (2007a) use machine learning to train the templates selection. The cost to create and maintain templates huge, which is challenge adapting current dialogue system to new domains different language. [Wiseman et al. (2018) introduce HMM into text generation, which can decode the state of generation templates. It's a useful method to generate templates, however the variability and efficiency not satisfied.
4.2 TRANSFORMER
Recently, a new neural architecture called transformer has surpassed RNNs on sequence to sequence tasks, the paper transformers by Vaswani et. al. [19] demonstrated that transformers produce state-of-the-ard results on machine translation, which allowing for increased parallelization and sig- nificantly reduced training time. Otherwise, there has been little work around the use of transformer on end-to-end dialog system.
4.3 POINTER-GENERATON NETWORKS
The pointer-generator was first proposed by Vinyals ,and then has been applied several natural language processing tasks, including translation (Gulcehre et al. [4]), language modeling (Merity et al. [7]), and summarization.
The motivation of this paper is how to effectively extract and use the relevant contexts for multi-turn dialogue generation. Different from previous studies, our proposed model can focus on the relevant contexts, with both long and short distant dependency relations, by using the transformer-pointer- generator mechanism.
5 CONCLUSION
In this paper,we propose an end-to-end trainable model called Response generation via templates Integration. The transformer pointer generator is trained to incorporate related reference responses into the learning framework which performs remarkably well. As result our model achieves state- of-the-art results in both BLEU and Entity Fl score in the Multiwoz Dataset. We hope that the proposed model can benefit future related research.
REFERENCES
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. arXiv preprint arXiv:1810.0027 2018.
Mihail Eric and Christopher Manning. copy augmeneddd sequence sequence-to-sequeene to-sequence architecture gives good performance task-oriented dialogue. Proceedings of the I 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, 2017. doi: 10. 10.18653//1//17-20000 URL htl.tttp::/ doi .eciiirrg//ii 18653/ /

---

Under review as a conference paper at ICLR 2020
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. Key-value trieval networks for task-oriented dialogue. Proceedings of the 18th Anmual SIGdial Meeting on Discourse and Dialogue, 2017. doi: 10.18633/11w11755000. URL http: http//xx.oo/// org/
10. 1653/v6lvvvv/ 1 -
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. Mem2seq:
Effectively incorporating knowl- edge bases into end-to-end task-oriented dialog systems. Proceedings of the 56th Annual Meer- ing of the Association for Computational Linguistics (Volume 1: Long Papers), 2018. doi: 10. 10.186333v1/p188 136. UR/httpp:/dxx
1ttp//1.18653////11111 1136
Amanda Stent, Rashmi Prasad, and Marilyn A. Walker. Trainable sentence planning for complex information presentations in spoken dialog systems. In Meeting Association for Computational Linguistics, 2004.
Shang-Yo Yu Su, Kai-Ling Lo. Yi Ting Ych, and Yun-Nung Chen. Natural language generation by hierarchical decoding with linguistic patterns. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language
Technologies, Volume (Short Pupers), 2018. doi: 10.18653//1/11-20011 URL http:
7/dx. (idxxdoi..or wtt.oo/g/.....331....
2010
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, al. End-to-end memory networks. In Advances in neural information processing systems, pp. 2440-2445 2448,2200 2015.
M. A. Walker, A. Stent, F. Mairesse, and R. Prasad. Individual and domain adaptation in sentence planning for dialogue. Journal of Artificial Intelligence Research, 30:413456, Nov 2007a. ISSN 1076-9757. doi: 0..1613/jai..3299
ptttt:://dx. doi wwg/10.go..0 1113110.111311.... 2222)
Marilyn A Walker, Amanda Stent, François Mairesse, and Rashmi Prasad. Individual and domain adaptation in sentence planning for dialogue. Journal of Artificial Intelligence Research, 30: 413-456, 2007b.
Tsung-Hsien Wen, Milica Gasic, Nikola Mrki, Pei-Hao Su, David Vandyke, and Steve Young. Se- mantically conditioned lstm-based natural language generation for spoken dialogue systems. Pmo ceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015.
doi: 10. 1ttp:///11111111111 1199. URL dx doi. bri.orr/lo 100 http:/1...66531111111/
1199
Tsung-Hsien Wen, David Vandyke, Nikola Mrki, Milica Gasic, Lina M. Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. A network- end-to-end trainable task-oriented dialogue system. Proceedings the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, 2017. doi: 10.18653/v1/e17-1044 URL
hhtpp://xxx doi. 11t.0oo0010..163/111111111111000
Sam Wiseman, Stuart Shieher, and Alexander Rush. Learning neural templates for text gener- ation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pm- cessing, 2018. doi: 10.18653/v1/d18-- 1311-1336. URL httpp:/ /xx doi. or1go.000....5.//vrr d18 1356


Reviews:
[{"review_id": "Skx9kUBXcH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The basic idea of integrating templates for dialog generation is interesting. However, the implementation is confusing, yet another architecture, with clear motivations, intuitions, or even clarity. Worst of all, the experimental results are missing in table 2, the only placeholder for results in the paper. Further, I would appreciate human evaluation, rather than using BLEU as the metric. The latter is known to be inappropriate for dialog modeling evaluation.\n\nSection 2.1 needs more explanation.\n\n(Strong Reject)", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJejyVYGqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a based model for the problem of task-oriented dialog. The main novelty seems to be: 1) the model can mix multiple templates to respond; 2) the response is generated using a combination of templates and a language model (directly) conditioned on the input.\n\nThe paper studies an interesting task however it is not ready for\npublication:\n- Empirical results of all baselines are missing (Table 2)\n- Other results are mentioned but are absent from the manuscript\n- The related work does not provide details as to how the proposed model to prior work\n- The figures are difficult to understand (although they do help make the model clearer and so I suggest polishing them for the next version of this work)\n- The manuscript should be carefully copy-edited.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkxAk9lVcS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method to incorporate candidate templates to aid in response generation within an end-to-end dialog system. While the motivation and task setup is interesting, the paper is clearly unfinished. Most jarringly, Table 2 which should contain the main results comparing the proposed RGTI model to existing baseline models is not filled in, and there appears to be no table showing results of the ablation study briefly described in Section 3.4.\n\nOther issues include the lack of detail around any of the subtasks mention. For example, in Sec 2.1, how is semantic information used as search criteria to retrieve responses? We also need to see more details about where responses are retrieved from. In Sec 2.2, how is the splitting done to construct tuples from the candidate templates (the word \"in\" is dropped in the example given, and it's not clear if this is a typo or whether some processing is done over the phrases before tuple construction is done).\n\nOther comments:\n- The paper needs to be heavily proofread. There are many spelling/grammar mistakes, for example:\n\t- dilogue -> dialogue (Figure 2, page 2)\n\t- acurrate -> accurate (top of page 3)\n        - \"First we should retrieval responses\" -> \"retrieve responses\" (page 3)\n        - BlEU -> BLEU (page 5)", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeBak8bqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary: The authors proposed a deep neural network-based model to generate responses fro task-oriented dialogue systems. The model mainly contains two parts, the first part is to retrieve relevant responses based on question and encode them into templates, the second part is a decoder to generate the response based on the encoded templates and input utterances.\n2. Assessment: The problem studied in this work is very interesting and important. The high-level idea also looks interesting. However, this paper is clearly not ready to submit as it contains a lot of grammar errors and lacks a lot of important details. \n2.1 \"In our model, we retrieval ...\" => \"we retrieve\", \"which is composed of a encoder ...\" => \"an encoder\". There are too many such language errors in this paper. \n2.2 Table 2 is not finished yet. There's only one result in it.\n2.3 What does \"triple form\" mean? What are the candidate responses to retrieve when doing inference? What are the details of reference response retrieval? To many such important details are missing.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a method to incorporate multiple candidate templates to aid in response generation for an end-to-end dialog system. Reviewers thought the basic idea is novel and interesting. However, they also agree that the paper is far from complete, results are missing, further experiments are needed as justification, and the presentation of the paper is not very clear. Given the these feedback from the reviews, I suggest rejecting the paper.
Decision: Reject