SlowGuess/ACL_2020_OCR huggingface dataset:
- score 0 (bad): Sub-messages for Rec and chitchat: not semantically relevant to the context or logically contradictory to the context. Sub-messages for task-oriented: No necessary slot value is involved in the conversation. Sub-messages for QA: Incorrect answer.
- score 1 (fair): relevant to the context as a whole, but using some irrelevant knowledge, or not answering questions asked by the users.
- score 2 (good): otherwise.
<sup>11</sup>It is an open source deep learning platform(https://www.paddlepaddle.org.cn/)
$^{12}$ Due to S2S model uses the same parameters as OpenNMT(http://github.com/OpenNMT/OpenNMT-py), its parameters are not listed.
module	Parameter	value
Goal-planning model	Embedding Size	256
Hidden Size	256
Batch Size	128
Learning Rate	0.002
Optimizer	Adam
Retrieval-based model	Dropout	0.1
Embedding Size	512
Hidden Size	512
Batch Size	32
Learning Rate	0.001
Optimizer	Adam
Weight Decay	0.01
Proportion Warmup	0.1
Generation-based model	Embedding Size	300
Hidden Size	800
Batch Size	16
Learning Rate	0.0005
Grad Clip	5
Dropout	0.2
Beam Size	10
Optimizer	Adam
Table 7: Model parameter settings.
Informativeness measures if the model makes full use of knowledge in the response:
- score 0 (bad): no knowledge is mentioned at all.
- score 1 (fair): only one knowledge triple is mentioned in the response.
- score 2 (good): more than one knowledge triple is mentioned in the response.
Proactivity measures if the model can introduce new knowledge/topics in conversation:
- score -1 (bad): some new topics are introduced but irrelevant to the context.
- score 0 (fair): no new topics/knowledge are used.
- score 1(good): some new topics relevant to the context are introduced.
4. Dialogue-level Human Evaluation Guideline
Goal Completion measures how good the given conversation goal is finished:
- score 0 (bad): less than half goals are achieved..
- score 1 (fair): less than half goals are achieved with minor use of knowledge or goal information.
- score 2 (good): more than half goals are achieved with full use of knowledge and goal information.
Coherence measures the overall fluency of the whole dialogue:
- score 0 (bad): two-thirds responses irrelevant or logically contradictory to the previous context.
- score 1 (fair): less than one-third responses irrelevant or logically contradictory to the previous context.
- score 2 (good): very few response irrelevant or logically contradictory to the previous context.
5. Case Study
Figure 4 shows the conversations generated by the models via conversing with humans, given the conversation goal and the related knowledge. It can be seen that our knowledge-aware generator can use more correct knowledge for diverse conversation generation. Even though the retrieval-based method can also produce knowledge-grounded responses, the used knowledge is relatively few and inappropriate. The seq2seq model can't successfully complete the given goal, as the knowledge is not fully used as our proposed knowledge-aware generator, making the generated conversation less diverse and sometimes dull.
Figure 4: Conversations generated by three different models: texts in red color represent correct knowledge being appropriate in current context, while texts in blue color represent inappropriate knowledge. Texts in purple color indicate that the use of knowledge is correct, but the response is not appropriate.