Question: How much is proposed model better than baselines in performed experiments?
Evidence: FLOAT SELECTED: Table 2: The performance of baselines and proposed model on DSTC2 and Maluuba dataset. T imefull is the time spent on training the whole model, T imeDP is the time spent on training the dialogue policy maker.
BPRA Results: As shown in Table TABREF35, most of the models have similar performance on BPRA on these two datasets, which can guarantee a consistent impact on the dialogue policy maker. All the models perform very well in BPRA on DSTC2 dataset. On Maluuba dataset, the BPRA decreases because of the complex domains. We can notice that BPRA of CDM is slightly worse than other models on Maluuba dataset, the reason is that the CDM's dialogue policy maker contains lots of classifications and has the bigger loss than other models because of complex domains, which affects the training of the dialogue belief tracker.
APRA Results: Compared with baselines, GDP achieves the best performance in APRA on two datasets. It can be noted that we do not compare with the E2ECM baseline in APRA. E2ECM only uses a simple classifier to recognize the label of the acts and ignores the parameters information. In our experiment, APRA of E2ECM is slightly better than our method. Considering the lack of parameters of the acts, it's unfair for our GDP method. Furthermore, the CDM baseline considers the parameters of the act. But GDP is far better than CDM in supervised learning and reinforcement learning.
BLEU Results: GDP significantly outperforms the baselines on BLEU. As mentioned above, E2ECM is actually slightly better than GDP in APRA. But in fact, we can find that the language quality of the response generated by GDP is still better than E2ECM, which proves that lack of enough parameters information makes it difficult to find the appropriate sentence template in NLG. It can be found that the BLEU of all models is very poor on Maluuba dataset. The reason is that Maluuba is a human-human task-oriented dialogue dataset, the utterances are very flexible, the natural language generator for all methods is difficult to generate an accurate utterance based on the context. And DSTC2 is a human-machine dialog dataset. The response is very regular so the effectiveness of NLG will be better than that of Maluuba. But from the results, the GDP is still better than the baselines on Maluuba dataset, which also verifies that our proposed method is more accurate in modeling dialogue policy on complex domains than the classification-based methods.
FLOAT SELECTED: Table 2: The performance of baselines and proposed model on DSTC2 and Maluuba dataset. T imefull is the time spent on training the whole model, T imeDP is the time spent on training the dialogue policy maker.
BPRA Results: As shown in Table TABREF35, most of the models have similar performance on BPRA on these two datasets, which can guarantee a consistent impact on the dialogue policy maker.
APRA Results: Compared with baselines, GDP achieves the best performance in APRA on two datasets.
Results: GDP significantly outperforms the baselines on BLEU.
Answer: most of the models have similar performance on BPRA: DSTC2 (+0.0015), Maluuba (+0.0729)
GDP achieves the best performance in APRA: DSTC2 (+0.2893), Maluuba (+0.2896)
GDP significantly outperforms the baselines on BLEU: DSTC2 (+0.0791), Maluuba (+0.0492)