## 5 Conclusions  
We introduce a MADM, where an end-to-end dialogue manager cooperates with a user simulator to fulfill a dialogue task. For user simulator reward function, we use the reward shaping technique based on the adjacency pairs to make the simulator learn real user behaviors quickly while learning from scratch. The experimental results show that reward shaping technique speeds up learning and avoids derivation from normal human-human conversation. In addition, we generalize the one-to-one learning strategy to one-to-many learning strategy where a dialogue manager cooperates with various user simulators, which are obtained by changing the adjacency pairs settings. The experimental results also show that the dialogue manager from MADM-OM achieves the best performance on human users involving cross-model evaluation.  
In our proposed MADM, there are several models that can be applied to get utterance embedding in dialogue manager, such as TextCNN [38], BERT [39] and XLnet [40]. But these contextualized model is orthogonal to MADM. In the future, we are planning to substitute these models to the bottom bidirectional LSTM in dialogue manager. In addition, we will collect more dataset to enrich the templates expressiveness for NLG and train the models iteratively.  
Methodology, S.L.; formal analysis, X.W.; data curation, S.L.; writing-original draft preparation, S.L.; writing-review and editing, C.Y.; funding acquisition, X.W. All authors have read and agreed to the published version of the manuscript.  
This research was funded by NSFC (No.61273365).  
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**MADM-S**} & \multicolumn{2}{c}{**Group-S**} & \multicolumn{2}{c}{**Rule-S**} & \multicolumn{2}{c}{**Human Users**} \\ \cline{2-9}  & **SR** & **AT** & **SR** & **AT** & **SR** & **AT** & **SR** & **AT** \\ \hline M-MADM-OM & 0.980 & 17.38 & 0.902 & 18.86 & 0.925 & 17.28 & 0.84 & 18.04 \\ M-MADM-OO & 0.975 & 17.47 & 0.775 & 21.27 & 0.935 & 18.23 & 0.78 & 19.80 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Cross-model evaluation on Success Rate (SR) and Average Turns (AT).  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Group-S**} & \multicolumn{2}{c}{**Rule-S**} & \multicolumn{2}{c}{**Corresponding-S**} \\ \cline{2-7}  & **SR** & **AT** & **SR** & **AT** & **SR** & **B** \\ \hline M-MADM-OM & **0.902** & **18.86** & **0.925** & **17.28** & 0.902 & 18.86 \\ \hline all\&csksk & 0.875 & 19.42 & 0.895 & 19.01 & 0.905 & 18.93 \\ all\&confirm & 0.860 & 19.64 & 0.895 & 18.95 & 0.910 & 18.92 \\ all\&bye & 0.825 & 20.34 & 0.905 & 18.65 & 0.905 & 18.85 \\ all\&naive & 0.835 & 20.07 & 0.880 & 19.27 & 0.900 & 18.94 \\ ask\&confirm & 0.825 & 24.94 & 0.645 & 23.73 & 0.895 & 18.91 \\ ask\&bye & 0.760 & 21.43 & 0.645 & 25.52 & 0.900 & 18.89 \\ ask\&naive & 0.815 & 20.02 & 0.550 & 24.73 & 0.895 & 18.93 \\ confirm\&bye & 0.730 & 22.08 & 0.590 & 18.95 & 0.895 & 18.91 \\ confirm\&naive & 0.725 & 22.23 & 0.505 & 17.04 & 0.905 & 18.89 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The different combinations of seed simulators in one-to-two learning on Success Rate (SR) and Average Turns (AT) (Corresponding-S denotes the corresponding training simulators).  
**Acknowledgments:** This paper is supported by 111 Project (No. B08004), Beijing Advanced Innovation Center for Imaging Technology, Engineering Research Center of Information Networks of MOE, China. The authors would like to thank the reviewers for their comments and suggestions.  
**Conflicts of Interest:** The authors declare no conflict of interest.