SlowGuess/NAACL_2022_OCR huggingface dataset:
Following Zheng et al. (2020) and Ze et al. (2020), we use automatic metrics to measure DTR on three aspects: Style Intensity, Relevance, and Diversity. For style intensity, we use the GPT-2 classifier prediction mentioned in section 5.3. Relevance is measured with F1, BLEU (Papineni et al., 2002) and Rouge (Lin, 2004). We use Distinct (Li et al., 2016) to measure Diversity of different models.
To measure the diversity between different styles, we propose inner Distinct: given a context and knowledge, we calculate distinct in three generated responses with three styles.
For human evaluation, we randomly sample 500 examples from test set, and recruit 3 well-educated annotators. To each annotator, two responses from different models are presented, which are randomly shuffled to hide their sources. The annotators then judge which response is better from four aspects: (1) Style Consistency: which response exhibits the desired style more (2) Knowledge Preservation: which response is more relevant to the knowledgeable document (3) Context Coherence: which response is more coherent with the dialogue context (4) Fluency: which response is more fluent and free from any grammar errors.
We use pre-trained MASS (Song et al., 2019) to initialize  $\mathcal{G}_G$  and  $\mathcal{G}_R$ . We adopt Adam optimizer as an initial learning rate of  $5 \times 10^{-4}$ , and the batch size is 4096 tokens for a NVIDIA 1080 Ti GPU. Since all the baselines don't have a knowledge selection module, we chose the ground-truth knowledge as input for Wizard and the top-1 knowledge sentence according to the BLEU-1 with the corresponding response as input for Topical Chat. We use beam search(size=5) to decode the response. We initialize  $\mathcal{F}$  with pre-trained BERT, the replace rate  $P_r$  is 25,  $Z$  in section 4.2 is 10. We use Glove (Jeffrey et al., 2014) 100d embedding and cosine similarity as  $\mathrm{Dis}(\cdot ,\cdot)$  to calculate distance  $d$ .  $\mu$  in Eq.8 is 0.2. To get the style intensity reward, we follow Ze et al. (2020) and train binary GPT-2 (Radford et al., 2019) classifiers. Early stopping on validation is adopted as a regularization strategy. All the above hyperparameters are determined by grid search.
5.4 Baselines
The following models are selected as baselines:
- StyleFusion (Gao et al., 2019) bridges conversation modeling and nonparallel style transfer by sharing a latent space. We use the code https://github.com/golsun/StyleFusion.
- StylisticDLV (Zhu et al., 2021) disentangles the content and style in latent space by diluting information in style representations. We use the code https://github.com/golsun/StyleFusion.
Style	Models	Wizard of Wikipedia	Topical Chat
Style Intensity	Relevance	Diversity	Average Length	Style Intensity	Relevance	Diversity	Average Length
F1	B-1	B-2	R	D-1	D-2	iD-1	iD-2	F1	B-1	B-2	R	D-1	D-2	iD-1	iD-2
Positive	StyleFusion	0.275	11.8	12.3	6.5	10.1	3.3	8.7	54.1	67.3	11.2	0.263	12.6	12.9	6.8	11.2	0.8	2.6	42.1	60.7
StylisticDLV	0.336	10.6	11.5	6.1	9.3	3.9	9.2	56.7	69.5	10.6	0.381	12.2	12.5	6.7	10.6	1.3	3.3	44.7	63.2
StylizedDU	0.342	15.7	17.4	9.6	18.5	14.1	34.8	50.3	65.5	14.5	0.417	16.2	15.8	10.1	15.4	3.6	10.5	46.2	65.8
StyleDGPT	0.354	21.7	24.3	17.1	24.8	12.5	33.2	61.4	73.2	10.8	0.392	20.4	18.6	14.6	18.7	2.8	7.8	58.6	74.9
DTR	0.338	31.3	32.6	20.7	32.6	12.9	35.5	59.6	76.9	20.3	0.448	26.4	30.2	18.9	26.0	3.9	11.8	63.8	76.0
Negative	StyleFusion	0.327	12.5	11.7	7.4	9.6	3.1	8.8	53.5	70.3	10.4	0.293	10.8	11.4	6.5	10.6	1.0	2.4	55.7	63.5
StylisticDLV	0.665	11.8	11.1	6.9	9.0	3.4	9.1	54.7	70.8	11.3	0.655	10.4	11.2	6.1	10.5	1.2	2.7	58.0	64.9
StylizedDU	0.640	16.1	16.7	9.4	15.9	13.6	31.3	56.1	69.6	13.8	0.642	15.7	15.5	11.3	15.8	3.2	8.4	58.0	65.4
StyleDGPT	0.713	22.5	24.9	17.5	25.0	11.8	32.0	62.9	74.2	12.4	0.686	21.3	22.1	16.5	19.2	2.3	6.6	64.6	70.1
DTR	0.783	32.0	31.1	20.6	31.8	14.3	34.5	66.4	78.7	18.7	0.715	27.9	31.2	19.7	26.5	4.5	12.8	67.2	75.3
Polite	StyleFusion	0.211	11.3	11.6	6.8	10.7	1.9	5.5	45.0	53.4	12.6	0.243	12.5	12.8	7.3	12.2	0.8	2.3	40.4	57.1
StylisticDLV	0.264	10.7	10.8	6.2	10.1	2.1	6.0	47.3	55.9	12.1	0.375	13.0	13.4	7.5	12.6	0.9	2.8	43.6	59.3
StylizedDU	0.270	14.9	16.2	10.2	17.4	11.5	35.1	43.3	63.2	14.7	0.382	16.4	15.3	10.9	14.7	3.8	12.4	42.8	60.9
StyleDGPT	0.262	24.8	22.2	15.7	23.8	12.2	33.1	51.9	65.7	13.3	0.316	20.8	19.4	15.3	20.8	3.0	9.2	45.7	58.3
DTR	0.287	30.6	29.3	20.5	31.6	12.8	37.4	55.4	68.1	20.3	0.403	27.6	30.5	19.8	29.1	4.2	14.6	47.2	62.5
Table 1: Automatic evaluation results. Numbers in bold mean that the improvement to the best baseline is statistically significant (t-test with  $p$ -value  $< 0.01$ ).
Manual evaluation results	Attractiveness evaluation results
Style	Models	Style Consistency W(%) L(%) T(%)	Knowledge Preservation W(%) L(%) T(%)	Context Coherence W(%) L(%) T(%)	Fluency W(%) L(%) T(%)	Kappa	Style	Models	Attractiveness W(%) L(%) T(%)	Kappa
Wizard of Wikipedia	Wizard of Wikipedia
Positive	DTR vs. StyleDGPT	56.8	22.2	21.0	58.0	22.8	19.2	52.4	19.5	28.1	54.9	22.3	22.8	0.67	Positive	DTR vs. DTR-s	60.4	18.1	21.5	0.68
Negative	DTR vs. StyleDGPT	54.8	18.4	26.8	58.2	17.9	23.9	55.0	19.6	25.4	51.0	28.6	20.4	0.65	Negative	DTR vs. DTR-s	13.7	58.3	28.0	0.65
Polite	DTR vs. StyleDGPT	58.0	21.6	20.4	60.2	21.1	18.7	56.7	20.7	22.6	50.5	29.2	20.3	0.68	Polite	DTR vs. DTR-s	56.2	12.3	31.5	0.64
Topical Chat	Topical Chat
Positive	DTR vs. StyleDGPT	48.2	23.3	28.5	57.3	20.5	22.2	48.8	24.0	27.2	53.6	23.9	22.5	0.65	Positive	DTR vs. DTR-s	54.6	23.1	22.3	0.65
Negative	DTR vs. StyleDGPT	56.7	21.6	21.7	51.6	27.4	21.0	54.0	22.6	23.4	52.6	22.9	24.5	0.64	Negative	DTR vs. DTR-s	26.4	54.5	19.1	0.65
Polite	DTR vs. StyleDGPT	49.8	19.3	30.9	46.5	28.1	25.4	45.6	27.3	27.1	53.5	21.1	25.4	0.65	Polite	DTR vs. DTR-s	49.6	21.7	28.7	0.67
Table 2: Manual evaluation results. W, L, and T refer to Win, Lose, and Tie. All of the Kappa scores are greater than 0.6, which indicates the good agreement among the annotators. Other models are shown in Appendix B.4.
Figure 5: F1 of DTR and StyleDGPT (positive), and SOTA KDG models in different evaluation settings.
Figure 6: F1 and inner Distinct with different replace rate on three different styles in Wizard Test set.
- StylizedDU (Zheng et al., 2020) leverages back-translation technique to generate pseudo stylized context-response pairs. We use the code https://github.com/silverriver/Stylized_Dialog.
- StyleDGPT (Ze et al., 2020) exploits the pre-trained language models on the
stylized response generation task. We use the code https://github.com/TobeyYang/StyleDGPT.
All the baselines are jointly learned with datasets  $\mathcal{D}_c$  and  $\mathcal{D}_s$ , and take the concatenation of knowledge and context as input.
Models	Wizard of Wikipedia	Topical Chat
Style Intensity	Relevance	Diversity	Style Intensity	Relevance	Diversity
F1	B-1	B-2	R	D-1	D-2	iD-1	iD-2	F1	B-1	B-2	R	D-1	D-2	iD-1	iD-2
DTR	0.338	31.3	32.6	20.7	32.6	12.9	35.5	59.6	76.9	0.448	26.4	30.2	18.9	26.0	3.9	11.8	63.8	76.0
w/o F WSL	0.186	23.4	21.2	13.5	21.6	15.1	38.4	67.3	80.4	0.287	15.5	17.9	11.6	16.3	4.8	14.9	69.1	81.3
w/o F (TFIDF)	0.244	28.7	28.5	19.0	29.6	13.5	36.6	61.5	78.6	0.369	22.7	26.2	16.0	21.5	4.3	12.3	65.5	76.4
w/o F (Classification)	0.256	31.7	30.2	20.9	31.5	13.1	35.3	58.6	76.2	0.375	23.3	26.7	16.5	22.6	3.8	12.1	64.7	75.0
w/o Rewards	0.307	31.4	28.9	20.2	29.6	12.7	35.1	58.1	76.4	0.424	25.1	29.0	18.3	24.3	4.2	11.9	63.7	74.2
w/o Cls	0.297	32.0	31.5	20.8	30.2	12.0	34.7	56.3	74.3	0.396	24.4	30.7	19.1	26.2	3.9	11.4	63.1	74.5
w/o Sim	0.340	30.4	28.6	20.2	29.4	12.3	36.5	61.0	78.3	0.452	26.8	28.8	17.9	24.3	4.3	12.1	64.8	75.6
Table 3: Ablation evaluation results of positive sentiment. Other styles are shown in Appendix B.3.
5.5 Evaluation Results
As shown in Table 1, our DTR model achieves competitive performance in style transfer and significantly outperforms the baselines in all the relevance metrics. This indicates that DTR can produce high-quality responses which are coherent to the context, related to the knowledge, and consistent with the target style simultaneously. We also observe that all SDG methods frequently lost the knowledge part (Appendix B.2). DTR significantly outperforms StyleDGPT on relevance, indicating that leveraging the style intensity score to optimize the decoupling of the template is superior to directly optimizing response generation (degrading language modeling). We observe the core component back-translation in StylizedDU fails to infer pseudo-knowledge from a response (generally, the knowledge carries much more information than the response). Table 2 reports the results of human evaluation, DTR significantly outperforms StyleDGPT on all aspects. DTR is also superior to all the baselines as the Case Study section in Appendix B.2.
Firstly, to verify the contributions of the proposed disentangler and weakly supervised learning method, we consider the following variants: (1)  $\mathbf{w} / \mathbf{o}$ $\mathcal{F}$  WSL: training DTR without the Weakly Supervised Learning of  $\mathcal{F}$  in section 4.3.1. (2) w/o  $\mathcal{F}$  (Classification): replace the pairwise ranking loss in  $\mathcal{F}$  with a binary classification loss. We define those tokens with  $d = 0$  (in section 4.3.1) as style words (label=1), otherwise non-style words (label=0). (3) w/o  $\mathcal{F}$  (TFIDF): replace  $\mathcal{F}$  with a TFIDF-based rule (replace the fragments as  $[^{*}]$  in a sentence with the lowest  $P_{r}\%$  TFIDF scores except stop words). Table 7 shows the results of the three variants. We can conclude that (1) the weakly supervised learning of  $\mathcal{F}$  is crucial to training DTR, since the variant with a simple TFIDF significantly outperforms the one without any initialization; and
(2) the ranking loss in  $\mathcal{F}$  plays a key role in the success of style transfer, there is a dramatic drop on the style intensity of  $\mathbf{w} / \mathbf{o}$ $\mathcal{F}$  (Classification). According to our observation, it is overfitting on the style corpus, leading to a low success rate.
Secondly, to investigate the RL rewards in Eq.(6), we consider the following variants: (1) w/o Rewards : remove the similarity and style intensity reward. (2) w/o Sim: remove the similarity reward. (3) w/o Cls: remove the style intensity reward. As shown in Table 7, removal any of the two rewards will cause performance drop, indicating that style intensity and similarity reward can enhance DTR. We also add Sim to StylizedDU, the improvement is +2.1 on F1, thus it's hard for Sim to bridge the huge gap. Negative and Polite are similar, these results are presented in Appendix B.3.
5.7 Discussions
Impact of stylized knowledge-grounded generation. We annotate the "Attractiveness" (the annotators are given two different responses with the same context and knowledge from two different models, and they should determine which response is more attractive and engaging in a holistic way) of DTR and DTR-s (without style transfer) following the same process in 5.2. Table 2 reports the evaluation results. We can see that introducing a positive sentiment or a polite style would enhance the engagement of the KDG model while establishing a negative sentiment harm the attractiveness.
Impact of style transfer on the conversational ability of SDG models. We are curious about to what extent the conversational ability of SDG models will be damaged after style transfer. We examine DTR and StyleDGPT in two settings: (1) Gold-K: the given knowledge is the ground-truth (2) Predicted-K: the given knowledge is selected from a knowledge selection model (Xueliang et al., 2020). As shown in Figure 5, after style transfer on Wizard, the F1 of DTR drops 2.28 and 2.1 in Gold-
K and Predicted-K, while the F1 of StyleDGPT drops 11.16 and 8.16 respectively. On Topical Chat, the F1 of DTR drops 1.77 and 1.51 in Gold-K and Predicted-K, while the F1 of StyleDGPT drops 7.1 and 6.16 respectively. Compared with StyleDGPT, DTR dramatically reduces the damage to the conversational ability while achieving a high success rate of style transfer. Thanks to the superior style transferring mechanism, our DTR achieves comparable performance with the state-of-the-art KDG models {KnowledGPT(Xueliang et al., 2020) on Wizard, UNILM(Li et al., 2020) on Topical Chat} in the standard KDG evaluation setting even after style transfer. The results of Negative and Polite are similar and presented in Appendix B.7. Impact of the replace rate  $P_r$ . As shown in Figure 6,  $P_r = 25$  achieves the best balance between relevance and diversity. A smaller  $P_r$  would remain a large number of original style fragments in the template, leading to tiny differences between different styles. On the contrary, a larger  $P_r$  would delete those content fragments, which are harder to restore by rewriter, but the responses from different styles will be more diverse. Topical Chat follows the same regularity as shown in Appendix B.5.
We explore stylized knowledge-grounded dialogue generation by proposing bridging the knowledge-grounded response generation with the stylized rewriting via sharing a disentangled template. Evaluation results on benchmarks of the task indicate that our model can achieve state-of-the-art performance and exhibits a superior generation ability over different knowledge domains and styles.
We thank anonymous reviewers for their insightful suggestions to improve this paper.
Reina Akama, Kazuaki Inada, Naoya Inoue, Sosuke Kobayashi, and Kentaro Inui. 2017. Generating stylistically consistent dialog responses with transfer learning. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 408-412, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference
on Empirical Methods in Natural Language Processing, pages 2256-2262, Austin, Texas. Association for Computational Linguistics.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In Proceedings of the 24th International Conference on Machine Learning, pages 42-55.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In AAAI Conference on Artificial Intelligence.
Xiang Gao, Yizhe Zhang, Sungjin Lee, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2019. Structuring latent spaces for stylized response generation. EMNLP 2019.
Leon Gatys, Alexander Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In CVPR, pages 2414-2423.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019. Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In Proc. Interspeech 2019, pages 1891-1895.
Pennington Jeffrey, Socher Richard, and Manning Christopher D. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, Lisa Orii, and Peter Szolovits. 2020. Hooks in the headline: Learning to generate headlines with controlled styles. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5082-5093, Online. Association for Computational Linguistics.
Li Juncen, Jia Robin, He He, and Liang Percy. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In *NAACL-HLT*, pages 1865-1874.
Byeongchang Kim, Jaewoo Ahn, and Gunhee Kim. 2020. Sequential latent knowledge selection for knowledge-grounded dialogue. arXiv preprint arXiv:2002.07510.
Guillaume Lample, Sandeep Subramanian, Michael Eric Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. ICLR.
Jinpeng Li, Yingce Xia, Rui Yan, Hongda Sun, Dongyan Zhao, and Tie-Yan Liu. 2021. Stylized dialogue generation with multi-pass dual learning. Advances in Neural Information Processing Systems, 34.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
Linxiao Li, Can Xu, Wei Wu, Yufan Zhao, Xueliang Zhao, and Chongyang Tao. 2020. Zero-resource knowledge-grounded dialogue generation. arXiv preprint arXiv:2008.12918.
Zujie Liang, Huang Hu, Can Xu, Chongyang Tao, Xi-ubo Geng, Yining Chen, Fan Liang, and Daxin Jiang. 2021. Maria: A visual experience powered conversational agent. arXiv preprint arXiv:2105.13073.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabas Poczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhumoye. 2020. Topological sort for sentence ordering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Tong Niu and Mohit Bansal. 2018. Polite dialogue generation without parallel data. Transactions of the Association for Computational Linguistics, 6:373-389.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021. Increasing faithfulness in knowledge-grounded dialogue with controllable features. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
pages 704-718, Online. Association for Computational Linguistics.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207.
Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston. 2018. Image chat: Engaging grounded conversations. arXiv preprint arXiv:1811.00945.
Eric Michael Smith, Diana Gonzalez-Rico, Emily Dinan, and Y-Lan Boureau. 2020. Controlling style in generated dialogue. arXiv preprint arXiv:2009.10855.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pretraining for language generation. In International Conference on Machine Learning, pages 5926-5936.
Yixuan Su, Deng Cai, Yan Wang, Simon Baker, Anna Korhonen, Nigel Collier, and Xiaojiang Liu. 2020a. Stylistic dialogue generation via information-guided reinforcement learning strategy.
Yixuan Su, Yan Wang, Simon Baker, Deng Cai, Xiaojiang Liu, Anna Korhonen, and Nigel Collier. 2020b. Prototype-to-style: Dialogue generation with style-aware editing on retrieval memory.
Qingfeng Sun, Yujing Wang, Can Xu, Kai Zheng, Yaming Yang, Huang Hu, Fei Xu, Jessica Zhang, Xiubo Geng, and Daxin Jiang. 2021. Multimodal dialogue response generation.
Richard S Sutton, David A McAllester, and Mansour Singh, Satinder P isand Yishay. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, page 1057-1063.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Zhiyuan Wen, Jiannong Cao, Ruosong Yang, and Senzhang Wang. 2020. Decode with template: Content preserving sentiment transfer. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4671-4679, Marseille, France. European Language Resources Association.
Zhao Xueliang, Wu Wei, Xu Can, Tao Chongyang, Zhao Dongyan, and Yan Rui. 2020. Knowledge-grounded dialogue generation with pre-trained language models. In EMNLP.
Ze Yang, Wei Wu, Huang Hu, Can Xu, and Zhoujun Li. 2020a. Open domain dialogue generation with latent images. arXiv preprint arXiv:2004.01981.
Ze Yang, Wei Wu, Can Xu, Xinnian Liang, Jiaqi Bai, Liran Wang, Wei Wang, and Zhoujun Li. 2020b. StyleDGPT: Stylized response generation with pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1548-1559, Online. Association for Computational Linguistics.
Yang Ze, Wu Wei, Xu Can, Liang Xinnian, Bai Ji aqi, Wang Liran, Wang Wei, and Li Zhoujun. 2020. Stylodgpt: Stylized response generation with pretrained language models.
Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. 2020a. Low-resource knowledge-grounded dialogue generation. In *Eighth International Conference on Learning Representations (ICLR)*.
Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020b. Knowledge-grounded dialogue generation with pre-trained language models. arXiv preprint arXiv:2010.08824.
Yinhe Zheng, Zikai Chen, Rongsheng Zhang, Shilei Huang, Xiaoxi Mao, and Minlie Huang. 2020. Stylized dialogue response generation using stylized unpaired texts. arXiv preprint arXiv:2009.12719.
Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020. The design and implementation of xiaoice, an empathetic social chatbot. Computational Linguistics, 46(1):53-93.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networkss. In Computer Vision (ICCV), 2017 IEEE International Conference on.
Qingfu Zhu, Wei-Nan Zhang, Ting Liu, and William Yang Wang. 2021. Neural stylistic response generation with disentangled latent variables. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4391-4401, Online. Association for Computational Linguistics.
A Method Detail
A.1 Disentangler BERT Learning Detail
We initialize  $\mathcal{F}$  with two pre-trained BERT. In the unsupervised initialization stage we only train the  $\mathrm{BERT}^{\alpha}$ , then in the reinforcement learning stage, we fix the parameters of  $\mathrm{BERT}^{\alpha}$  and loosen  $\mathrm{BERT}^{\beta}$ .
$$
P _ {\mathcal {F}} (A | \bar {Y}, U, K) = \prod_ {i = 1} ^ {m} P \left(a _ {i} | \bar {Y}, U, K\right) \tag {10}
$$
$$
P \left(a _ {i} | \bar {Y}, U, K\right) = x _ {i} = x _ {i} ^ {\alpha} + x _ {i} ^ {\beta} \tag {11}
$$
$$
x _ {i} ^ {\alpha} = \operatorname {s i g m o i d} \left(\boldsymbol {W} ^ {\alpha} e _ {i} ^ {\alpha}\right) \tag {12}
$$
$$
x _ {i} ^ {\beta} = \operatorname {s i g m o i d} \left(\boldsymbol {W} ^ {\beta} e _ {i} ^ {\beta}\right) \tag {13}
$$
$$
\left\{e _ {1} ^ {\alpha}, \dots , e _ {m} ^ {\alpha} \right\} = \operatorname {B E R T} ^ {\alpha} (\bar {Y}) \tag {14}
$$
$$
\left\{e _ {1} ^ {\beta}, \dots , e _ {m} ^ {\beta} \right\} = \operatorname {B E R T} ^ {\beta} (\bar {Y}, U, K) \tag {15}
$$
Table 4 reports the statistics of the Wizard of Wikipedia dataset and the Topical Chat dataset.
B.2 Case Study