SlowGuess/NAACL_2021_OCR huggingface dataset:
Response Appropriateness Furthermore, we conduct manual evaluation on the best models according to the automatic metrics. We only manually evaluate the model performance on large-scale datasets<sup>7</sup>. We ask human evaluators to rate a response in  $\{0,1,2\}$ . A score of 0 means that the response might have flaw in fluency and logic or be incoherent. Special cases are for example completely coping from the dialogue history as the output, and a bland response such as "I don't know what you mean". A score of 1 represents a coherent but generic response. 2 represents a coherent and informative response. We also do a pair-wise evaluation to compare two models and indicate which one is better. The evaluation is based on 200 random samples. Each generated response is rated by three annotators. The inter-rater annotation agreement in Cohen's kappa (Cohen, 1960) is 0.441 on average, which indicates moderate agreement.
Condition Consistency We observe that automatic metrics fail to evaluate condition consistency since BERT that does not consider conditions outperforms C-Trans-ED and C-Trans-Dec. Thus, we perform manual evaluation on the condition consistency. A generated response is rated in  $\{0,1,2\}$ . The scores 0, 1 and 2 mean respectively that the response is inconsistent to the condition, somehow related, and consistent. However, if the response has flaw in fluency or logic, it will get a score of 0. For Topic Dialogue, it is easy to measure whether a generated response is in the topic. However, for persona consistency, it is difficult for a human evaluator to know the speaking style of each user. Thus, before evaluation we first automatically determine those frequently used words by a user in responses and show them to the annotators to help their evaluations.
Table 2 and 3 gives automatic evaluation results, and Table 4 gives human evaluation results. Appendix B shows some generated responses. The results can be summarized as follow:
BERT vs. Trans-ED & Trans-Dec C-Trans-Dec has a clear advantage over C-Trans-ED in almost all automatic metrics, which can also
<sup>7</sup>We did not manually evaluate the results with small datasets due to its high cost. However, we expect even larger difference when small data are used for training, as indicated by the automatic metrics.
$^{5}$ http://opennmt.net/  
 $^{6}$ We use an open-source evaluation tool: https://github.com/Maluuba/nlg-eval
Model	BLEU-1	BLEU-2	BLEU-3	ROUGE-L	CIDEr	Dist-1	Dist-2	avgLen
Sp-Model	10.539 (**)	3.152 (**)	1.396 (**)	0.116 (**)	0.056 (**)	0.012 (**)	0.044 (**)	12.6
MT-Speaker	10.970 (**)	3.488 (**)	1.540 (**)	0.118 (**)	0.059 (**)	0.009 (**)	0.034 (**)	12.7
C-Trans-ED	13.548 (*)	3.881 (**)	1.529 (**)	0.113 (**)	0.045 (**)	0.005 (**)	0.025 (**)	18.7
C-Trans-Dec	12.964 (**)	4.182 (**)	1.781 (**)	0.117 (**)	0.060 (**)	0.023 (**)	0.097 (**)	16.7
BERT	12.928 (*)	4.405 (/)	1.764 (**)	0.119 (**)	0.062 (**)	0.014 (**)	0.052 (**)	26.1
Ours	14.052	4.891	2.149	0.122	0.070	0.024	0.098	23.3
Two-Step FT	13.714 (/)	4.870 (/)	2.160 (/)	0.122 (/)	0.071 (/)	0.023 (/)	0.102 (*)	25.0
w/o ctext	13.015 (*)	4.563 (/)	1.956 (/)	0.113 (**)	0.061 (**)	0.023 (/)	0.106 (*)	25.7
w/o tfidf	13.581 (*)	4.705 (/)	2.000 (/)	0.118 (**)	0.070 (/)	0.023 (/)	0.095 (*)	24.0
Sp-Model	10.467 (**)	3.039 (**)	1.239 (**)	0.116 (**)	0.049 (**)	0.007 (**)	0.027 (**)	12.3
MT-Speaker	10.286 (**)	2.932 (**)	1.174 (**)	0.114 (**)	0.047 (**)	0.007 (**)	0.030 (**)	12.3
C-Trans-ED	10.968 (**)	3.247 (**)	1.295 (**)	0.106 (**)	0.040 (**)	0.001 (**)	0.006 (**)	14.7
C-Trans-Dec	11.263 (**)	3.390 (**)	1.274 (**)	0.106 (**)	0.043 (**)	0.020 (**)	0.075 (**)	16.2
BERT	12.766 (*)	4.195 (*)	1.805 (*)	0.118 (/)	0.063 (*)	0.022 (/)	0.071 (**)	15.3
Ours	13.517	4.517	1.988	0.119	0.068	0.021	0.066	16.4
Two-Step FT	10.125 (**)	3.295 (**)	1.388 (**)	0.111 (**)	0.052 (**)	0.015 (**)	0.043 (**)	12.7
w/o ctext	11.776 (**)	3.821 (**)	1.631 (**)	0.115 (**)	0.059 (**)	0.020 (*)	0.062 (**)	14.4
w/o tfidf	13.475 (/)	4.409 (/)	1.853 (/)	0.118 (/)	0.064 (*)	0.023 (/)	0.078 (*)	16.7
Table 2: Evaluation results on large-scale (upper half) and small-scale (lower half) Persona Reddit. Two-Step FT means using our model architecture but applying sequential fine-tuning. w/o ctext is without leveraging conditioned text data. w/o tfidf means without applying TF-IDF based masking. *  $(p < 0.05)$  or **  $(p < 0.01)$  show statistically significant differences with our model by two-sided t-test.
Model	BLEU-1	BLEU-2	BLEU-3	ROUGE-L	CIDEr	Dist-1	Dist-2	avgLen
TA-Seq2Seq	10.197 (**)	3.307 (**)	1.602 (**)	0.121 (**)	0.098 (**)	0.016 (**)	0.051 (**)	9.7
THRED	9.061 (**)	3.035 (**)	1.468 (**)	0.118 (**)	0.098 (**)	0.015 (**)	0.048 (**)	8.8
C-Trans-ED	13.990 (**)	5.359 (**)	2.689 (**)	0.131 (**)	0.147 (**)	0.055 (**)	0.222 (**)	12.5
C-Trans-Dec	14.544 (**)	5.475 (**)	2.669 (**)	0.136 (**)	0.154 (**)	0.046 (**)	0.177 (**)	13.2
BERT	15.287 (/)	6.243 (/)	3.283 (/)	0.141 (/)	0.168 (**)	0.057 (**)	0.227 (**)	12.5
Ours	15.639	6.484	3.455	0.140	0.185	0.060	0.243	13.0
Two-Step FT	15.926 (/)	6.431 (/)	3.376 (/)	0.143 (/)	0.185 (/)	0.059 (*)	0.239 (*)	13.1
w/o ctext	15.491 (*)	6.397 (/)	3.399 (/)	0.142 (/)	0.190 (*)	0.063 (*)	0.262 (**)	12.8
w/o tfidf	15.393 (/)	6.302 (/)	3.351 (/)	0.139 (/)	0.185 (/)	0.059 (**)	0.230 (**)	13.1
C-Trans-ED	13.874 (**)	5.145 (**)	2.503 (*)	0.124 (**)	0.124 (**)	0.039 (**)	0.150 (**)	13.1
C-Trans-Dec	14.899 (/)	5.648 (/)	2.690 (/)	0.133 (**)	0.150 (/)	0.043 (*)	0.176 (*)	15.2
BERT	14.457 (/)	5.583 (/)	2.802 (/)	0.135 (**)	0.136 (**)	0.037 (**)	0.133 (**)	12.4
Ours	14.587	5.747	2.894	0.139	0.152	0.050	0.186	12.0
Two-Step FT	13.941 (**)	5.463 (/)	2.765 (/)	0.136 (*)	0.140 (**)	0.045 (**)	0.169 (**)	11.7
w/o ctext	13.211 (**)	5.179 (**)	2.655 (/)	0.137 (/)	0.142 (**)	0.046 (**)	0.163 (**)	10.8
w/o tfidf	13.964 (**)	5.485 (**)	2.809 (/)	0.135 (**)	0.145 (*)	0.048 (*)	0.178 (**)	11.8
Table 3: Evaluation results on large-scale and small-scale Topic Dialogue. Topic labels are predicted by LDA.
be observed in their generated responses. Finetuning BERT without considering conditions outperforms C-Trans-Dec on most similarity metrics such as BLEU. We explain this by the fact that bi-directional attention could enable a model to better encode dialogue history, and thus to generate responses more similar to the ground truth. The ablation model using w/o ctext is fine-tuning C-BERT (with our condition-aware transformer blocks) on labeled dialogue data. The performance of w/o ctext is similar to C-Trans-Dec's, with a slight advantage in condition consistency and small disadvantage in response appropriateness. These results show that our approach is built upon a strong base model. As mentioned, other pre-trained models can also be used.
With Condition When large Persona Dialogue is available, w/o ctext (i.e. C-BERT) outperforms BERT in almost all automatic metrics. However, we observe that when only small-scale labeled dialogue data are available, all three conditioned models perform worse than BERT. This shows that the model cannot learn the condition-related features well from the limited labeled dialogue data. Thus, it is important to leverage the labeled texts that are easier to collect, and the results on small-scale Persona Reddit show that our multi-task learning approach significantly outperforms BERT on similarity metrics such as BLEU and CIDEr.
For Topic Dialogue, the labels are given by LDA model. LDA is an unsupervised method and the predicted condition labels can be very noisy. Nev
Model	Persona	Topic
Appropriateness	Consistency	Appropriateness	Consistency
Score	Pair-wise	Score	Pair-wise	Score	Pair-wise	Score	Pair-wise
C-Trans-Dec	0.96	(28%, 39%)	0.85	(20%, 39%)	0.77	(26%, 34%)	0.71	(21%, 31%)
BERT	0.77	(11%, 40%)	0.78	(22%, 43%)	0.55	(17%, 40%)	0.46	(16%, 40%)
Ours	1.15	-	1.24	-	0.83	-	0.80	-
w/o ctext	0.91	(26%, 39%)	0.90	(23%, 38%)	0.73	(27%, 35%)	0.72	(23%, 30%)
Figure 2: Performance comparison between sequential fine-tuning and our approach given 1M labeled text data and different size of labeled dialogue data.
ertheless, similarly, with large data C-BERT outperforms BERT in all metrics, but when only small-scale labeled dialogue data are available, C-BERT performs worse than BERT in terms of BLEU. The result again shows the importance of exploiting labeled texts, and our approach is the best on small-scale Topic Dialogue.
Leveraging Labeled Texts In general, our approach significantly outperforms all baselines and w/o ctext that do not exploit labeled text data, either with large-scale or small-scale data. With small-scale data, our approach outperforms BERT while w/o ctext itself cannot achieve this, which shows that conditioned dialogue generation can be helped by extra labeled text data. On Topic Dialogue, with such noisy labels, our model leveraging the labeled texts still produces the best performance, which confirms the robustness of our multi-task learning approach to work with different types of labels. The human evaluation on appropriateness and condition consistency further confirms the effectiveness of our approach.
Not all methods utilizing extra labeled text can obtain such performance improvement as we did. MT-Speaker that employs an extra auto-encoder
Table 4: Human evaluation of generated responses on appropriateness and condition consistency. Pair-wise comparisons show the wining percentages of (baseline, ours).
Model	BLEU-1	BLEU-2	Dist-2
Single Gate	13.880 (*)	4.853 (/)	0.090 (**)
Double Gates	13.988 (*)	4.889 (/)	0.094 (*)
Attn. Routing	14.052	4.891	0.098
Single Gate	11.703 (**)	3.891 (**)	0.090 (**)
Double Gates	11.336 (**)	3.698 (**)	0.091 (**)
Attn. Gating	13.517	4.517	0.066
Table 5: Comparison of gating mechanisms on large-scale and small-scale Persona Reddit.
does not gain much improvement over Sp-Model. This result shows that using additional model components to leverage labeled texts is inefficient for conditioned dialogue generation. Furthermore, Two-Step FT that first fine-tuning on labeled texts and then on labeled dialogue data does not always produce good performance. It achieves comparable performance to our approach on large-scale datasets, but on small-scale datasets it can even perform worse than w/o ctext (Table 2). This result shows that with small-scale dataset, it is better to avoid sequential fine-tuning because first finetuning on labeled texts will erase the effect of the previous step of pre-training. Furthermore, we investigate how the ratio of the size of labeled text data to the size of dialogue data influence model performance. As shown in Figure 2, given 1M labeled text data, when the ratio is less than 6.7, our approach performs better than Two-Step FT. However, when labeled text corpus is much larger than dialogue corpus, sequential fine-tuning is better. We assume that with large labeled text corpus the pre-trained language model can be tuned to conditioned language generation. Besides, the final task in sequential fine-tuning is purely conditioned dialogue generation, which is expected to achieve better performance on dialogue than a multi-task learning approach. However, in real application situations, one cannot always expect that a large labeled text corpus as supplement for the dialogue data is available.
TF-IDF Masking and Attention Gating We assumed that the general language features have already been captured by the pre-trained models. Thus, to better utilize labeled text data, we mask more condition-related words using TF-IDF based masking. Our ablation study confirms that TF-IDF masking brings improvement in almost all automatic metrics although the improvement might not always be statistically significant.
Our attention gating is a non-parametric gating mechanism to fuse the condition into the decoder. We expected it to be efficient, which is particularly important when labeled data are limited. Here, we compare it with two common parametric gating mechanisms: 1) setting a single gate on  $\mathbf{C}^i$  to get a weight; 2) setting gates on both  $\mathbf{C}^i$  and  $\mathbf{v}^c$  to get two weights. Then, we combine the weighted  $\mathbf{C}^i$  and  $\mathbf{v}^c$  to get  $\mathbf{C}^{\prime i}$  as in our attention gating. Experimental results in Table 5 confirm that our method is more efficient. When only small-scale labeled data are available, the model with attention gating generates responses that are significantly more similar to the ground-truth.
In this paper, we examined the data scarcity issue of conditioned dialogue generation. Pre-training on unlabeled text or dialogue data is not helpful to conditioned generation. Thus, we exploited labeled text data that are easier to collect than labeled dialogues. We expected these data can contribute to better representations of conditions and better use the conditions in natural language generation, which complement what is lacking in the pre-trained models.
To leverage these two types of data, we proposed a simple and efficient multi-task learning approach. Three tasks are considered: conditioned dialogue generation task on the labeled dialogue data, conditioned language encoding task and conditioned language generation task on the labeled text data. We conducted experiments under persona and topic conditions. Experimental results show that our approach outperforms the state-of-the-art models by leveraging labeled texts, and it also obtains larger improvement in performance comparing to the previous methods leveraging text data.
Siqi Bao, Huang He, Fan Wang, and Hua Wu. 2019. Plato: Pre-trained dialogue generation
model with discrete latent variable. arXiv preprint arXiv:1910.07931.
Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhen Guo, Zhibin Liu, and Xinchao Xu. 2020. Plato-2: Towards building an open-domain chatbot via curriculum learning. arXiv preprint arXiv:2006.16779.
Chaotao Chen, Jinhua Peng, Fan Wang, Jun Xu, and Hua Wu. 2019. Generating multiple diverse responses with multi-mapping and posterior mapping selection. arXiv preprint arXiv:1906.01781.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019. The second conversational intelligence challenge (convai2). arXiv preprint arXiv:1902.00098.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, pages 13042-13054.
Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar R Zaiane. 2019. Augmenting neural response generation with context-aware topical attention. In Proceedings of the First Workshop on NLP for Conversational AI, pages 18-31.
Michel Galley, Chris Brockett, Xiang Gao, Jianfeng Gao, and Bill Dolan. 2019. Grounded response generation task at dstdc7. In AAAI Dialog System Technology Challenges Workshop.
Jun Gao, Wei Bi, Xiaojiang Liu, Junhui Li, and Shuming Shi. 2019. Generating multiple diverse responses for short-text conversation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6383-6390.
Xiaodong Gu, Kyunghyun Cho, Jung-Woo Ha, and Sunghun Kim. 2018. Dialogwae: Multimodal response generation with conditional wasserstein autoencoder. arXiv preprint arXiv:1805.12352.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119.
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 994-1003.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986-995.
Zhaojiang Lin, Peng Xu, Genta Indra Winata, Zihan Liu, and Pascale Fung. 2019. Caire: An end-to-end empathetic chatbot. arXiv preprint arXiv:1907.12108.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Yi Luan, Chris Brockett, Bill Dolan, Jianfeng Gao, and Michel Galley. 2017. Multi-task learning for speaker-role adaptation in neural conversation models. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 605-614.
Andrea Madotto, Zhaojiang Lin, Yejin Bang, and Pascale Fung. 2020. The adapter-bot: All-in-one controllable conversational model. arXiv preprint arXiv:2008.12579.
Tong Niu and Mohit Bansal. 2018. Polite dialogue generation without parallel data. Transactions of the Association for Computational Linguistics, 6:373-389.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper.pdf.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370-5381.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. 2020. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637.
Shoetsu Sato, Naoki Yoshinaga, Masashi Toyoda, and Masaru Kitsuregawa. 2017. Modeling situations in neural chat bots. In Proceedings of ACL 2017, Student Research Workshop, pages 120-127.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence.
Xiaoyu Shen, Hui Su, Shuzi Niu, and Vera Demberg. 2018. Improving variational encoder-decoders in dialogue generation. In Thirty-Second AAAI Conference on Artificial Intelligence.
Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, and Jason Weston. 2019. The dialogue dodecathlon: Open-domain knowledge and image grounded conversational agents. arXiv preprint arXiv:1911.03768.
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019. Learning to speak and act in a fantasy text adventure game. arXiv preprint arXiv:1903.03094.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575.
Thomas Wolf, Victor Sanh, Julien Chaumont, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149.
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In Thirty-First AAAI Conference on Artificial Intelligence.
Yan Zeng and Jian-Yun Nie. 2020. Open-domain dialogue generation based on pre-trained language models. arXiv preprint arXiv:2010.12780.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536.
Yinhe Zheng, Rongsheng Zhang, Xiaoxi Mao, and Minlie Huang. 2019. A pre-training based personalized dialogue generation model with persona-sparse data. arXiv preprint arXiv:1911.04700.
Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018. Emotional chatting machine: Emotional conversation generation with internal and external memory. In Thirty-Second AAAI Conference on Artificial Intelligence.
Xianda Zhou and William Yang Wang. 2018. Mojitalk: Generating emotional responses at scale. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1128-1137.
For the small-scale datasets, we trained the models until the performances stop to increase on validation set to prevent over-fitting. For large-scale datasets, we fine-tuned the models for four epochs. In Table 6, the average runtime is tested using a 1080Ti GPU device, and the batch size is set to take all of the GPU memories. TA-Seq2Seq and THRED are implemented in TensorFlow. Other models are implemented in PyTorch. Notice that the runtime will be influenced by code implementation in additional to model structure. When experimenting with the small-scale Persona Reddit dataset, we decrease the number of parameters of Sp-Model and MT-Speaker models to 48M and 52M respectively in order to avoid over-fitting. C-Trans-ED loads the pre-training results of GPT. In the original paper, they pre-trained by themselves using a Chinese corpus, which cannot be used in our experiments.
Model	Parameters	Runtime(min/M)
Sp-Model	80M	25
MT-Speaker	90M	40
TA-Seq2Seq	155M	150
THRED	174M	135
C-Trans-ED	120M	180
C-Trans-Dec	126M	290
BERT	110M	140
Ours	113M	145
Table 6: The number of parameters of each tested approach and the average runtime (minutes) for every million training samples.
Hyper-parameters	Value
C-Tranformer layers	2
mask probability	0.25
max length	80
batch size	160
learning rate	3e-5
warmup proportion	0.1
label smoothing	0
weight decay	0.01
dropout probability	0.1
Table 7: Hyper-parameters for our fine-tuning approach. There are in total 6M data. Thus, we use a large batch size.
Additionally, we explored the idea proposed in Zeng and Nie (2020) to decrease finetune
generation discrepancy introduced by MLM training objective. Nevertheless, the conditioned language encoding task cannot employ this method because it applies bi-directional self-attention. Experimental result on small-scale Persona Reddit shows that eliminating this discrepancy helps to decrease perplexity from 55 to 52.
B Generation Samples