Question: In XPersona, the training sets are translated automatically or manually?
Evidence: Personalized dialogue agents have been shown efficient in conducting human-like conversation. This progress has been catalyzed thanks to existing conversational dataset such as Persona-chat BIBREF0, BIBREF1. However, the training data are provided in a single language (e.g., English), and thus the resulting systems can perform conversations only in the training language. For wide, commercial dialogue systems are required to handle a large number of languages since the smart home devices market is increasingly international BIBREF2. Therefore, creating multilingual conversational benchmarks is essential, yet challenging since it is costly to perform human annotation of data in all languages.
A possible solution is to use translation systems before and after the model inference, a two-step translation from any language to English and from English to any language. This comes with three major problems: 1) amplification of translation errors since the current dialogue systems are far from perfect, especially with noisy input; 2) the three-stage pipeline system is significantly slower in terms of inference speed; and 3) high translation costs since the current state-of-the-art models, especially in low resources languages, are only available using costly APIs.
In this paper, we analyze two possible workarounds to alleviate the aforementioned challenges. The first is to build a cross-lingual transferable system by aligning cross-lingual representations, as in BIBREF3, in which the system is trained on one language and zero-shot to another language. The second is to learn a multilingual system directly from noisy multilingual data (e.g., translated data), thus getting rid of the translation system dependence at inference time.
To evaluate the aforementioned systems, we propose a dataset called Multilingual Persona-Chat, or XPersona, by extending the Persona-Chat corpora BIBREF1 to six languages: Chinese, French, Indonesian, Italian, Korean, and Japanese. In XPersona, the training sets are automatically translated using translation APIs with several human-in-the-loop passes of mistake correction. In contrast, the validation and test sets are annotated by human experts to facilitate both automatic and human evaluations in multiple languages.
Furthermore, we propose competitive baselines in two training settings, namely, cross-lingual and multilingual, and compare them with translation pipeline models. Our baselines leverage pre-trained cross-lingual BIBREF4 and multilingual BIBREF5 models.
An extensive automatic and human evaluation BIBREF6 of our models shows that a multilingual system is able to outperform strong translation-based models and on par with or even improve the monolingual model. The cross-lingual performance is still lower than other models, which indicates that cross-lingual conversation modeling is very challenging. The main contribution of this paper are summarized as follows:
We present the first multilingual non-goal-oriented dialogue benchmark for evaluating multilingual generative chatbots.
We provide both cross-lingual and multilingual baselines and discuss their limitations to inspire future research.
We show the potential of multilingual systems to understand the mixed language dialogue context and generate coherent responses.
In XPersona, the training sets are automatically translated using translation APIs with several human-in-the-loop passes of mistake correction.
Answer: Automatically.