Overprotective Training Environments Fall Short at Testing Time: Let Models Contribute to Their Own Training
Authors: Alberto Testoni, Raffaella Bernardi (Università di Trento, 2020)
Topic: Chatbots and Dialogue Systems

Abstract
Despite important progress, conversational systems often generate dialogues that sound unnatural to humans. We conjecture that the reason lies in their different training and testing conditions: agents are trained in a controlled ``lab'' setting but tested in the ``wild''. During training, they learn to generate an utterance given the human dialogue history. On the other hand, during testing, they must interact with each other, and hence deal with noisy data. We propose to fill this gap by training the model with mixed batches containing both samples of human and machinegenerated dialogues. We assess the validity of the proposed method on GuessWhat?!, a visual referential game.

Introduction
Important progress has been made in the last years on developing conversational agents, thanks to the introduction of the encoder-decoder framework (Sutskever et al., 2014) that allows learning directly from raw data for both natural language understanding and generation. Promising results were obtained both for chit-chat (Vinyals and Le, 2015) and task-oriented dialogues (Lewis et al., 2017). The framework has been further extended to develop agents that can communicate about a visual content using natural language (de Vries et al., 2017; Mostafazadeh et al., 2017; Das et al., 2017a). It is not easy to evaluate the performance of dialogue systems, but one crucial aspect is the quality of the generated dialogue. These systems must in fact produce a dialogue that sounds natural to humans in order to be employed in realworld scenarios. Although there is not a general agreement on what makes a machine-generated text sound natural, some features can be easily identified: for instance, natural language respects syntactic rules and semantic constraints, it is coherent, it contains words with different frequency distribution but that crucially are informative for the conveyed message, and it does not have repetitions, both at a token and a sentence level. Unfortunately, even state-of-the-art dialogue systems often generate a language that sounds unnatural to humans, in particular with respect to the large number of repetitions contained in the generated output. We conjecture that part of the problem is due to the training paradigm adopted by most of the systems. In the Supervised Learning training paradigm, the utterances generated by the models during training are used only to compute a Log Likelihood loss function with the gold-standard human dialogues and they are then thrown away. In a multi-turn dialogue setting, for instance, the follow-up utterance is always generated starting from the human dialogue and not from the previously generated output. In this way, conversational agents never really interact one with the other. This procedure resembles a controlled ``laboratory setting'', where the agents are always exposed to ``clean'' human data at training time. Crucially, when tested, the agents are instead left alone ``in the wild'', without any human supervision. They have to ``survive'' in a new environment by exploiting the skills learned in the controlled lab setting and by interacting one with the other. Agents trained in a Reinforcement Learning fashion are instead trained ``in the wild'' by maximizing a reward function based on the task success of the agent, at the cost of a significant increase of computational complexity. Agents trained according to this paradigm generate many repetitions and the quality of the dialogue degrades. This issue is mildly solved by the Cooperative Learning training, but still, several repetitions occur in the dialogues, making them sound unnatural. In this paper, we propose a simple but effective method to alter the training environment so that it becomes more similar to the testing one (see Figure 1). In particular, we propose to replace part of the human training data with dialogues generated by conversational agents talking to each other; these dialogues are ``noisy'', since they may contain repetitions, a limited vocabulary etc. We then propose to train a new instance of the same conversational agent on this new training set. The model is now trained ``out of the lab'' since the data it is exposed to are less controlled and they get the model used to live in an environment more similar to the one it will encounter during testing. We assessed the validity of the proposed method on a referential visual dialogue game, GuessWhat?! (de Vries et al., 2017). We found that the model trained according to our method outperforms the one trained only on human data with respect both to the accuracy in the guessing game and to the linguistic quality of the generated dialogues. In particular, the number of games with repeated questions drops significantly.

Conclusion
Despite impressive progress on developing proficient conversational agents, current state-of-theart systems produce dialogues that do not sound as natural as they should. In particular, they contain a high number of repetitions. To address this issue, methods presented so far in the literature implement new loss functions, or modify the models’ architecture. When applied to referential guessing games, these techniques have the drawback of gaining little improvement, degrading the accuracy of the referential game, or producing incoherent dialogues. Our work presents a simple but effective method to improve the linguistic output of conversational agents playing the GuessWhat?! game. We modify the training set by replacing part of the dialogues produced by human annotators with machine-generated dialogues. We show that a state-of-the-art model benefits from being trained on this new mixed dataset: being exposed to a small number of ``imperfect'' dialogues at training time improves the quality of the output without deteriorating its accuracy on the task. Our results show an absolute improvement in the accuracy of +1.8% and a drop in the number of dialogues containing duplicated questions of around -14%. Further work is required to check the effectiveness of this approach on other tasks/datasets, and to explore other kinds of perturbations on the input of generative neural dialogue systems.