Question: What is the most common approach to training a dialogue agent?
Evidence: Training a dialogue agent to converse like a human requires extensive supervision. The most common approach is to train models to imitate humans in large corpora of crowdsourced or scraped conversations BIBREF0 . These fully-supervised conversations tend to be expensive to collect in sufficient quantity and/or occur in settings with significant differences from the deployment environment BIBREF1 . Instead, dialogue agents would ideally learn directly from dialogue, the conversations they participate in after deployment, which are usually abundant, task-specific, dynamic, and cheap. This corresponds to the way humans learn to converse—not merely observing others engaging in “expert-level” conversations, but instead actively adjusting and correcting our speech based on feedback woven throughout our own conversations BIBREF2 , BIBREF3 . Giving a dialogue agent this ability would enable it to continuously improve and adapt over its lifetime, rather than requiring additional annotation costs for each and every improvement.
However, naively training a dialogue agent on its own conversations yields poor results. For example, training a model on its own output can simply reinforce its existing failure modes, and mistakes by the agent can lead to absurd conversations that no longer resemble the target domain BIBREF4 . To combat this, one approach is to allow the agent to request feedback during conversations BIBREF5 , BIBREF6 , e.g., when it believes it is about to make a mistake. This approach, however, falls victim to the Dunning-Kruger effect BIBREF7 , which in this case suggests that a bad model will also be bad at knowing when it is doing a bad job. Regardless of when feedback is requested, existing methods typically require accompanying scalar rewards or adherence to particular templates or structure to ensure that the feedback is usable by the model BIBREF8 , BIBREF9 , BIBREF10 . These requirements may be acceptable for paid annotators, but they impose unnatural workflows on unpaid conversation partners in a standard dialogue environment. Humans are able to request and provide feedback using only natural language; ideally, dialogue agents would be able to do the same.
In this work we propose the self-feeding chatbot, a dialogue agent with the ability to extract new examples from the conversations it participates in after deployment (Figure FIGREF1 ). Concretely, in addition to being trained on the primary Dialogue task, the agent is trained to predict its speaking partner's satisfaction with its responses. When the conversation seems to be going well, the user's responses (but not the bot's own utterances) become the targets in new training examples for the Dialogue task. When the agent believes it has made a mistake, it instead requests feedback on what it could have said instead. Predicting the feedback that will be provided in a given context becomes an auxiliary task (Feedback) on which the model is also trained. Importantly, these new examples improve the agent's dialogue abilities while using only natural responses from the user that do not require special structure, accompanying numerical feedback, or additional human intervention in order to be used.
With this approach, the conversations the chatbot participates in are sliced into two complementary datasets—one largely protected from the chatbot's mistakes (Dialogue examples), and one which directly addresses them (Feedback examples). We validate our approach on the PersonaChat BIBREF11 dialogue dataset, finding empirically that regardless of the number of available supervised examples, the dialogue ability of the chatbot is always improved by adding the automatically extracted examples of either type, and improves the most by adding both.
The main contributions of this work thus include the following:
The datasets and models described in this paper are available via the ParlAI platform BIBREF12 , along with training code. Hyperparameter values are included in Appendix SECREF13 .
The most common approach is to train models to imitate humans in large corpora of crowdsourced or scraped conversations
Answer: The most common approach is to train models to imitate humans in large corpora of crowdsourced or scraped conversations.