Applying Psychology of Persuasion to Conversational Agents through Reinforcement Learning: an Exploratory Study
Authors: Francesca Di Massimo, Valentina Carfora, Patrizia Catellani, Marco Piastra (Università di Pavia, Università Cattolica del Sacro Cuore, 2019)
Topic: Chatbots and Dialogue Systems

Abstract
This study is set in the framework of task-oriented conversational agents in which dialogue management is obtained via Reinforcement Learning. The aim is to explore the possibility to overcome the typical end-to-end training approach through the integration of a quantitative model developed in the field of persuasion psychology. Such integration is expected to accelerate the training phase and improve the quality of the dialogue obtained. In this way, the resulting agent would take advantage of some subtle psychological aspects of the interaction that would be difficult to elicit via end-to-end training. We propose a theoretical architecture in which the psychological model above is translated into a probabilistic predictor and then integrated in the reinforcement learning process, intended in its partially observable variant. The experimental validation of the architecture proposed is currently ongoing.

Introduction
A typical conversational agent has a multi-stage architecture: spoken language, written language, and dialogue management, see Allen et al. (2001). This study focuses on dialogue management for task-oriented conversational agents. In particular, we focus on the creation of a dialogue manager aimed at inducing healthier nutritional habits in the interactant. Given that the task considered involves psychosocial aspects that are difficult to program directly, the idea of achieving an effective dialogue manager via machine learning techniques, reinforcement learning (RL) in particular, may seem attractive. At present, many RL-based approaches involve training an agent end-to-end from a dataset of recorded dialogues, see for instance Liu (2018). However, the chance of obtaining significant results in this way entails substantial efforts in both collecting sample data and performing experiments. Worse yet, such efforts ought to rely on the even stronger hypothesis that the RL agent would be able to elicit psychosocial aspects on its own. As an alternative, in this study we envisage the possibility to enhance the RL process by harnessing a model developed and accepted in the field of social psychology to provide a more reliable learning ground and a substantial accelerator for the process itself. Our study relies on a quantitative, causal model of human behavior being studied in the field of social psychology (see Carfora et al., 2019) aimed at assessing the effectiveness of message framing to induce healthier nutritional habits. The goal of the model is to assess whether messages with different frames can be differentially persuasive according to the users’ psychosocial characteristics.

Conclusion
In this work we explored the possibility of harnessing a complete and experimentally assessed SEM, developed in the field of persuasion psychology, as the basis for the reinforcement learning of a dialogue manager that drives a conversational agent whose task is inducing healthier nutritional habits in the interactant. The fundamental component of the method proposed is a DBN, which is derived from the SEM above and acts like a predictor for the belief state value in a POMDP. The main expected advantage is that, by doing so, the RL agent will not need a time-consuming period of training, possibly requiring the involvement of human interactants, but can be trained ‘in house’ – at least at the beginning – and be released in production at a later stage, once a first effective strategy has been achieved through the DBN. Such method still requires an experimental validation, which is the current objective of our working group.